Re: webv4 and web crawlers
By: Al to echicken on Mon Aug 10 2020 13:12:31
I have been messing around with the stock web interface and webv4 and I am just wondering if there
is a way to block web crawlers from the message areas the way the stock interface does it with the
robots.txt?
You can put a robots.txt file inside the document root (as on any webserver); I don't remember the exact path offhand, but the directory name is 'root' and it has an 'index.xjs' file in it.
You may also want to configure a web 'guest' user and limit its access to message areas, giving unauthenticated bots less surface area to crawl over. (In modopts.ini->[web], set 'guest' to the username of whatever account you want guests to be treated as.)
---
echicken
electronic chicken bbs - bbs.electronicchicken.com
þ Synchronet þ electronic chicken bbs - bbs.electronicchicken.com
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)