27
Apr
Putting all crawler access in a separate apache2 log file?
Is there an easy way to configure apache2 so that all known crawler accesses (googlebot, yahoo slurp, etc) all go into their own log file, and everything else remains in the regular .log file? I’m hoping for something like “access.log”, “crawl.log” and “error.log”. Seems like it would have to have a well maintained list [...]
Continue Reading →