Posted: 10/27/2012 1:21:29 PM EDT
|
I have noticed on another forum, under the members that are logged in, that it shows Google[bot] will be logged in as a registered user but has no profile, nor is a registered user. Does anyone know what the hell this is, because it seems like everytime I notice it listed, within the next day or so, the forum gets spammed by fake users selling off the wall crap not pertaining to the forum.
Is this how Google caches the page for their search engine or something else. |
|
The webmaster can put a "robots.txt" file on the home page that tells the bots or crawlers where they can and can't go. It can also refuse to let them on the site entirely. This is of course if the crawlers are following the rules. We only let a few of the major search engines crawl our site at work because we have a little bit of a bandwidth issue. |
|
Quoted:
The webmaster can put a "robots.txt" file on the home page that tells the bots or crawlers where they can and can't go. It can also refuse to let them on the site entirely. This is of course if the crawlers are following the rules. We only let a few of the major search engines crawl our site at work because we have a little bit of a bandwidth issue. Can you PM me and let me know how your "robots.txt" file is worded. Thanks. |
|
Quoted: Quoted: The webmaster can put a "robots.txt" file on the home page that tells the bots or crawlers where they can and can't go. It can also refuse to let them on the site entirely. This is of course if the crawlers are following the rules. We only let a few of the major search engines crawl our site at work because we have a little bit of a bandwidth issue. Can you PM me and let me know how your "robots.txt" file is worded. Thanks. You bet, but I will have to do it on Monday when I'm back in the office. There are several good tutorials on the 'net concerning robots.txt and a few generators. |
|
Quoted:
Quoted:
Quoted:
The webmaster can put a "robots.txt" file on the home page that tells the bots or crawlers where they can and can't go. It can also refuse to let them on the site entirely. This is of course if the crawlers are following the rules. We only let a few of the major search engines crawl our site at work because we have a little bit of a bandwidth issue. Can you PM me and let me know how your "robots.txt" file is worded. Thanks. You bet, but I will have to do it on Monday when I'm back in the office. There are several good tutorials on the 'net concerning robots.txt and a few generators. Thanks. |
|
Sent. I forgot I had a copy on a thumb drive. I plugged it into a couple robot.txt file analyzers and it looks like I have a couple of errors in the syntax. Its been about a year and a half since I last looked it it. I will send you a new copy after I clean it up. Basically we allow Google and DMOZ and disallow everyone else. We also keep them out of some of the FrontPage folders to save bandwidth. |
|
Heres googles page on the subject
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449 |