Robots.txt exclude
Robots.txt exclude
Hey, to forum / site admin(s):
I'd suggest you add a robots.txt exclude file to the root web directory here, and have it exclude the /forum subfolder. If you think about it... do you really want people to be able to do a google search and come up with results and links coming right to the forums? Do you want google caching everything in here?
Just an idea. I see neither the main site folder nor the forum subfolder have a robots.txt exclude file. That means google bot and all the other search engines will index every post in the forum. If you want that... then ok. But, I don't see why you'd want random google searches to link to some forum content.
I'd suggest you add a robots.txt exclude file to the root web directory here, and have it exclude the /forum subfolder. If you think about it... do you really want people to be able to do a google search and come up with results and links coming right to the forums? Do you want google caching everything in here?
Just an idea. I see neither the main site folder nor the forum subfolder have a robots.txt exclude file. That means google bot and all the other search engines will index every post in the forum. If you want that... then ok. But, I don't see why you'd want random google searches to link to some forum content.
Originally Posted by bagpiperdude90
Hey, to forum / site admin(s):
I'd suggest you add a robots.txt exclude file to the root web directory here, and have it exclude the /forum subfolder. If you think about it... do you really want people to be able to do a google search and come up with results and links coming right to the forums? Do you want google caching everything in here?
Just an idea. I see neither the main site folder nor the forum subfolder have a robots.txt exclude file. That means google bot and all the other search engines will index every post in the forum. If you want that... then ok. But, I don't see why you'd want random google searches to link to some forum content.
I'd suggest you add a robots.txt exclude file to the root web directory here, and have it exclude the /forum subfolder. If you think about it... do you really want people to be able to do a google search and come up with results and links coming right to the forums? Do you want google caching everything in here?
Just an idea. I see neither the main site folder nor the forum subfolder have a robots.txt exclude file. That means google bot and all the other search engines will index every post in the forum. If you want that... then ok. But, I don't see why you'd want random google searches to link to some forum content.
Isn't that the reason for search engines? Were helping Google find our content so that they can help searchers find results.
yeah, thats the reason for google.
I was under the impression this place was more of a "grandfathered in" place... i.e. people hear of it through a friend, who got it through a friend, who got it through a friend. Kinda like the black market. You're not going to search google and get the 3rd hit to be a black market site selling crack.
I guess I was under the wrong impression. Sorry! Go ahead and delete this topic if you want.
I was under the impression this place was more of a "grandfathered in" place... i.e. people hear of it through a friend, who got it through a friend, who got it through a friend. Kinda like the black market. You're not going to search google and get the 3rd hit to be a black market site selling crack.
I guess I was under the wrong impression. Sorry! Go ahead and delete this topic if you want.
