The robots.txt file sits on your server at this address www.mysite.com/robots.txt and tells search engines what they are, and aren’t allowed to do on your website. A robots file was traditionally used my websites who didn’t want their pages appearing in a search index. With a robots.txt file you can speak to all robots or [...]
Categories: Tips and Tricks
Tagged: New Liberty, POP, robots.txt