×

175,000+ Hours in Digital Marketing & Revenue Generation | EST. 2005

review

(888) 275-9221


#1 SEO Company in SC by Google | #1 Local SEO Agency in SC by Peers


Sometimes, there may be places on your website where you do not want the content to be indexed. A great tool to use for these areas is the robots.txt file in your website code. For example, you could

have a unique customer specific section which is not password protected and this page is not found in your site’s navigation. While this area may not be sensitive information, you also don’t want the public to see it either. By using the robots.txt file, you essentially tell the engines: “do not index this page.” Another example would be a print targeted version of a web page which does not feature the normal graphics, rich media files and other heavy bandwidth type features. If the search engines were to index this page, a duplicate content penalty could be levied on your site as this page is basically a stripped down copy (for a printer device) of another. Once again, the robots.txt file will stop the search engines from indexing this page.

The placement of your robots.txt file is very important. The best place is your main directory file because this is the first place the search engines check. By telling them early, your chances of having them comply are exponentially greater than if you bury the file deep in your site.

For the highest security concerns, I recommend not using the robots.txt file. Instead, create a username and password protected area. This gives you a much greater chance of keeping the bad guys out and the good guys in!

About the Author: Rhett DeMille, the owner of PalmettoSoft, is a leading search engine optimization consultant located in the Charlotte NC and Charleston SC areas.

Leave a Comment

What clients say

We make it easy to get started. No pressure and no contracts.

PalmettoSoft strives to serve its customers with marketing best practices and unsurpassed service, creating lasting and successful relationships with every business owner, marketing director, or digital dreamer who signs on.

LET’S GET STARTED