Robots are used by search engines (such as Google and Bing) to classify and catalogue data on websites. Site owners give Robots specific information on which pages the search engine can or cannot crawl. This is administrated via robots.txt file and the robots will choose to read this file before accessing the rest of the website. What can be seen and not seen by a search engine will ultimately affect your Search Engine Optimisation.
Originating from 1994 robots.txt or Robots Exclusions Protocol (REP) regulates what can be indexed by a web crawler. The text file is placed within the websites hierarchy and can look similar to this:
This is the file that contains the information for the search engine to crawl. The file tells the search engine where it has access to and where it doesn’t thus regulating its actions. A website can disallow a robot any access to their site completely or disallow it from specific areas, the instructions slightly differ in the appearance:
1. User-agent: * Disallow: /
The / indicates that all web crawlers are denied access to all directories.
2. User-agent: * Disallow: /example/
This example shows all robots are denied access to two directories.
3. User-agent: * Disallow: /no-google/
This example shows that one specific website has no access to any of the directories on the site.
For further information on how SEO Junkies can help improve your search engine optimisation then contact us today!
We offer a wealth of knowledge and experience that can help you improve your campaign’s rankings using our proven track record of results in the SERPs (Search Engine Results Pages).
Building 4 Millars Brook, Molly Millars Lane, Wokingham, Berkshire RG41 2AD, Telephone: 0845 373 0595, Email: firstname.lastname@example.org
If you would like to link to this blog then please copy and paste the HTML code below into your website.