{"id":245,"date":"2015-04-27T14:28:09","date_gmt":"2015-04-27T14:28:09","guid":{"rendered":"http:\/\/betakb.veeble.org\/?p=245"},"modified":"2025-03-28T12:09:38","modified_gmt":"2025-03-28T06:39:38","slug":"robot-txt","status":"publish","type":"post","link":"https:\/\/www.veeble.com\/kb\/robot-txt\/","title":{"rendered":"Robot.txt"},"content":{"rendered":"\n

There are several reasons and scenarios where we need to control the access of web robots or web crawlers or simple spiders, to our website or a part of our website. Like Google-bot (Google Spider) visiting our website, spam bots too will visit. Spam bots usually visit and collect private information from our website. When a robot crawls our website it uses a considerable amount of the website\u2019s bandwidth too! It is easy to control robots by disallowing the access of the web robots to our website through the usage of a simple \u2018robots.txt\u2019 file.<\/p>\n\n\n\n

Creating a robots.txt:<\/p>\n\n\n\n

Open a new File in any Text Editor Like Notepad.<\/p>\n\n\n\n

The rules in the robots.txt file are entered in a \u2018field\u2019: \u2018value\u2019 pair.<\/p>\n\n\n\n

<field>:<value>\n<\/pre>\n\n\n\n
<field><\/b><\/p>\n\n\n\n
Can have possible two values: allow or disallow for a particular URL.<\/p>\n\n\n\n
<value><\/b><\/p>\n\n\n\n
A URL or URI that the access or rule is specified.<\/p>\n\n\n\n
Examples:<\/p>\n\n\n\n
To exclude all the search engine robots from indexing our entire website.<\/p>\n\n\n\n
User-agent: *\nDisallow: \/\n<\/pre>\n\n\n\n
To exclude all the bots from a certain directory within our website.<\/p>\n\n\n\n
User-agent: *\nDisallow: \/aboutme\/\n<\/pre>\n\n\n\n
Disallow multiple directories.<\/p>\n\n\n\n
User-agent: *\nDisallow: \/aboutme\/\nDisallow: \/stats\/\n<\/pre>\n\n\n\n
To control access to specific documents.<\/p>\n\n\n\n
User-agent: *\nDisallow: \/myFolder\/name_me.html\n<\/pre>\n\n\n\n
To disallow a specific search engine bot<\/a> from indexing our website,<\/p>\n\n\n\n
User-agent: Robot_Name\nDisallow: \/\n<\/pre>\n\n\n\n
Advantages of Using Robots.txt:<\/p>\n\n\n\n