4 Steps to create a robots.txt file in Django

robots.txt is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform web robots about which areas of the website should not be processed or scanned. The pages or the URL patterns included in the robots.txt file will not be indexed by the search engines.

When a site owner wishes to give instructions to web robots they place a text file ( called robots.txt) in the root of the website hierarchy (e.g. https://www.example.com/robots.txt). This text file contains the instructions in a specific format (see examples below). Robots that choose to follow the instructions try to fetch this file and read the instructions before fetching or scanning any other file from the website. If this file doesn't exist, web robots assume that the website owner is not wishing to place any limitations on crawling the entire site.
 

#example
User-agent: *
allow: /blog/
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

Creating a robots.txt file in Django is a simple task. Just follow the following steps to make a robots.txt file in Django.

Step 1

We will use the Django-robots module to make a robots.txt file for our website.

Install the package by using the following command :
pip install django-robots
 

Step 2

Include the "robots" app in the INSTALLED_APPS list :

#in settings.py
INSTALLED_APPS=[
.
.
.
'robots',
]

Now run the migrate command to include robots app tables in the Django admin app -
python manage.py migrate

 

Step 3

Add a URL pattern for the robots.txt file in the urls.py file.

#in urls.py
urlpatterns=[
.
.
url(r'^robots.txt$' , include('robots.urls')),
]

Use the example.com/robots.txt url for a robots.txt file.


Step 4

Now, go to django admin app and add the URL patterns which you want to allow or disallow in Url .
Set rules for these URLs in the Rules table. You can select the URLs manually to apply rules for them.