Add robots.txt to a Django website
Adding a robots.txt file is a helpful piece of Search Engine Optimization as well as a way to request which bots, like ChatGPT and others, you do not wish to crawl your site.
Adding a robots.txt file is a helpful piece of Search Engine Optimization as well as a way to request which bots, like ChatGPT and others, you do not wish to crawl your site.