When the people behind Magento 2 come up with some new updates, many people with a Magneto 2 site are quick to add them to their site. However, there are often issues that come up when people try to make use of these features. In this post, we are going to discuss some of the issues that many people are encountering when trying to edit robot.txt in Magento 2 admin. Make sure to keep reading if you’d like to find out more.
The Status Code
One of the first issues that many people are encountering is in relation to the status code. When you install Magento 2 and refrain from adding your own robots.txt files you try to access the robots.txt file, you’ll get a status 200 page instead of the regular 404. You might also notice that it weights some of the bytes. When it comes to the robots.txt file, you’ll find that it doesn’t actually exist in the root of your store. Sadly, you cannot disable the status 200 page and replace it with a 404 using the Magento admin panel.
The Sitemap Injection
If you want to add the sitemap directive to your robots.txt file, you’ll need to do the following: Go to Store, then Configuration, then Catalog, then XML Sitemap. You will then need to find Search Engine Submission Settings and choose the setting that lets you add the sitemap. When you do this, the URL that points towards your main sitemap index file gets added to the robots.txt file. When you visit your store, you’ll find that your sitemap looks like this: http://m221.example/sitemap.xml
Content Design Configuration
The next thing that you might encounter when you are editing robots.txt in Magento admin is in the Configuration section which you’ll find under Content then Design. From here, you might want to open the section called Search Engine Robots which you might find blank when you have clicked on it. Sadly, this is true despite the fact that you have added the sitemap directive. Working with this is not easy and when you use the rest to default button, you don’t get the result that you would like. You will get a boilerplate for the Magento 2 robots.txt file which you should not use for your site. This can disallow everything that you have with a parameter on your store and it can cause major problems.
Summary
When it comes to adding the robots.txt files to your store, you should make sure to do it in the way that you always have. The new updates can cause problems and so you should add a text file into the root if you want it to run successfully. This will help your Magento 2 store to stay running and you can be happy with the work that you have put into it.