I recently saw
net (55.la) on the home page of a new online robots production function, the user opens robots.55.la, the search engine is not allowed to access part of the input to the Disallow:, click the "generate robots.txt files, can immediately generate. This is a practical and efficient webmaster tools.
maybe a lot of webmasters are familiar with robots.txt, but for some new people, it might be a little vague. I’ll talk about some of the related content of robots.
1, what is robots? What’s its function?
search engines automatically access web pages on the Internet and obtain web information through a program called robot (also known as spider).
can create a text file robots.txt in your website, the website does not want to be part of the robot access, declared in this file so that the site is all or part of the content can not be included in search engines, or specify the search engine included only the specified content. In fact, that is, through robots.txt can control SE included content, tell spiders what files and directories can be included, which can not be included.
2, why do you want to set robots.txt?
properly set up robots.txt, you can better maintain the Web server, improve the comprehensive performance of the site.
studies suggest that, if the site uses a custom 404 error page, then Spider will take it as robots.txt, although it is not a pure text file – this will give Spider index. A great deal of distress, affect the search engine on the web page included.
robots.txt can prevent unnecessary search engines from occupying the valuable bandwidth of the server, such as image strippers, and it doesn’t make much sense to most non graphics websites, but consumes a lot of bandwidth.
. Robots.txt can stop search engines from crawling and indexing non public pages.
4. For sites rich in content and large in number of pages, configuring robots.txt can prevent flooding Spider access, and if not controlled, it can even affect the normal access of the site.
from the point of view of SEO, the robots.txt file must be set, because
this is often the case on websites: different links point to similar web content. This is not in line with the principles of web content interaction, which is written on SEO". Using robots.txt files can shield secondary links.
website revision or URL rewrite optimization when the original >