ROBOTs in the mist On the other hand, there are probably some of you who do not wish your pages to be indexed by the spiders at all. Worse yet, you may not have access to the robots.txt file. The robots META attribute was designed with this problem in mind. The default for the robot attribute is "all". This would allow all of the files to be indexed. "None" would tell the spider not to index any files, and not to follow the hyperlinks on the page to other pages. "Index" indicates that this page may be indexed by the spider, while "follow" would mean that the spider is free to follow the links from this page to other pages. (C) Webdevloper