We use cookies to personalize content, to provide social media features and to analyze our traffic. We also share information about your use of our site with our social media, advertising and analytics partners. For information on how to change your cookie settings, please see our Privacy policy. Otherwise, if you agree to our use of cookies, please continue to use our website.


Yahoo has introduced a new tag, "Robots-Nocontent", which allows web masters to keep the Yahoo spider from reading certain portions of a web page. The robots.txt protocol has long allowed webmasters to keep spiders off entire pages. This new tag gives webmasters intra-page exclusion control.
Webmasters can now mark parts of a page with a 'robots-nocontent' tag which will indicate to our crawler what parts of a page are unrelated to the main content and are only useful for visitors. We won't use the terms contained in these special tagged sections as information for finding the page or for the abstract in the search results.
The most obvious use of this tag is wrapping navigational and menu elements. On many websites, identical navigation category links appear on every page. These links help users get around the site, but could confuse a spider about a page's meaning. The code is just a CSS class applied to a DIV or P or SPAN tag:

Because this paragraph is styled with the robots-nocontent class, the Yahoo spider should ignore it.

I can't see any downside to this useful convention; hopefully Google will soon follow suit.
Join the Discussion