The Meta Robots attribute, or Robots Meta tag, is used to provide search engines with instructions on how web pages should be crawled, indexed, and presented in search results.. As with all meta elements, the content of Meta Robots tags are not directly visible to website visitors. Robots Meta tags are more effective than robots.txt directives in preventing Google from indexing unimportant or duplicate content.