Algojunkie

Search Experience Optimization Resources

Google introduced X-Robots-Tag directive

Google introduced the new X-Robots-Tag directive, which adds Robots Exclusion Protocol (REP) META tag support for non-HTML pages, like PDF, Videos, Spreadsheet files, etc. You can have the same control over such indexed file types.

X-Robots-Tag: unavailable_after: 31 Dec 2007 23:59:59 EST

Also be aware that REP META tags can be useful for implementing noarchive, nosnippet, and now unavailable_after tags for page-level instruction, as opposed to robots.txt, which is controlled at the domain root.

Categories
SEO