Set pages with a specific name in the URL to noindex

Header set X-Robots tag "noindex, follow" env=MY_SET_HEADER

It should be X-Robots-Tag, not X-Robots tag.

RewriteRule .* - [E=MY_SET_HEADER:1].

You also have an erroneous dot at the end of your RewriteRule directive.

RewriteCond %{REQUEST_URI} .*\/test\/page\/

The regex can be simplified but otherwise it still matches any URL containing /test/page/. Slashes do not need to be backslash escaped and the .* prefix is superfluous. ie. It’s the same as simply /test/page/.

The mod_rewrite directives would also need to go before the existing WordPress directives. If you placed the rule at the end of your .htaccess file (after the WP front-controller) then the mod_rewrite directive will simply never be processed.

That should resolve your problem, however, your rule can be simplified. You don’t need to use mod_rewrite here. You can use mod_setenvif to set your env var.

For example:

SetEnvIf Request_URI "/test/page/" NOINDEX
Header set X-Robots-Tag "noindex, follow" env=NOINDEX

By not using mod_rewrite the directives don’t necessarily need to go at the top of the .htaccess file.


Using WordPress / Front-Controller

Since you are using WordPress and consequently rewriting the request through a front-controller (ie. index.php), you will need to check for the REDIRECT_NOINDEX env var instead.

For example:

SetEnvIf Request_URI "/test/page/" NOINDEX
Header set X-Robots-Tag "noindex, follow" env=REDIRECT_NOINDEX

Whilst we set the NOINDEX env var, when the request is internally rewritten/redirected (by WordPress) the env var is renamed by prefixing REDIRECT_. So, NOINDEX becomes REDIRCT_NOINDEX after the first round of processing when the Header directive is processed (late).

CLICK HERE to find out more related problems solutions.

Leave a Comment

Your email address will not be published.

Scroll to Top