Down the rabbit hole with meta robots

Mike Taylor | 29 September 2019

Oh, Google. You do like to keep us on our toes. Especially when it comes to SEO, or more specifically meta robots.


A fair point. Meta robots are tags used on your pages that you can use to instruct search engines (like Google) how you would like them to crawl or index parts of your website.

Do they actually crawl? Like spiders? Is this the Matrix?

Outdated early noughties references aside, sort of. They crawl pages and pick up tags. It’s basically like giving search engines rules.

Rules? I laugh in the face of rules! The rules were made to be broken! 

Not in this case. Search engines will simply ignore instructions which they don’t understand, so you need to get this stuff right. And that means knowing which search engines support which specific parameters.

Absolute minefield. You’re saying there’s different rules for different search engines?

Well, sometimes you might want to provide specific instructions to a specific search engine and not others. You might also want to provide completely different instructions to completely different search engines. But now we’re going deep.

…into the Matrix?

Let it go.

Fine. So, meta robots or tags are instructions for Google’s robot crawler thingys?

You’re getting there. It’s tricky to stay on top of this stuff, Google change their minds a lot – take the Meta Robots Nofollow update for example.

Oh god, they’ve got names. What is this witchcraft?

Previously, the Meta Robots Nofollow tag was a directive Google was obligated to obey when it came across it. Now, it’ll be treated as a hint. In a nutshell, Google may or may not obey the tag when it encounters it.

For real? It’s pointless then?

The idea of the tag was to create a page that could be indexed, but had no outbound links. The change in the Meta Robots Nofollow tag happened because Google couldn’t follow a link if it was prohibited from crawling the page. 

So if the crawlers can’t crawl through a website, that’s bad for business?

Hey presto, you got it. You want a crawler moving through your website, not stopping every which way. That’s just good SEO practice. 

Whoa. That actually makes sense. What now?

I can only show you the door. You’re the one who has to walk through it.

I guess s-wait, is that a Matrix quote?


Looking for more SEO guidance? Drop BBT a line today.