fbpx

Everything You Need To Know About Googlebot User Agent 

All search engine web crawlers have user agents.

The Googlebot is one of the most common ones you will see in your server logs.

Here’s the problem:

Sometimes people mask themselves as Googlebot. This can affect your SEO results and even end up with you having content stolen by spammers.

But if you understand Googlebot’s user agents, you can:

  • Identify fake Googlebots
  • Use Googlebot to improve your SEO

In this blog, I will tell you everything you need to know about Googlebot user agent and how to use it to improve your SEO.

What is Googlebot User Agent?

Googlebot user agent identifies Googlebot as it makes a request to crawl the content on your site.

googlebot user agent

Googlebot has a number of user agents that it uses to do its job properly.

How Does Googlebot User Agent Work?

You have lots of different elements on your site:

  • Images
  • Links
  • Text
  • Video

And more.

google images user agent

When Googlebot comes to crawl your site it needs to understand each of those elements. Googlebot has different user agents to understand different elements on a web page.

Googlebot also has user agents specific to different devices.

For example, it may crawl your site from the perspective of:

  • An iPhone
  • A Samsung phone

googlebot smartphone

Each would look slightly different.

The Difference between User-Agents and Crawlers

Think of it like this:

Googlebot is the name of the search engine crawler whose job is to scan through the internet to find and follow links.

The user agent identifies that it’s Googlebot that wants to access the information on the site. In other words, the user agent is the name of Googlebot.

You then have user-agent strings that identify the more specific details about Googlebot.

The user agent string will tell you the:

  • Browser used by Googlebot to crawl your site
  • Device that Googlebot used
  • Whether it was desktop or mobile
  • IP address

A typical user agent string from Googlebot looks similar to this:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Web browsers like Google Chrome and Firefox also have user agents.

Any time you go to a new site, the browser user agent requests the site’s current version.

List Of The Current Googlebot User Agents

Google is a very powerful search engine.

They collect a lot of data on tons of different things to help users find what they are looking for. To help with this Google has 18 different user agents:

Crawler Name User Agent
APIs-Google APIs-Google
Adsense Mediapartners-Google
AdsBot Mobile Web Android AdsBot-Google-Mobile
AdsBot Mobile Web AdsBot-Google-Mobile
AdsBot AdsBot-Google
Googlebot Images Googlebot

Googlebot-image

Googlebot News Googlebot

Googlebot-news

Googlebot Video Googlebot

Googlebot-video

Googlebot Desktop Googlebot
Googlebot Smartphone Googlebot
Mobile Adsense Mediapartners-Google
Mobile Apps Android AdsBot-Google-Mobile-Apps
Feedfetcher FeedFetcher-Google
Google Read Aloud Google-Read-Aloud
Duplex on the web DuplexWeb-Google
Google Favicon Google Favicon
Web Light googleweblight
Google StoreBot Storebot-Google

As you can see Google has different user agents to deliver all of their different services.

Not all of them are relevant for SEO.

How To Use Googlebot User Agent for SEO

If you know what information Google’s crawlers are using, you can verify whether Googlebot sees the same content on a website as a regular user does.

You can easily do this now with the Google Chrome browser:

Load a website, right-click on the page then click Inspect.

inspect tool chrome

Click the 3 dot symbol, mouseover more tools then click Network Conditions.

network conditions

Now select the Googlebot user agent from the dropdown.

select googlebot user agent

Reload the page. You will now see the page the same as how Googlebot sees it.

The Best User Agents for SEO

Not all Googlebot User Agents are relevant for SEO.

There is a much smaller list to focus on:

Googlebot Images Googlebot

Googlebot-image

Googlebot News Googlebot

Googlebot-news

Googlebot Video Googlebot

Googlebot-video

Googlebot Desktop Googlebot
Googlebot Smartphone Googlebot
Mobile Apps Android AdsBot-Google-Mobile-Apps

Keep these in mind when working with Googlebot user agents for SEO purposes.

How To Block Fake Googlebot Crawlers

Seeing Googlebot traffic but not sure it’s real?

It’s common for spammers to mask themselves as Googlebot and access your site.

You can verify whether or not it really is Googlebot. The easiest way to do this is by using the IP address because faking an IP address is really hard.

Here’s how to check:

  • Copy the IP address of Googlebot from your server logs
  • Paste it into the What Is My IP tool.

reverse dns lookup

Now verify that the root domain name is either googlebot.com or google.com.

root domain googlebot

Found a fake bot?

Take action to block it from accessing your website. You can also report it to Google.

GROWTH HACK: Sign up for my free video training series.

Wrapping It Up

Googlebot’s User Agents are very important to help Google provide the best results to their users.

Make sure that you optimise your site for crawl performance so Googlebot can do its job properly.

Remember:

Spammers often mask themselves as Googlebot user agents. If you notice your site suddenly getting lots of traffic from Google, double-check the IP address from your server logs.

Getting ahead of it will save you trouble in the future.

Want to learn more about bots? Check out Semrushbot, Ahrefsbot and Bingbot.

Frequently Asked Questions

How do I set user agent on Googlebot?

You can set your user agent on the Chrome web browser to Googlebot. This allows you to see any web just as Googlebot sees it. You will need to use the built-in network conditions tool on Chrome.

How do I use Googlebot?

To use Googlebot, you need to fetch your website as Googlebot. This enables you to see the HTML version of your website just as Google sees it. Use the fetch as Google tool in Google Search Console.

How does Googlebot see my website?

Googlebot sees your website as a bunch of HTML code and links. It uses the code to understand what your website is about. It uses the links to discover and crawl other pages of your website.

What are your thoughts?

* Name, Email, Comment are Required
3 Shares
Share
Tweet3
Share