All search engine web crawlers have user agents.
The Googlebot is one of the most common ones you will see in your server logs.
Here’s the problem:
Sometimes people mask themselves as Googlebot. This can affect your SEO results and even end up with you having content stolen by spammers.
But if you understand Googlebot’s user agents, you can:
- Identify fake Googlebots
- Use Googlebot to improve your SEO
In this blog, I will tell you everything you need to know about Googlebot user agent and how to use it to improve your SEO.
What Will I Learn?
What is Googlebot User Agent?
Googlebot user agent identifies Googlebot as it makes a request to crawl the content on your site.
Googlebot has a number of user agents that it uses to do its job properly.
How Does Googlebot User Agent Work?
You have lots of different elements on your site:
When Googlebot comes to crawl your site it needs to understand each of those elements. Googlebot has different user agents to understand different elements on a web page.
Googlebot also has user agents specific to different devices.
For example, it may crawl your site from the perspective of:
- An iPhone
- A Samsung phone
Each would look slightly different.
The Difference between User-Agents and Crawlers
Think of it like this:
Googlebot is the name of the search engine crawler whose job is to scan through the internet to find and follow links.
The user agent identifies that it’s Googlebot that wants to access the information on the site. In other words, the user agent is the name of Googlebot.
You then have user-agent strings that identify the more specific details about Googlebot.
The user agent string will tell you the:
- Browser used by Googlebot to crawl your site
- Device that Googlebot used
- Whether it was desktop or mobile
- IP address
A typical user agent string from Googlebot looks similar to this:
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Web browsers like Google Chrome and Firefox also have user agents.
Any time you go to a new site, the browser user agent requests the site’s current version.
List Of The Current Googlebot User Agents
Google is a very powerful search engine.
They collect a lot of data on tons of different things to help users find what they are looking for. To help with this Google has 18 different user agents:
|Crawler Name||User Agent|
|AdsBot Mobile Web Android||AdsBot-Google-Mobile|
|AdsBot Mobile Web||AdsBot-Google-Mobile|
|Mobile Apps Android||AdsBot-Google-Mobile-Apps|
|Google Read Aloud||Google-Read-Aloud|
|Duplex on the web||DuplexWeb-Google|
|Google Favicon||Google Favicon|
Not all of them are relevant for SEO.
How To Use Googlebot User Agent for SEO
If you know what information Google’s crawlers are using, you can verify whether Googlebot sees the same content on a website as a regular user does.
You can easily do this now with the Google Chrome browser:
Load a website, right-click on the page then click Inspect.
Click the 3 dot symbol, mouseover more tools then click Network Conditions.
Now select the Googlebot user agent from the dropdown.
Reload the page. You will now see the page the same as how Googlebot sees it.
The Best User Agents for SEO
Not all Googlebot User Agents are relevant for SEO.
There is a much smaller list to focus on:
|Mobile Apps Android||AdsBot-Google-Mobile-Apps|
How To Block Fake Googlebot Crawlers
Seeing Googlebot traffic but not sure it’s real?
It’s common for spammers to mask themselves as Googlebot and access your site.
You can verify whether or not it really is Googlebot. The easiest way to do this is by using the IP address because faking an IP address is really hard.
Here’s how to check:
- Copy the IP address of Googlebot from your server logs
- Paste it into the What Is My IP tool.
Now verify that the root domain name is either googlebot.com or google.com.
Found a fake bot?
GROWTH HACK: Sign up for my free video training series.
Wrapping It Up
Googlebot’s User Agents are very important to help Google provide the best results to their users.
Make sure that you optimise your site for crawl performance so Googlebot can do its job properly.
Spammers often mask themselves as Googlebot user agents. If you notice your site suddenly getting lots of traffic from Google, double-check the IP address from your server logs.
Getting ahead of it will save you trouble in the future.
Frequently Asked Questions
How do I set user agent on Googlebot?
You can set your user agent on the Chrome web browser to Googlebot. This allows you to see any web just as Googlebot sees it. You will need to use the built-in network conditions tool on Chrome.
How do I use Googlebot?
To use Googlebot, you need to fetch your website as Googlebot. This enables you to see the HTML version of your website just as Google sees it. Use the fetch as Google tool in Google Search Console.
How does Googlebot see my website?
Googlebot sees your website as a bunch of HTML code and links. It uses the code to understand what your website is about. It uses the links to discover and crawl other pages of your website.