Did you know Google has an army of search engine evaluators ready to manually review your website?
There are currently 10,000 active evaluators looking at results pages.
And, your site could be next on their list.
All it takes is for you to get the thumbs down:
And you’ll see a dramatic decrease in traffic (more on what I have observed with that later).
Because today I am going to show you-
- How to check if a search engine evaluator has ever visited your site
- How to get an instant notification when they visit your site
I am going to tell you exactly what you need to do if they have paid you a visit!
What Will I Learn?
- What Are Search Engine Evaluators?
- Why Are Search Engine Evaluators Important?
- What Are Search Engine Evaluators Looking For Exactly?
- How To Tell If Search Engine Evaluators Have Visited Your Site
- How To Set Up Automatic Search Engine Evaluator Alerts
- Eek! They’re Watching Me! What Should I Do?
- Evaluator Flagged Your Site? Here’s What To Do…
What Are Search Engine Evaluators?
Search engine evaluators are real people who assess the quality of Google’s search results.
They perform “manual reviews” of your website to help Google interpret the results crawlers can’t.
(Later, I’ll show you how to see if your site has been reviewed and what to do about it.)
Evaluators are remote workers for companies like:
Which Google has been subcontracting since 2005.
Although their work is for Google they have no contact with Google directly.
They instead work to an (often leaked) 200+ page Google Search Quality Rating Guidelines.
Which brings us squarely to why they are so important…
Why Are Search Engine Evaluators Important?
Google’s algorithm is constantly evolving and changing.
Historically it changes hundreds of times a year.
Google needs “human input” to ensure they are returning the best results possible.
This is how it works:
A search engine evaluator will be given a target keyword and asked to assess the quality of a ranking page.
If they were given the keyword “How to make money on Fiverr” there’s a good chance they would end up on my site:
But how do search engine evaluators define quality?
Insiders report this is subjective and open to interpretation.
This is why Google often requests six people assess the same query.
So it’s important that we pay attention to them because their collective opinion makes a difference. You should think of them as a real life Jury.
What Are Search Engine Evaluators Looking For Exactly?
Thanks to ex-employee interviews and documents like this, we know precisely what search engine evaluators are looking for.
Google provide two frameworks to help evaluators make informed decisions-
- Needs met: do these results fulfill the needs of the person searching?
- E-A-T rating: the expertise, authority, and trustworthiness of a ranking page
Needs Met: Is Your Page Relevant And Accurate?
The Needs Met Rating Guideline is a metric to determine whether a result is helpful to mobile users.
It’s measured on a sliding five-point scale from “Fails to Meet” through to “Fully Meets”:
Each point on the scale requires you meet certain criteria.
- Fails to Meet: unsatisfying result, almost all mobile users would want (or need) to see additional results
- Slightly Meets: helpful for a small percentage of mobile users and has a loose connection to the search term
- Moderately Meets: helpful to a large portion of mobile users, though some may need to see additional results
- Highly Meets: helpful to the majority of mobile users, though some may still need to see additional results
- Fully Meets: almost all mobile users would be happy with this result and wouldn’t need to see any additional results
“Fully Meets” results have a special category and can only be achieved in rare cases where keywords are an exact match to the intent
Someone searching for “YouTube” would expect the first result to be the YouTube homepage:
Let’s look at these criteria in action to give them more context.
If you were a search engine evaluator assessing the query “How To Start A Blog”, you would assign:
- Fails: to a site that tells you how to start writing your first blog post
- Slightly: to a list of reasons to start a blog
- Moderately: an interview of how someone started a blog
- Highly meets: to a post listing blogging platforms to start a blog
- Fully meets: to a result that guides people through the process of starting a blog
The more a result meets the potential needs of the user, the higher up the scale it ranks.
When creating your own content it’s worth asking yourself the question-
“Does what I’m creating meet the needs of the person who will find it?”
The answer will reveal a lot about your SEO strategy.
E-A-T Rating: What Is The True Purpose Of Your Page?
Google’s E-A-T Rating is the second framework evaluators use.
E-A-T looks for:
- Expertise: of the content creator
- Authority: of the creator, the content and the website
- Trustworthiness: of the creator, the content and the website
And is the main rating which determines whether a page is high or low quality.
Because the combination of these 3x factors can help uncover the true purpose of a web page.
Specifically, is this page:
- Sharing trustworthy & helpful information?
- Sharing unreliable information could negatively impact the reader?
They base this on a wide range of factors such as:
- Industry guidelines
- Professional qualifications
- Research and citations
- Depth of website content
- Affiliations with relevant organisations
- Tone of voice
When giving an E-A-T Rating they would check if the content is from an accredited healthcare professional–
…written in a professional tone with reputable research-backed information…
… on a reputable website with a history of well-written medical content:
These factors indicate a trustworthy page containing information which will help the user.
A non-medical website which acts as a sales page for a supplement to treat a disease would be considered low quality.
This can also be applied to fields without official qualifications.
In this case, Google is looking for “everyday expertise”–
“If it seems as if the person creating the content has the type and amount of life experience to make him or her an “expert” on the topic, we will value this “everyday expertise” and not penalize the person/webpage/website for not having “formal” education or training in the field.”
To use me as an example…
If a search engine evaluator were to come to my blog they would see-
- Expertise: Award winning blog
- Authority: Published work in major publications & international speaker
- Trustworthiness: Teaching-focused content and testimonials
Take a look at this example from the evaluator guidelines for the search query “Chicken Recipes”–
When creating your website and content keep this E-A-T rating in mind.
Have you done enough to present yourself as a trustworthy authority in your niche?
What would you want a search engine evaluator to write about you in the E-A-T column?
This Section In Short
Google’s “Needs Met” and E-A-T ratings are 2x manual tests you need to pay attention to.
When creating content you should keep these two ratings in mind to ensure your content can pass these tests.
For the “Needs Met” rating:
- Focus on one potential intent for your keyword
- Make sure your content matches that intent
- Remove any irrelevant content from your page
For the E-A-T Rating:
- Create long-form in-depth content
- Use a tone of voice consistent with your industry
- Link to (or cite) reputable external resources
- Create a bio/about page thats shows your qualifications/everyday expertise
None of that is particularly hard to do and I recommend you bake those guidelines into your core processes as soon as possible!
How To Tell If Search Engine Evaluators Have Visited Your Site
So you might be wondering:
Well you are in luck!
Because you can monitor search engine evaluators behavior using a custom Google Analytics segment.
This will tell you-
- Which pages they visited
- How long they visited for
- How often they’ve visited
- Where they visited from
- What device they used
You can do this by creating a custom Google Analytics segment that tracks the traffic coming from search engine evaluator portals-
Or if that’s too much hassle for you-
Once you have it setup:
Set the date range to show the last 10 years.
This will give you a complete overview of how many times you were visited by search engine evaluators-
To get a deeper look at which pages were visited head to:
Behaviour > Site content > All pages
Set the secondary dimension to “full referrer” and you will see which evaluator visited & the page(s) they reviewed-
You can also use my custom dashboard to get a complete overview.
Including which country you were reviewed from and the device they were using-
Because this is a desktop device I would guess they were performing a “E-A-T” rating on my page.
How To Set Up Automatic Search Engine Evaluator Alerts
Google Analytics make’s it easy for us to setup custom alerts.
That means we can get an email or text message the moment a search engine evaluator visits our site.
Just jump into Google Analytics and head to:
Admin > Custom Alerts
And setup an alert that looks like this-
Whenever an evaluator visits your website you will receive an email or text message about it instantly.
Which brings us swiftly to the next section…
Eek! They’re Watching Me! What Should I Do?
If you have been visited by search engine evaluators it’s time to check your traffic.
Check your organic traffic against the segment you created in the last section.
If there is a traffic spike you’ve likely passed:
If there is a decrease you may have failed:
But don’t panic yet.
First you should fill out this form to help me, help you.
It’s possible the loss in traffic may also come from a recent change in Google’s algorithm.
Use one of these Google penalty checker tools to check this.
If it turns out you’ve failed, here’s what to do…
Evaluator Flagged Your Site? Here’s What To Do…
There are a few steps you can take to get your website back on track.
Take the reviewed page’s URL and run it through Google Search Console to see which keywords it was ranking for:
Take the top three keywords it was ranking for and search them in Google:
Go through the top ranking pages for these keywords and compare them against your content.
You’re looking to answer:
- What makes this content different from yours?
- What makes it better than yours?
- Does it satisfy the needs met rating?
- Does it satisfy EAT?
This should create a long list of actionable changes.
Then apply those changes to improve your content.
If you have been penalised and you’re unsure about how to recover, I’d love to help you get your traffic back.
Wrapping This Up…
They use two systems to help them grade the quality of a website:
- The “Needs Met” rating looks at how relevant and helpful a result us to a user.
- The E-A-T rating looks at the quality and intent of a page’s content.
The criteria for each of these ratings should be kept in mind when creating content for your website.
If you have been visited you should:
- Check the impact on your traffic
- Assess if you’ve passed or failed
- If you’ve failed, do your own Needs Met/E-A-T analysis & improve your content
From all the sites I have reviewed that have had visits from search engine raters…
…traffic has either increased or decreased sharply within 6 months of the visit.
So please please please please let me know in this form.
And if so:
I would love to collect some more data on this to help understand the exact role that search engine evaluators play.
IMPORTANT UPDATE This post is getting a lot of heat.
Please before you form an opinion, go and look at your own data.
SEO is data driven, the data is there for you to see – go and see it.
So far responses have fallen into 3 categories-
- People who were rated and saw a traffic increase
- People who were rated and saw a traffic decrease
- People who say it has no impact
Both category 1 and 2 of people have provided screenshots from Google Analytics to support their findings-
Interesting: just added @MattWoodwardUK's custom Google Analytics segment that tracks any traffic from the sites that Google uses to hire Search Quality Raters. A pretty clear trend from Raters looking at this URL to organic traffic gains.
One of many similar examples. pic.twitter.com/DDlJy7TMaQ
— Matthew Howells-Barby (@matthewbarby) March 12, 2019
Category 3 have yet to produce any data that shows it has no impact.
And Google are saying that I categorically misunderstand how Google uses search evaluators-
This post fundamentally misunderstands how Google uses raters for search quality — we've talked about that a lot of times, eg https://t.co/OMyWje5gPM and https://t.co/0PA6w6c1bM . They don't evaluate & penalize sites.
— 🍌 John 🍌 (@JohnMu) March 12, 2019
All I know is this:
I am looking for a single case of a review taking place and their being zero impact in organic traffic.
If you are reading this text it means I have still not found an example
If you have any data please share it.
I’ve reviewed around 30 sites so far and this is what I found-This is what I haven’t found yet-
- All 30 sites showed sharp increases or decreases in organic traffic after a review took place
- All 30 sites were affected purely on a page level
- All of these examples span 2014, 2015, 2016, 2017 and 2018
- There is no correlation of traffic increases/decreases with known algorithm updates
- The search engine evaluator team do a great job and in my opinion are 100% right in their decisions of promoting/demoting sites (assuming that is what is actually happening)
- An example from 2019
- A site that received a review and had no significant change in organic traffic
If you have an example of that please let me know.
I will update this post if find those examples, if you are reading this text it means I have not found one yet.
I also want to be clear – this has nothing to do with Google’s in house web spam team. This purely focuses on the external “quality raters” that Google contracts out to other companies.
I dont want your opinion.
But if you do want my opinion, it’s in the form of a question…