How To Check If Search Engine Evaluators Have Manually Reviewed Your Site

Did you know Google has an army of search engine evaluators ready to manually review your website?

There are currently 10,000 active evaluators looking at results pages.

And, your site could be next on their list.

All it takes is for you to get the thumbs down:

And you’ll see a dramatic decrease in traffic (more on what I have observed with that later).

But don’t panic.

Because today I am going to show you-

  • How to check if a search engine evaluator has ever visited your site
  • How to get an instant notification when they visit your site

And more importantly than that…

I am going to tell you exactly what you need to do if they have paid you a visit!

What Are Search Engine Evaluators?

Search engine evaluators are real people who assess the quality of Google’s search results.

They perform “manual reviews” of your website to help Google interpret the results crawlers can’t.

(Later, I’ll show you how to see if your site has been reviewed and what to do about it.)

Evaluators are remote workers for companies like:

Which Google has been subcontracting since 2005.

Although their work is for Google they have no contact with Google directly.

They instead work to an (often leaked) 200+ page Google Search Quality Rating Guidelines.

Which brings us squarely to why they are so important…

Why Are Search Engine Evaluators Important?

Google’s algorithm is constantly evolving and changing.

Historically it changes hundreds of times a year.

Google needs “human input” to ensure they are returning the best results possible.

This is how it works:

A search engine evaluator will be given a target keyword and asked to assess the quality of a ranking page.

If they were given the keyword How to make money on Fiverr there’s a good chance they would end up on my site:

But how do search engine evaluators define quality?

Good question.

Insiders report this is subjective and open to interpretation.

This is why Google often requests six people assess the same query.

So it’s important that we pay attention to them because their collective opinion makes a difference. You should think of them as a real life Jury.

What Are Search Engine Evaluators Looking For Exactly?

Thanks to ex-employee interviews and documents like this, we know precisely what search engine evaluators are looking for.

/Phew!

Google provide two frameworks to help evaluators make informed decisions-

  • Needs met: do these results fulfill the needs of the person searching?
  • E-A-T rating: the expertise, authority, and trustworthiness of a ranking page

Let’s take a closer look at these frameworks and what they mean for you…

Needs Met: Is Your Page Relevant And Accurate?

The Needs Met Rating Guideline is a metric to determine whether a result is helpful to mobile users.

It’s measured on a sliding five-point scale from “Fails to Meet” through to “Fully Meets”:

Each point on the scale requires you meet certain criteria.

Google outlines these as:

  • Fails to Meet: unsatisfying result, almost all mobile users would want (or need) to see additional results
  • Slightly Meets: helpful for a small percentage of mobile users and has a loose connection to the search term
  • Moderately Meets: helpful to a large portion of mobile users, though some may need to see additional results
  • Highly Meets: helpful to the majority of mobile users, though some may still need to see additional results
  • Fully Meets: almost all mobile users would be happy with this result and wouldn’t need to see any additional results

It’s worth noting:

“Fully Meets” results have a special category and can only be achieved in rare cases where keywords are an exact match to the intent

For example:

Someone searching for “YouTube” would expect the first result to be the YouTube homepage:

youtube search example

Let’s look at these criteria in action to give them more context.

If you were a search engine evaluator assessing the query How To Start A Blog, you would assign:

  • Fails: to a site that tells you how to start writing your first blog post
  • Slightly: to a list of reasons to start a blog
  • Moderately: an interview of how someone started a blog
  • Highly meets: to a post listing blogging platforms to start a blog
  • Fully meets: to a result that guides people through the process of starting a blog

The more a result meets the potential needs of the user, the higher up the scale it ranks.

When creating your own content it’s worth asking yourself the question-

“Does what I’m creating meet the needs of the person who will find it?”

The answer will reveal a lot about your SEO strategy.

Want me to rank your site?

E-A-T Rating: What Is The True Purpose Of Your Page?

Google’s E-A-T Rating is the second framework evaluators use.

E-A-T looks for:

  • Expertise: of the content creator
  • Authority: of the creator, the content and the website
  • Trustworthiness: of the creator, the content and the website

This rating is “blind” to the type of page being evaluated.

And is the main rating which determines whether a page is high or low quality.

Why?

Because the combination of these 3x factors can help uncover the true purpose of a web page.

Specifically, is this page:

  • Sharing trustworthy & helpful information?
  • Sharing unreliable information could negatively impact the reader?

They base this on a wide range of factors such as:

  • Industry guidelines
  • Professional qualifications
  • Research and citations
  • Depth of website content
  • Affiliations with relevant organisations
  • Tone of voice

Let’s say a search engine evaluator is given a medical search result.

When giving an E-A-T Rating they would check if the content is from an accredited healthcare professional

…written in a professional tone with reputable research-backed information…

… on a reputable website with a history of well-written medical content:

These factors indicate a trustworthy page containing information which will help the user.

In contrast:

A non-medical website which acts as a sales page for a supplement to treat a disease would be considered low quality.

This can also be applied to fields without official qualifications.

In this case, Google is looking for “everyday expertise”

Here’s what the official guidelines say:

“If it seems as if the person creating the content has the type and amount of life experience to make him or her an “expert” on the topic, we will value this “everyday expertise” and not penalize the person/webpage/website for not having “formal” education or training in the field.”

To use me as an example…

If a search engine evaluator were to come to my blog they would see-

  • Expertise: Award winning blog
  • Authority: Published work in major publications & international speaker
  • Trustworthiness: Teaching-focused content and testimonials

And Google are looking at all types of results.

Take a look at this example from the evaluator guidelines for the search query “Chicken Recipes”

chicken recipes eat rating example

When creating your website and content keep this E-A-T rating in mind.

Have you done enough to present yourself as a trustworthy authority in your niche?

What would you want a search engine evaluator to write about you in the E-A-T column?

This Section In Short

Google’s “Needs Met” and E-A-T ratings are 2x manual tests you need to pay attention to.

When creating content you should keep these two ratings in mind to ensure your content can pass these tests.

For the “Needs Met” rating:

  1. Focus on one potential intent for your keyword
  2. Make sure your content matches that intent
  3. Remove any irrelevant content from your page

For the E-A-T Rating:

  1. Create long-form in-depth content
  2. Use a tone of voice consistent with your industry
  3. Link to (or cite) reputable external resources
  4. Create a bio/about page thats shows your qualifications/everyday expertise

None of that is particularly hard to do and I recommend you bake those guidelines into your core processes as soon as possible!

How To Tell If Search Engine Evaluators Have Visited Your Site

So you might be wondering:

Has a search engine evaluator ever manually reviewed my site?

Well you are in luck!

Because you can monitor search engine evaluators behavior using a custom Google Analytics segment.

This will tell you-

  • Which pages they visited
  • How long they visited for
  • How often they’ve visited
  • Where they visited from
  • What device they used

All of which will help you to understand what the evaluator was looking & why.

You can do this by creating a custom Google Analytics segment that tracks the traffic coming from search engine evaluator portals-

How to check if search engine evaluators visited your site with Google Analytics

Or if that’s too much hassle for you-

Click Here To Import My Custom Google Analytics Segment Now

Once you have it setup:

Set the date range to show the last 10 years.

This will give you a complete overview of how many times you were visited by search engine evaluators-

To get a deeper look at which pages were visited head to:

Behaviour > Site content > All pages

Set the secondary dimension to “full referrer” and you will see which evaluator visited & the page(s) they reviewed-

You can also use my custom dashboard to get a complete overview.

Including which country you were reviewed from and the device they were using-

Because this is a desktop device I would guess they were performing a “E-A-T” rating on my page.

How To Set Up Automatic Search Engine Evaluator Alerts

Google Analytics make’s it easy for us to setup custom alerts.

That means we can get an email or text message the moment a search engine evaluator visits our site.

I highly suggest you set this up right now.

Just jump into Google Analytics and head to:

Admin > Custom Alerts

And setup an alert that looks like this-

Then:

Whenever an evaluator visits your website you will receive an email or text message about it instantly.

Which brings us swiftly to the next section…

Eek! They’re Watching Me! What Should I Do?

If you have been visited by search engine evaluators it’s time to check your traffic.

Check your organic traffic against the segment you created in the last section.

If there is a traffic spike you’ve likely passed:

If there is a decrease you may have failed:

But don’t panic yet.

First you should fill out this form to help me, help you.

It’s possible the loss in traffic may also come from a recent change in Google’s algorithm.

Use one of these Google penalty checker tools to check this.

If it turns out you’ve failed, here’s what to do…

Evaluator Flagged Your Site? Here’s What To Do…

There are a few steps you can take to get your website back on track.

Take the reviewed page’s URL and run it through Google Search Console to see which keywords it was ranking for:

Take the top three keywords it was ranking for and search them in Google:

Go through the top ranking pages for these keywords and compare them against your content.

You’re looking to answer:

  • What makes this content different from yours?
  • What makes it better than yours?
  • Does it satisfy the needs met rating?
  • Does it satisfy EAT?

Essentially you want to ask yourself:

What would I want a search engine evaluator to write about my page?

This should create a long list of actionable changes.

Then apply those changes to improve your content.

If you have been penalised and you’re unsure about how to recover, I’d love to help you get your traffic back.

Wrapping This Up…

Search engine evaluators are Google’s human quality-interpreters.

They use two systems to help them grade the quality of a website:

  • The “Needs Met” rating looks at how relevant and helpful a result us to a user.
  • The E-A-T rating looks at the quality and intent of a page’s content.

The criteria for each of these ratings should be kept in mind when creating content for your website.

You can track search engine evaluator’s visits to your site by creating a custom Google Analytics segment and by creating an automatic alert.

If you have been visited you should:

  • Check the impact on your traffic
  • Assess if you’ve passed or failed
  • If you’ve failed, do your own Needs Met/E-A-T analysis & improve your content

In the mean time, I’m waiting to see what happens with this site that had a manual review recently:

From all the sites I have reviewed that have had visits from search engine raters…

…traffic has either increased or decreased sharply within 6 months of the visit.

So please please please please let me know in this form.

Has a search engine evaluator ever visited your site?

And if so:

Please fill out this quick form and let me know what happened

I would love to collect some more data on this to help understand the exact role that search engine evaluators play.

IMPORTANT UPDATE This post is getting a lot of heat.

Please before you form an opinion, go and look at your own data.

SEO is data driven, the data is there for you to see – go and see it.

So far responses have fallen into 3 categories-

  1. People who were rated and saw a traffic increase
  2. People who were rated and saw a traffic decrease
  3. People who say it has no impact

Both category 1 and 2 of people have provided screenshots from Google Analytics to support their findings-

Category 3 have yet to produce any data that shows it has no impact.

And Google are saying that I categorically misunderstand how Google uses search evaluators-

All I know is this:

I am looking for a single case of a review taking place and their being zero impact in organic traffic.

If you have that data – please share it!

If you are reading this text it means I have still not found an example

In fact:

If you have any data please share it.

I’ve reviewed around 30 sites so far and this is what I found-

  • All 30 sites showed sharp increases or decreases in organic traffic after a review took place
  • All 30 sites were affected purely on a page level
  • All of these examples span 2014, 2015, 2016, 2017 and 2018
  • There is no correlation of traffic increases/decreases with known algorithm updates
  • The search engine evaluator team do a great job and in my opinion are 100% right in their decisions of promoting/demoting sites (assuming that is what is actually happening)

This is what I haven’t found yet-

  • An example from 2019
  • A site that received a review and had no significant change in organic traffic

If you have an example of that please let me know.

I will update this post if find those examples, if you are reading this text it means I have not found one yet.

I also want to be clear – this has nothing to do with Google’s in house web spam team. This purely focuses on the external “quality raters” that Google contracts out to other companies.

Remember:

I dont want your opinion.

I want your data.

But if you do want my opinion, it’s in the form of a question…

100 Responses

  1. 8 ball pool
    7.1.2019

    Nice article, I have found new techniques from your website. It’s really helped me.

    • Matthew Woodward
      July 1st, 2019 at 11:39 pm

      No problem, pleased to hear its helped.

  2. Bulk SMS Service
    5.28.2019

    Nice Article Sir

    • Matthew Woodward
      May 29th, 2019 at 5:26 am

      Thanks very much – hope you found it useful.

  3. Gaurav Khurana
    5.6.2019

    Highly energetic post, I enjoyed that a lot.

    • Matthew Woodward
      May 8th, 2019 at 11:23 am

      Glad to hear it Gaurav :)

  4. Gal Gadot Net Worth
    5.5.2019

    This is a great website which help people to get knowledge in online field great work keep it up

    • Matthew Woodward
      May 6th, 2019 at 6:26 am

      Hope you found it useful!

  5. IGMI
    5.2.2019

    Hello,
    Thanks for sharing the amazing information. It really helped me out.

    • Matthew Woodward
      May 2nd, 2019 at 9:22 am

      No problem at all. Glad you found it useful :)

  6. Brij Bhushan
    5.1.2019

    Hello,
    This is a nice article you have shared with us. I agree with you as you explained, people are more interested to read it. You Are a very Good writer make understand better for everyone. Thanks a lot for posting this article.

    • Matthew Woodward
      May 2nd, 2019 at 9:19 am

      No worries Brij. Glad you liked the article and found it easy to understand.

  7. pleksi
    4.25.2019

    Hi, i have a hair site and decreased it traffic. I will try your suggestions. Thank you very much this informational article.

    • Matthew Woodward
      April 26th, 2019 at 2:53 am

      Best of luck.

  8. SKD Emarketing
    4.23.2019

    Thanks for sharing such great and informative article to check weather your site has reviewed by search engine evaluators.

    • Matthew Woodward
      April 24th, 2019 at 11:05 am

      No worries, hope it helped.

  9. Geetika
    4.20.2019

    Wow! Such a useful information, this really helps. Thanks for sharing it.

    • Matthew Woodward
      April 22nd, 2019 at 2:01 am

      No problem, hope you can put it to good use.

  10. Gyan Ranjan
    4.18.2019

    Such a great post with several ideas to increase search traffic. Thanks for this useful and helpful post for bloggers.

    • Matthew Woodward
      April 22nd, 2019 at 2:16 am

      No worries Gyan! Pleased it was helpful :)

  11. Riad Ahmed
    4.17.2019

    you write a good post. its really help me.

    • Matthew Woodward
      April 22nd, 2019 at 2:30 am

      Thanks Riad!

  12. bestpendiary
    4.1.2019

    this post was very helpful.

    • Matthew Woodward
      April 1st, 2019 at 4:34 am

      Great news

  13. Deepak Kumar Das
    3.28.2019

    Matthew you are a masterpiece in search engine ranking hack.Your think is outstanding and out of the box. Knowledge i get reading over your contents not found anywhere.

    • Matthew Woodward
      March 29th, 2019 at 9:20 am

      Thanks Deepak – great to know :)

  14. Jay Rowden
    3.28.2019

    I love your blog posts! Thanks so much for sharing your wisdom, as ever :)

    • Matthew Woodward
      March 29th, 2019 at 9:12 am

      Cheers Jay! Hope you found it interesting :)

  15. Richard James Brian
    3.28.2019

    Nice work Sir…

    • Matthew Woodward
      March 29th, 2019 at 9:14 am

      Thanks a lot Richard!

  16. Robin Khokhar
    3.27.2019

    Hello,
    Thanks for sharing the amazing information. I really appreciate your effort in making this post.
    Have a good day ahead.

    • Matthew Woodward
      March 29th, 2019 at 9:16 am

      Hey Robin, no problem I’m glad you liked it :)

  17. Piergiorgio Zotti
    3.24.2019

    Amazing post, Matthew. Thanks

    • Matthew Woodward
      March 25th, 2019 at 3:03 am

      No problem :)

  18. Master Kamal
    3.22.2019

    Great Post! Thanks for sharing your valuable thoughts. Keep sharing

    • Matthew Woodward
      March 22nd, 2019 at 3:09 am

      I certainly will do, thanks very much :)

  19. Jp narayan
    3.22.2019

    Your post is very good and the information spoken in it is of great use. But you tell me a solution so that I can increase the traffic on my blog.

  20. tere t
    3.20.2019

    thanks for sharing a good article I think it’s helpful.

    • Matthew Woodward
      March 21st, 2019 at 6:29 am

      No worries! Glad to hear it :)

  21. Hyder Ali Khan
    3.18.2019

    Sunder pichai also accepted that in front of the congress that search results are evaluated by 10k search quality raters and they are human.

    • Matthew Woodward
      March 18th, 2019 at 10:52 pm

      Interesting do you have a link to that?

  22. Deva
    3.17.2019

    Nice post thanks for sharing

    • Matthew Woodward
      March 18th, 2019 at 8:25 am

      Glad I could help Deva :)

  23. Mangal Singh
    3.17.2019

    Thank you for the blog was very helpful

    • Matthew Woodward
      March 18th, 2019 at 8:30 am

      Thanks Mangal! Glad it was helpful!

  24. alisha
    3.17.2019

    Thank you for the blog was very helpful

    • Matthew Woodward
      March 18th, 2019 at 8:28 am

      Glad to hear it Alisha

  25. tarkan yıldız
    3.16.2019

    thank you. has been shared lets continue.

    • Matthew Woodward
      March 18th, 2019 at 8:36 am

      Thanks for sharing Tarkan.

  26. Nishant Maitre
    3.15.2019

    Hey, Thanks! It’s quite interesting. Will definitely check. keep up the good work.

    • Matthew Woodward
      March 16th, 2019 at 6:49 am

      Thanks Nishant, let me know how you get on!

  27. simbainfotech
    3.15.2019

    nice blog.. thank you…

    • Matthew Woodward
      March 16th, 2019 at 7:04 am

      No worries, hope its useful.

  28. simbainfotech
    3.14.2019

    nice blog…thank you..

    • Matthew Woodward
      March 16th, 2019 at 7:13 am

      Cheers!

  29. smith
    3.14.2019

    nice…..

    • Matthew Woodward
      March 16th, 2019 at 7:18 am

      Thanks :)

  30. Allindiawatches
    3.14.2019

    Hi Matthew Woodward,
    Very nice article, keepup the good work

    • Matthew Woodward
      March 16th, 2019 at 7:18 am

      I certainly will do… thanks :)

  31. Olga Bolgurtseva
    3.13.2019

    Matthew, thanks – amazing post. I also checked, but have no visitors from this segment yet. My thoughts – there can be more domains, related to the assessor group.

    • Matthew Woodward
      March 13th, 2019 at 9:21 am

      Actually its highly unlikely a site will ever be reviewed just because of the sheer numbers

  32. midjack
    3.13.2019

    One of the very unique article i have seen a long time didn’t know any thing about it

    • Matthew Woodward
      March 13th, 2019 at 9:21 am

      Thanks – tell your friends :)

  33. umair
    3.13.2019

    This is superb post. Never thought about positively influencing manual review.

    Again great job Matt.

    • Matthew Woodward
      March 13th, 2019 at 9:21 am

      No problems Umair!

  34. Gabi
    3.13.2019

    Nice tut here Matthew! Thanks!

    Just set up my segment and found I’ve never been visited for a manual review. My site is just over 5 years old.

    Got my alert set up though for any possible future visits.

    • Matthew Woodward
      March 13th, 2019 at 9:21 am

      Good job Gabi!

  35. Bill Burniece
    3.13.2019

    Thanks for this tutorial Matthew. I had no idea you could actually find out whether or not your sites were evaluated other than a sudden traffic dropoff. Thanks again mate and keep up the great work. Sincerely – Bill

    • Matthew Woodward
      March 13th, 2019 at 9:22 am

      No problem Bill!

  36. Himal Ghale
    3.12.2019

    the very nice unique tutorial, I like so much your unique tutorial, hope will get other new tutorials in the futures.

    • Matthew Woodward
      March 13th, 2019 at 9:22 am

      Thanks Himal!

  37. Dave
    3.12.2019

    That’s a clever way to know if my site was reviewed, thank you. You have made things easier especially by making import process, One of the sites I work with is suffering from penalty and I ran the import on that website’s analytics. Looks like it’s a algorthmic penalty than manual.

    • Matthew Woodward
      March 13th, 2019 at 9:23 am

      No problem Dave!

      To be clear – this wont track googles actual manual review team

  38. Zach
    3.12.2019

    Very nice thank you, how to find Google evaluators, for other countries specific Google, my sites are manual reviewed, I get alert from google, about that, since we run banner ads etc. they send even a nice video, on what they dont like, but using your Google segment, show 0 visits.

    • Matthew Woodward
      March 12th, 2019 at 6:44 pm

      Can you clarify what you mean please?

  39. Sasha Khilova
    3.12.2019

    I have read it twice. Thanks!

    • Matthew Woodward
      March 12th, 2019 at 6:44 pm

      No problem – did you get a visit?

  40. chidi
    3.12.2019

    This is awesome!
    Thank you so much.
    Rushing now to check if my site have been evaluated.

    • Matthew Woodward
      March 12th, 2019 at 6:44 pm

      What is the verdict?

  41. Wade Cockfield
    3.12.2019

    Awesome post Matthew. Your Google Analytics audience segment import is a real time saver.

    • Matthew Woodward
      March 12th, 2019 at 6:44 pm

      No problem Wade!

  42. Tishka
    3.12.2019

    Thank you! This is awesome additional steps for checking why rankings dropped. I’ve checked 2 sites and have found 2 checks for 1 site (2015).

    • Matthew Woodward
      March 12th, 2019 at 6:45 pm

      Nice do you see a traffic increase or decrease in the next months?

  43. Jean-Christophe Chouinard
    3.12.2019

    This is an amazing post!

    The custom Analytics segment is brilliant!

    However, I’ve ran it on a fairly large website in Canada, and no activity was ever recorded using this custom segment. I’ll investigate…

    Thanks for the Post Matthew

    • Matthew Woodward
      March 12th, 2019 at 6:45 pm

      That is good news! Make sure you setup an alert

  44. Royan Shaw
    3.12.2019

    Amazed! That’s the word I have after reading this post. Matthew deserved easily every single dollar he earns times three

    • Matthew Woodward
      March 12th, 2019 at 6:45 pm

      Thanks glad you enjoyed it!

  45. Greg Baka
    3.12.2019

    How likely is it to have ZERO reviews in 10 years?
    I checked both of our US ecommerce sites that have been active since 2007. Neither showed any “review” traffic.
    But we have had wild fluctuations in organic traffic over that period.

    • Matthew Woodward
      March 12th, 2019 at 6:46 pm

      It’s very likely to have never had one

  46. Ed Orton
    3.12.2019

    I ran through it. It was an interesting exercise, but we have never been manually reviewed by anyone. Is there a way to get my site manually reviewed?

    I would LOVE to be manually reviewed because we are the only serious independent site online covering international dating. It is a very tough niche, because there is so much misinformation out there.

    International dating is great for a lot of guys and usually the best thing to ever happen to the women. And guess what? That is not from me that is from a feminist college professor who actually studied mail order brides and is still tracking them a decade later.

    For anyone interested, my site is: https://www.internationallovescout.com/

    • Matthew Woodward
      March 12th, 2019 at 6:46 pm

      HAHAHAHA that is a great way to look at it – I have never asked myself the question how to trigger it, but I am going to think about it…

  47. Daniel S
    3.12.2019

    Just checked it. No search evaluators came yet. Btw, they are checking only pages in English or international ones as well?

    • Matthew Woodward
      March 12th, 2019 at 6:47 pm

      I will let you know when I know – relying on people filling out the form to share results

    • Samuel Lavoie
      March 12th, 2019 at 10:25 pm

      I can confirm Google did/does have international based evaluators, even for smaller market such as French-Canada

      • Matthew Woodward
        March 13th, 2019 at 9:22 am

        Oooo do you know the names of the companies that are doing it?

  48. Rohan Verma
    3.12.2019

    Nice and very unique tutorial. As expected from Matthew

    • Matthew Woodward
      March 12th, 2019 at 12:35 pm

      Thanks!

      Did you have a review at all?

      • Rohan Verma
        March 12th, 2019 at 12:39 pm

        Thankfully no reviewers till now

        • Matthew Woodward
          March 12th, 2019 at 6:47 pm

          Excellent!

What are your thoughts?

* Name, Email, Comment are Required
607 Shares
Share555
Tweet52
Pin
Share