How Do Search Engines Work? What You Need To Know

how do search engines workYou’re about to learn the most important information many new SEOs miss out on:

How Do Search Engines Work
 
Understanding how search engines work is paramount!

Why?

Because you need to know how the system works in order to try and leverage it!

You can’t fix a car’s engine problem without knowing what’s going on under the hood…

… and the same rules apply for all search engines.

But, you don’t need to know everything either.

I’m going to take you through how search engines work step by step.

Let’s start with the essentials to lay the foundation for a successful SEO career.

How Google Works

Google works around these two main functions:

Crawling and Indexing.

We will be looking at these in more detail in a moment.

Google is the pioneer of many techniques you will see in this guide.

search engine market share

But how does the search engine we all know and love function?

It’s actually pretty simple and happens in a 2 stage process-

  1. Crawling: First Google “crawls the web” finding pages to add to their database
  2. Indexing: Then results are organised or “indexed” and added to their database

On a basic level, think of it like someone creating a huge library of books.

  • Crawling is finding new books to add to that library.
  • Indexing is putting the books you do have in a specific order (like genre or author).

crawling vs indexing

The only difference between a library and Google – is that Google has billions of books.

What Is Crawling?

When you do a Google Search, you might assume that Google searches the entire internet in that moment.

What’s really happening is that the search engine has compiled a huge database of pages and you are searching that database, NOT the entire internet.

Why does Google do this?

  • It can access this database reliably
  • It provides a quicker and more user-friendly experience
  • It allows Google to add its own “tags” to these pages

The first stage in adding pages to this database is called crawling.

Google has “crawlers” (or “spiders”) that it uses to scour the internet.

These Crawlers have 2x jobs-

  • Find new pages to index
  • Scrape information about those pages

I like to think of these crawlers like spies

They secretly go behind enemy lines to gather information and report back to HQ.

But how do they find websites, get access and recover that information?

Well, it’s not as complex as you might think…

How Does A Crawler Work?

All websites are part of a network called the World Wide Web which is basically a huge spider web.

The only difference is the world wide web is held together by links (also known as hyperlinks or backlinks).

And search engine crawlers (or spiders) use these links to travel around the web and discover new pages!

how a crawler works

Once crawlers find a new page, they start reading all of the content and code of the page.

In the ideal world we want the code to be as easy as possible for Google to interpret and understand.

Crawlers aren’t human-manned and work autonomously to decide whether pages they find should be added to the Google index or not.

For example:

Crawlers know that sites where you can buy guns and drugs shouldn’t be added to their database.

But once a crawler has decided that a page should be added to the database…

…it’s time for a site to enter the second stage of the process – indexing.

What Is Indexing?

Once a website has been crawled, it’s time to for it to be added to the database.

Indexing is where a search engine files away what’s been found and “tags” it.
(It’s a little more complex than that, but tagging works for now).

Think of it in terms of the library example I gave earlier:

If a box of assorted books is dropped off at a library, they will be organised, tagged and placed into their relevant sections-

  • Crime
  • Fantasy
  • Reference
  • Biographies

indexing

Let’s say you’re interested in learning French verbs.

You might go along to Google and type in “French verb list”.

google search result

The search engine will search its database for the pages that match that search term:

(in a matter of seconds)

google search results

There are a range of factors that determine why those pages show up in that order.

These factors vary depending on the search engine you’re using for example Amazon’s ranking factors are very different to Google’s (more on that later).

First just understand that the “indexing” process is fluid and as websites grow/add new content/delete new content – it will be re-crawled and re-indexed.

Why Do Some Pages Show Up Higher Than Others?

All search engines work on an algorithm which determines the order the pages appear.

That’s a series of equations that are based on different factors that help the computer decide where each piece of content should rank.

With Google, the simplest way I can explain it is through the concept of voting. In its very basic essence, the more “votes” a website has – the higher it ranks.

Let me explain it in more detail with a live example.

If you wanted to learn how to make money through Fiverr, it’s likely you would do a search like this in Google-

google search box

And you’ll find is that lots of web pages show up.

But there is one at the top.

That’s my how to make money on Fiverr tutorial:

fiverr google search results

That means of all the pages that relate to that topic, my site has the “most votes”

But how do websites get these “votes”?

Well…

“Votes” are otherwise known as backlinks meaning that when one website links to another website, they are essentially “voting” for it.

For example Niche Pursuits voted for my article when they linked to it-

link from other websites

But that’s not the only vote/backlink that page has.

It also has votes/backlinks from 83 other websites-

number of links or votes

And for the most part it’s these votes/backlinks that are driving the number #1 position in Google.

BUT!

It’s not quite as simple as that because not all “votes/backlinks” are created equal.

The bigger the site and the more well known it is, the more weight their “vote” carries.

If you were to get a link from a big media outlet like The New York Times or The Guardian, it would carry significantly more weight than a link for a brand new unestablished blog.

high authority links vs low authority links

I’ll show you how all of this pieces together later.

But for now:

Remember that in general the more “votes” a page has, the better and that not all “votes” are created equal.

How Other Search Engines Work

bing google same but differentHow do other search engines work in comparison to Google?

Many “web” search engines (like Bing) work on similar principles.

They will crawl and index web pages to use in their search results.

BUT they each have different ranking factors to consider which I will talk about in a minute.

Because there are also other types of search engines you should be aware of like-

  • YouTube
  • Amazon
  • eBay

  • Alibaba
  • Etsy
  • Quora

These sites all work as search engines within themselves but each of these sites has millions of visitors we could tap into.

Which is another reason  SEO isn’t dead !

And as long as search engines exist, we will be able to optimise for them.

How Amazon Works

Amazon is one of the world’s leading product search engines!

They offered hundereds of millions of products on their site and it’s internal search engine A9, has one main mission:

Showing the right products to the right customers.
 
amazon search bar

Whilst this may sound like Google’s philosophy:

Startupbros point out in their guide to ranking on Amazon, they’re slightly different.

  • Google says, “What results most accurately answer the searcher’s query?”
  • Amazon says, “What products is the searcher most likely to buy?””

google seo amazon seo

This is a huge difference because Google’s search engine relies on spiders to collect and organise external data.

Whereas Amazon’s search engines rely on internal data that is fully focused in serving their own self interest.

(Meaning: They want to sell the most products per person)

And Amazon does that by making sure the products you are most likely to buy, show up at the top of its search results.

They use many different factors to figure that out such as-

  • How many people have previously bought a product (the conversion rate)
  • How well it fits the needs of the buyer (the relevancy)
  • The previous actions of the buyer (what they are likely to want to buy this time)
  • The overall rating and satisfaction the product provides (how other buyers felt)

amazon ranking factors

This information enables them to show the most buyable product at the top of the page.

To give you an example:

If I was to search French learning books:

amazon search

That would give me this set of results.

Can you work out why?

amazon search results

Trying to figure out why is why search engine optimisation exists, because search engines aren’t big on sharing their ranking secrets.

But by observing results and applying a bit of intelligent thought, we can figure out what is happening-

  • The books closely match the keywords I used
  • These books are targeted at older learners (I’m logged into my account as an adult)
  • They have lots of high ratings (35 to 84 ratings each which is social proof)
  • There are tons of positive reviews
  • They are affordable (it’s not a product where I’d want to pay $300+)
  • They come from a reputable source I’m likely to trust (Collins is a huge UK brand)

If I wanted to buy a French learning book, all of these factors combined would be highly convincing.

And Amazon knows that! Their algorithm automatically optimise results to make sure the perfect product is in front of me when it needs to be.

You can figure out how other search engines work as well just by observing results and applying intelligent thought.

What Do Search Engines Want?

All search engines have a target goal with the information they provide.

They all want to meet the needs of their audience.

But these needs vary for each search engine…

Google’s Goal…

 
Is to provide you the most accurate answer/solution to your problem.

Whether that’s finding the local news, finding out which nickname Kanye West is now using or planning your wedding.

Amazon’s Goal…

 
Is to show you the right product for you to buy.

It finds quality products that meet your needs, backed by a history of buyer signals and ratings (with fast delivery).

YouTube’s Goal…

 
Is to show you the most relevant video.

Whether that’s a tutorial on brewing coffee or watching gamers play. This keeps you engaged and watching ads!

Quora’s Goal…

 
Is to pair you with the right answer to your question.

Whether that’s a question that already exists or a question you are asking for the first time.

Trust me when I tell you this:

Taking time to observe and understand what a search engine wants for their users will make your life much easier as an SEO.

Because once you know that, all you have to do is optimise for that!

what search engines want

Wrapping It Up

If you want to become a successful SEO…

Knowing how a search engine works is one of the most important steps.

I am going to cover more technical aspects with you soon.

For now let’s quickly recap what you’ve learned:

  • Google’s database is created by crawling and indexing
  • Crawling is when spiders follow links to discover new webpages
  • Indexing is when Google adds new pages to it’s database
  • Many factors influence how sites appear in Google, the main one is links
  • Each backlink to a page counts as one “vote”
  • Not all backlinks (votes) are created equal
  • Understanding a search engines goal helps you understand how they work

So now you know the basics of how search engines work…

…let’s get our hands dirty by taking a deep dive into Google ranking factors.

 Please feel free to ask any questions you have in the comments below… 
 

10 Responses

  1. Himanshu
    11.20.2019

    Hi,
    Your post is really amazing. I am a regular reader of your blog and every time I get something new. This post is also informative for me. I was not aware of a few things which you have explained here.

    • Matthew Woodward
      November 21st, 2019 at 2:45 am

      Thanks Himanshu – glad you continue to get value from coming to my blog.

  2. Ellery Prefontaine
    11.16.2019

    I just wanted to thank you for giving this information away to those of us just starting out. Also thank you for arranging it in such an easy, step by step manner and you can effectively search on it!

    • Matthew Woodward
      November 19th, 2019 at 4:16 am

      Hey Ellery – no problem at all. Welcome to ‘behind the scenes’ of the online world :)

  3. Prem mali
    10.12.2019

    Can you help me personaly, plzzz. I am facing a lots of difficulties i need someone to guide me.
    I AM REQUESTING YOU

    One of your visitors.

  4. vaijanath
    7.22.2019

    Thank u for sharing knowledge

    • Matthew Woodward
      July 23rd, 2019 at 8:16 am

      Glad I could help :)

  5. Osi Smog
    5.9.2019

    Hmmmm just stumbled on this concise and informative content, have more clear understanding on how search engine works. thanks for the write up

    • Matthew Woodward
      May 10th, 2019 at 4:35 am

      No worries, understanding how they work is the first step to ranking your websites on them :)

What are your thoughts?

* Name, Email, Comment are Required
27 Shares
Share26
Tweet1
Share