The which backlink checker is best argument constantly shows up on forums.
I tried to get to the bottom of this with my backlink checker experiment at the start of the year.
To be honest the post didn’t go down well with some people.
Interesting Background Story
When I published the original comparison post it sparked a lot of controversy. I had just setup a small test to see which is better based on 3 sites I own and have full visibility and control of.
It turned out that lots of customers of the featured companies were writing in and referencing my post to see what was going on.
Hats off to Viktar for dealing with it in the way that he did! Customers love transparency but companies are terrified by it – kudos sir!
The Majestic SEO team chose to respond very differently with the marketing CEO Dixon Jones taking the lead.
It started with this comment on the article-
I’m a director here at Majestic. Thanks for the study.
It is true that Ahrefs are increasing their index size. However, using resources for “discovery” comes at a cost. The cost is the calculation as to the quality of each link.
It’s hard to comment on a study marked Site A, Site B, Site C. The data is public for all the data sets except GWT, so I see no reason why studies can’t use verifiable sites.
You also did not specify whether you used Majestic’s Fresh or Historic Index. As you dig into those data sets you will find some links only one index and some only in the other, however – both indexes are excellent for different types of analysis.
Dropping into the analysis, however, is where analysis REALLY needs exploring in 2013. Penguin tells us that it is quality, Not quantity that will count from here on in. Looking at the top 20 links in each list will probably be plenty to come to a conclusion that may not be the one you suggest.
Anyway – can you list the three sites you used for the analysis?
The first line of his response is to instantly discredit the competition – I don’t like that, its not their fault you came second no matter how you dress it up.
But he does make some very valuable and critical points, I had used a small dataset that I had chosen to keep anonymous.
A couple of days after Dixons comment on the 14th January 2013 Ann Smarty stopped by to leave a comment! Yes, Ann Smarty herself!
I was totally stoked because I have followed her for a number of years, and now here she is on my blog! Wow!
Thanks for the test! I didn’t know Raven and SEO SpyGlass had their own indices. I thought they were using someone else’s APIs
As for Majestic, have you tried using their “Historic” index as well? It returns much more data in my experience.
She also makes a valuable point in that those 2 indexes do return different datasets.
The Twist In The Tale
A couple of weeks later on February 5th Ann Smarty wrote this post – aHrefs versus MajesticSEO: The Real Test
Interesting choice of post name, and very similar to what I had published and she had commented on just a couple of weeks earlier.
You would have thought that for the benefit of the readers and to offer them the braodest look at the debate she would have referenced my post as well – no dice.
Especially when you read this line-
since there have been a few rather vague comparisons recently, here’s the indepth experiment.
I can understand why she didn’t include my results though, I mean her dataset focuses on 100 random domains while mine focuses on only 3.
You should read that post though so you get a better understanding of the tools.
Then Dixon dropped this comment on the article-
Thanks ever so much for writing this all up Ann. For the record I have a clear and obvious preference for MajesticSEO.
Jacob: There were (as Link Assistant or Spyglass or someone said) quite a few things found wanting only the Matthew Woodward post. The most glaringly obvious is that the analysis was done with three sites he knows well, but will not disclose. Three sites with clear bias vs 100 sites with a random element is a significant difference.
What wasn’t said was this: I don’t know whether he included his own site in his list of three, but the objective nature of the analysis gets stretched further when you see that he is blocking Majestic’s bot but not Ahrefs’: https://www.majesticseo.com/reports/site-explorer?folder=&IndexDataSource=F&q=http%3A%2F%2Fwww.matthewwoodward.co.uk%2F
(You need access to the site explorer to see the block in this link). You also won’t see the robots.txt file unless you know what you are looking for. Now – we do not need to crawl his site to see links TO his site, but if he insists on saying one system is better than another, he really shouldn’t make the claim on a site which shows such obvious bias.
There. I said it. I tried to bite my lip for a few weeks, but good people are reading that review and not seeing the lack of objectivity.
Seeing as we aren’t biting lips anymore-
Notice how he opens up and highlights his bias which after establishing, allows him to start pointing the bias finger.
Marketing 101 folks. Establish, Finger. Profit.
I actually made a Dixon based meme with those words to go here – but I removed it during editing to be nice. But if you guys want it…
Either way he has the right approach, I’m doing exactly that with this blog! One day I’ll finger you with a paid product – in fact i’m fingering you right now because this post will probably earn me affiliate commision.
Anyway as you can tell, that didn’t sit well with me-
Dixon I have noted the weaknesses in my initial tests from various sources and while you suggest my test was done with ‘bias’ because they are 3 of my sites at random is complete bullshit.
I realise I left to much wiggle room for people to come up with justifications as to why Majestic didn’t win the test.
To be honest I didn’t think it would cause this much of a stir!
The follow up test is going to leave no wiggle room and make this ‘random’ URL sample look insignificant.
And seeing as your doing everything you can to discredit me and my data, I’ll also be highlighting the relationship between Ann Smarty, yourselves and IMN. I wonder how many people read this very post and weren’t aware of that?
For a CEO to come out and try to discredit people makes you look very weak – you should take a leaf out of Viktar Khamianok’s book and handle it like a boss http://www.link-assistant.com/blog/backlink-checkers-compared-seo-spyglass-ahrefs-majestic-seo-seomoz-raven-tools/
We’ll see how random those 100 URL’s really were with my follow up – its going to be air tight this time.
ps. I dont see what in my robots.txt file https://www.matthewwoodward.co.uk/robots.txt would be blocking it?
To explain a few things here. It turns out that my new host WPEngine blocks Majestic at the server level before it ever reaches my site – Dixon personally investigated this and confirmed that to be the case.
The ironic thing about all of this though, is his claim of bias.
What Dixon didn’t know is that the company behind SEO Spyglass had just given me over $2,000 worth of prizes for my readers.
They came last in my test.
Seeing as Dixon leans on it heavily in his argument, and establishes he is not biased – lets explore that for a second and put all of our cards on the table.
The Ahrefs / Majestic SEO / Ann Smarty Relationship
This is where things get interesting.
Ann Smarty is the community manager at InternetMarketingNinjas.com (IMN) which is a highly successful internet marketing agency.
Actually its the first IM agency site I’ve ever come across where I think, they actually know what they are talking about! Check them out!
IMN also control sites like WebmasterWorld, SEOChat, Crea8aSiteForums, ThreadWatch and so forth.
Up until the start of February – Ahrefs and IMN were in a partnership together. In fact Ahrefs linked to IMN in the footer!
On the 1st February 2013 the partnership between Ahrefs and IMN ended.
By the 8th of February 2013 IMN had plastered promotions for Majestic SEO all over their network of sites.
It was latest discovered that on the 29th January 2013 this tweet was made on the Ahrefs Twitter account.
That Tweet not only promotes Dixon, but if you read the linked posts is all about Majestic SEO – Ahrefs main competitor.
According to the Ahrefs team, IMN were in control of Ahrefs social media accounts at this time and they did not make the tweet themselves.
Timeline of events
To make it easy to understand-
- 11th Jan – I publish my initial comparison post
- 11th Jan – Dixon is aware and submits a comment
- 14th Jan – Ann Smarty is aware and submits a comment
- 29th Jan – IMN (Ann Smarty is a manager) posts a Tweet on Ahrefs account promoting Dixon (Majestic)
- 01st Feb – The relationship between IMN and Ahrefs ends
- 05th Feb – Ann Smarty publishes her 100 random domain comparison
- 05th Feb – Dixon Jones (Majestic) discredits my posts and says I’m biased
- 08th Feb – Majestic SEO banners start appearing on IMN properties
- 08th Feb – Dixon finally gets in touch with me directly in the correct manner
I’ll let you do the math on that.
I should highlight that since Dixon got in touch with me he has being very friendly behind closed doors, its a shame he didnt take that approach in the public domain.
However that doesn’t come before telling it how it is and how this post transpired.
In the public domain he chose to discredit me, competitors and vaguely string together bias, publically. He put me in a position where I have to defend myself publically.
All the other people in my original test sent me a nice email asking how they could help.
I should also make you aware of my relationships upfront with each party that is involved-
Everything you see here plus a few friendly private emails clearing the air, diagnosing the host issue and discussing my follow up. I initially delayed this test for Majestic to roll out new features so it wasn’t invalidated. Dixon called me out publically and disputed the original test, this is my air tight response.
They have provided me with API access to conduct the test free of charge and filled in some of the holes above. They are also offering the same access to any blogger that will independently verify this test, see the end of this post for details. I also use their affiliate links throughout this post, Majestic do not have an affiliate program or I would use theirs as well.
I’m a long time reader of her content and a fan.
Internet Marketing Ninjas
It is hard not to know who they are, I know of them – never dealt with them or any employee to my knowledge.
So Which One Is Better?
So all of this has distracted from the route question, which is better – Ahrefs or Majestic SEO?
Only cold hard data can decide!
In Dixons efforts to discredit me and highlight bias, he did raise some valid points that cannot be ignored-
- The original test was done with 3 of my personal sites I would not disclose
- That makes the entire test biased in Dixons opinion
- The data set was very small with just 3 sites analysed and compared
- The results could not be independently verified
I promised that the “The follow up test is going to leave no wiggle room and make this ‘random’ URL sample look insignificant.”
So with the points above and that promise in mind, I give you-
Ahrefs vs Majestic SEO – The 1 Million Domain Showdown
My original post used 3 of my own sites, Ann Smartys posts used 100 sites.
This Ahrefs vs Majestic SEO test is going to use 1 million sites. No more messing about eh?
The Test Details
This is how the Ahrefs vs Majestic SEO test is put together – you will be able to independently verify all of these results yourself, instructions at the end of the post.
Initial Data Source – The Majestic Million
For those that don’t know, the Majestic SEO team publish what is called the Majestic Million.
The Majestic Million is a list of the top 1 million website in the world, based on the number of referring IP’s found for that domain in the fresh index.
These are the 1 million sites that Majestic SEO has more fresh backlink data on than any others.
So with this, Majestic SEO are outright telling us these are the sites they know the most about in terms of backlinks.
Out of all of their data, these 1 million sites are the most linked to in the world according to them.
So thats the “bias” taken care of upfront.
You can download a copy for yourself free of charge and it will tell you the total number of linking subnets (RefSubNets) and the total number of linking IPs (RefIPs) for each domain in the top million.
Making The Comparison
To compare Majestic SEO with Ahrefs I’m going to look up the total number of linking subnets and IP’s for all of the domains in the majestic million.
To give you an example with this blog-
- Majestic SEO – This report shows 209 IP’s & 193 subnets
- Ahrefs – This report shows 331 IP’s & 288 subnets
So in that specific instance, Ahrefs wins the test.
Now imagine doing the same – but for 1 million sites.
That is the test I have done based on what Majestic say are the top 1 million sites in the world in terms of links.
Like I said – no wiggle room this time.
This is what the CSV data looks like in its raw form-
Don’t be fooled by what you can see in that image though!
Feel free to download a copy of the data that is used in this test.
The Test Results
With such a huge amount of data I decided to split the results into 10 groups of 100,000 URLs and then compare those groups.
This is what the group numbers look like (refer to by_groups.csv in the data pack)–
|Group||Ahref Wins (IP)||Majestic Wins (IP)||Ahref Wins (Subnet)||Majestic Wins (Subnet)|
Just looks like a bunch of meaningless numbers right?
What about now-
Wins By IP Address
Wins By Subnets
Pretty brutal to look at if you are a current Majestic SEO customer right?
Now might be the right time to switch to Ahrefs – click here to claim a free account.
But before you do lets look at the totals-
In wee pretty graph format-
Ahrefs Vs Majestic SEO Conclusion
Irrespective of opinion, relationships, bias and discreditation – the data does not lie.
The data never lies, its why I love working with data so much. Want to settle which design is better? Test it.
Want to see which is the best backlink checker? Test it.
Data does not lie.
Ahrefs is clearly the winner here by a huge margin – a much larger margin than my initial test with just 3 domains highlighted.
In the Wins by IP test, Ahrefs is finding more than twice the amount than Majestic SEO is – 57% more to be exact. They are winning on 71% of domains compared to Majestic SEO’s 29%.
In the Wins by Subnets test, Ahrefs is still the clear winner finding 284,551 more links than Majestic SEO. They are winning on 64% of domains compared to Majestic SEO’s 36%.
So quite clearly, the best backlink checker is Ahrefs PERIOD.
Even if the source data is Majestic SEO’s very own Majestic Million – Ahrefs still knows more about them than Majestic does.
UPDATE #1: PLEASE SEE MAJESTIC’S RESPONSE AT THE END OF THIS POST
UPDATE #2: AHREFS RESPONSE – FIXES APPLIED – NEW DATA
UPDATE #3: MAJESTIC GO ALL OUT
How To Verify The Data For Yourself
In an effort to be unbiased and transparent I have 2 seperate ways for you to independently verify the data yourself.
All of the data used in this test is available to download here.
Verification Method #1 – Easy
This is the easiest way to spot check the data.
- Download the source data
- Open the by_domains.csv file
- Select a line at random
- Enter the domain from column A into the search box at Majestic SEO and Ahrefs
- For Majestic SEO look at Referring IP addresses and Referring Class C subnets:
- For Ahrefs look at Referring IPs and Referring subnets
You can manually spot check these at random!
Verification Method #2 – Hard
What if you want to verify the data for all million domains though?
Well for that you will need a linux server with shell access, an Ahrefs API key and these files.
WARNING: Before you do this be aware that 1,000,000 API credits with Ahrefs costs $10,000.
For full transparency’s sake Ahrefs gave me an API key with 1,000,000 credits so I could run this test.
If any established bloggers want to confirm these results themselves and publish on their blog then please get in touch with me.
To try and remove any doubt around this test I asked the Ahrefs team if there was a way people could verify the results in bulk independently which isn’t possible.
So they suggesteed that they will provide API keys to other established bloggers to confirm and publish the results themselves.
This should help remove any wiggle room for discreditation of the test or data.
Anyway here is how to do it-
- Download this & you will find 2 files, get_stats.php and load.sh
- Open get_stats.php and update it with your Ahrefs API key on line 3
- Download the Majestic Million CSV
- Upload get_stats.php, load.sh & majestic_million.csv to your server
- Make sure all of the files are in the same folder
- Connect to your server with an SSH client and execute the load.sh script
- Wait a few hours
- This will output a file called by_domains.txt, rename to by_domains.csv
Remember folks regardless of all of the controversy the data does not lie.
The simple fact is Ahrefs knows 57% more about Majestic SEO’s Majestic Million than they do.
You can sign up for a free Ahrefs account here.
If I ever meet Dixon at a conference hopefully he won’t drop kick me through a table. Perhaps I would let him write Majestic SEO across my head in permanent marker for an hour though =D
Next time you see this topic in a forum – feel free to use this post to instantly win the argument.
Update #1: Majestic SEO’s Initial Response
Alex Chudnovsky the Majestic SEO managing director submitted this in a comment but it is only fair this is highlighted and made part of the post.
Here is what he had to say-
It’s nice to see that Majestic Million data that we offer for free would cost $10000 elsewhere.
Data does not lie, but it can be wrong/bogus/false and a good analyst would always keep an eye for anomalies before jumping to big conclusions.
We are pretty busy right now setting up new hardware, so we’ll get back to you on this in detail next week, in the meantime I think you should double check your key numbers – ahrefs appears to be the only backlink provider which somehow shows MORE referring IPs/subnets than number of referring domains (that’s like 2+2=5).
Using your own example listed in the post above –
Referring IPs: 331
Referring subnets: 288
Referring domains: 282
Naturally at least one of those numbers is incorrect.
We can’t easily get list of those IPs, but the referring domain count is confirmed by External links tab using 1 link per ref domain limit –
This shows 282 matches, the same as on the overview page… the same but potentially misleading. How many of you know that ahrefs counts hostnames as referring domains? It’s easy to check –
There are 3 referring domains shown there when limiting them to blogspot.com:
It’s the same root domain (blogspot.com) and Majestic counts it as 1 referring domain. Every major back link provider does the same as otherwise numbers will be artificially inflated and likely to mislead those who’d trust them blindly.
Perhaps you should ask ahrefs why their ratio of IPs/RefDomains is over 1 (this is just wrong), whereas more typical ratio is a lot less, for example Open Site Explorer (OSE) shows 85 linking C blocks (subnets) and 174 ref domains – ratio of 0.48 and in our case it is 193/287=0.67. For big domains such as google.com this ratio will be a lot lower – 0.024 in Majestic’s case and 0.032 in OSE. Homework – calculate same ratio for ahrefs and spot another anomaly shown for google.com!
We’ll get back to you on this next week with a more detailed answer but I hope that you’ll get explanation from ahrefs on their innovative counting of referring IPs that can be greater than the number of referring domains.
We’ll see what the Ahrefs team say tomorrow, I’m going to sleep its late here. Get a feeling I’m going to wake up to war!
Update #2 – Ahrefs Bites Back
The Ahrefs team have just sent me this update through. I’m uber busy at the moment but this update and new set of data is going to take some time to digest and turn into graphs.
Will probably do that across the weekend, but here is what they had to say-
First of all, thank you, Matt, for such a high-quality approach to comparison and data verification.
Really, there is nothing surprising in the fact that Ahrefs won this test. At the moment, our index is 25% larger in size and our crawler runs 2-3 times faster. With this initial parameters under our belt, Ahrefs will win any objective test held on a large sample of domains.
With regard to the comments made by Alex Chudnovsky.
1. The fact that Ahrefs counts hostnames instead of referring domains – this is not true. We count the actual domains but there are exceptions, there are domains the subdomains of which are considered as separate domains, and I can explain why.
For example, blogspot.com. On this domain, all subdomains represent someone’s blogs, that is, every subdomain has its owner who controls it. In fact, subdomains of blogspot.com do not differ much from regular domains in the .com zone.
Large quantity of ref domains is treated by search engines as a good sign – but why? Because they are more difficult to obtain! But if we are talking about blogspot, it is difficult to get links from different subdomains over there.
In other words, we believe that Google and other search engines treat every blog on blogspot.com as a separate domain, because it is reasonable to do so. Therefore, we also show each blog as a separate domain. Due to this, on the Referring Domains tab at Ahrefs you can see every blog separately instead of one and the same blogspot.com which renders little information. It is important whether it is 1, 10, or 1000 blogs that links to you.
Other examples of services which we consider to be an exception: livejournal.com, wordpress.com, tumblr.com, webs.com.
The fact that we do not put all domains in the same row blindly, that we look at their features and try to be closer to the interpretation of a domain similar to the one by search engines – I think it’s our advantage.
2. Speaking about the cost of data of $10,000. I wonder how much it costs to users to make 1,000,000 API requests to get these stats from MajesticSEO? I mean the API which allows to further distribute the data. I did not find any prices on the website. Maybe someone knows and can drop me a line about it in the comments?
3. Number of IPs and referring domains. Thanks to your comments, we now have found a couple of small bugs in the calculation of IPs. The first one – the old IPs were not always disposed, and the second one – the query for the number of referring domains for popular domains could fail with a timeout (this is what happened to Google.com). The problems have been fixed already, as you can see – one of them worked in favor of Ahrefs, the second one – in favor of Majestic SEO. I used the scripts that Matt shared to get the stats again and to check whether those issues were significant.
By IP Ahrefs wins 70% domains Majestic SEO 30%
By Subnets Ahrefs wins 63% domains Majestic SEO 37%
As I mentioned in the beginning, Ahrefs has an objectively larger index and a faster crawler, so it is not surprising at all that we will win any test; minor errors will not change the overall picture.
I will send Matt the archive with these files so anyone could double-check the data.
By the way, thank you guys for your attention to these discrepancies. This is cool to get a bug report from the competitors, I do not think that we could have gained it under any other conditions :)
I do not want to be in debt and, in turn, will tell you about a few situations when 2 +2 = 5 when it comes to domains and IPs.
There are at least two reasons why there can be more Referring IPs than Referring Domains.
The first case:
The same domain can have multiple IP addresses. Usually this is done to spread the load. More information: http://en.wikipedia.org/wiki/Round-robin_DNS
The second case:
Let’s suppose, X is the domain for which there are only three backlinks from these pages:
http://subdomain1.domain.com/index.html is located on the server with IP 188.8.131.52
http://subdomain2.domain.com/index.html is located on the server with IP 184.108.40.206
http://subdomain3.domain.com/index.html is located on the server with IP 220.127.116.11
It turns out that Domain X has only one referring domain (domain.com) but 3 referring IPs (18.104.22.168, 22.214.171.124, 126.96.36.199)
In any case, I am glad that you have started to actively improve your index.
We enjoy competition and look forward to having a strong opponent in this field.
A very interesting response that directly deals with all of the issues raised.
It seems that Majestic did uncover some flaws with the Ahrefs data and highlight 2 bugs that caused data issues but these are resolved within 24 hours of them being identified.
This hasn’t had much of an impact for Majestic though, previously Ahrefs won on 71% of domains by IP which is now reduced to 70%.
Wins on subnet were previously 64% and is now residing at 63%!
I haven’t had time to verify the new data yet but if you want to do so you can download it here.
Update #3 – Majestic Goes All Out
Majestic have now published a full response on their blog.
It is pretty hard to digest so stick with it but it does touch on some good points.
Our full response to the points raised in your original piece can be found
on our blog.
I hope this serves as a useful summary of our position for your readership:
We welcome comparisons between aspects of our product offering and those of
our competitors. We believe that the best comparisons will follow the model
set by the SeoMoz Open Site Explorer / Majestic SEO shoot out ( link to
http://www.stonetemple.com/majestic-seo-open-site-explorer-shootout/ ) –
comparing various aspects of the product offering in order to achieve a
balanced review. Naturally, different people have different needs from their
link intelligence toolset, and we respect that for some, regardless of data
size and quality, Ahrefs may be a better match for their particular needs at
We believe that the evidence from Ahrefs – which we first became aware of
from your blog, for which we are grateful, demonstrates that they calculate
IP, and Domain counts in a manner that we believe exaggerates the size of
their index, certainly when used in comparison of size against our Fresh
Index. We also present evidence that suggests that our Fresh Index is likely
to be larger than Ahrefs – based on an update of the test you described
after we fixed a bug in our system. We haven’t tested our Historic index as
we are confident it would “win” by a significant margin.
Both Ahrefs and Majestic have sufficiently large index’s ( in terms of our
“Fresh” product ) that we do not feel that size alone provides a realistic
comparison. Comparisons of crawl rate are also meaningless – our crawl has
undergone a massive upgrade – and is far more intelligent, crawling sites
more sensitively than ever before, whilst achieving a deeper crawl rate. We
don’t want our bots crashing around like a bull in the china shop – it isn’t
good for us and it isn’t good for the industry as a whole. That’s why we
have invested in innovating with tools like flow metrics, and link profiles,
and continue to do so.
I’ll close with the same conclusion as in our post – Competition is good, it
serves the market well, and forces all involved to invest in innovation. The
real winners in this battle for market share are the end users. It’s an
exciting time to be an SEO.
Majestic SEO Managing Director
What do you think? Have we got an answer to the core question of which one is best yet?