Is the Ahrefsbot giving your site trouble?
Ahrefsbot crawls your website regularly. It acts just the same as Googlebot – coming and going as it pleases.
This isn’t usually a problem but sometimes it can cause problems especially if it’s overloading your server and slowing your site down.
The good news?
You can quickly and easily block Ahrefsbot from visiting your website. You just need to choose the right method that works best for you.
In this article, I’m going to share everything you need to know about Ahrefsbot and the 2 best methods to block it if you need to.
What is Ahrefsbot?
Over the years Ahrefsbot has collected and added over 12+ trillion active links to the Ahrefs database. You can see why Ahrefsbot is an integral part of the Ahrefs tool.
How Does Ahrefsbot Work?
Ahrefsbot crawls as many websites as possible. It collects a range of quality link data on each crawl and sends that data back to the Ahrefs database.
Ahrefs can display accurate data to their users, providing detailed information for marketing and optimising websites.
According to Ahrefs, Ahrefsbot…
- Crawls over 6 billion web pages every 24 hours.
- Updates Ahrefs index of links every 15 – 30 minutes.
That’s a lot of power!
Ahrefsbot: A Good or Bad Robot?
There are a number of bots used online.
The four most common types are:
All of these bots power different applications and make them more useful for users. These are called ‘good‘ bots.
But there are also ‘bad‘ bots out there. These bots do nasty things like:
- Credential stuffing
- Content scraping
- Spam traffic
They can also use up tons of your server bandwidth and cause a lot of havoc on your site.
While Ahrefsbot is doing good things most of the time – not everything it does is perfect. There is also a potential that it could be used for bad.
Ahrefsbot Offers A Few Benefits…
Ahrefsbot crawls your website and analyses both:
- Internal links.
- External links.
This data is added to the Ahrefs ever-growing database. But what are the benefits of letting Ahrefs crawl your website?
Practical SEO Data You Can Use
The Ahrefsbot data offers a huge amount of value to the SEO and marketing community.
The data that it collects allows SEO’s and marketers to understand how to better optimise their websites for large search engines like Google.
Allowing Ahrefsbot to crawl your website ensures that it is always up to date in their database. You get the most accurate data on your own website and you can even compare it to your competitors.
This helps you:
- Optimise your own website.
- Prioritise what you need to focus on to rank higher.
…But It Can Also Cause Several Issues To Your Site
As with anything, there are a few potential downsides to Ahrefsbot that you need to be aware of.
Each of these can cause unwanted issues on your site that may not be worth the risk.
Ahrefsbot Can Make Your Site Slower
No one likes a slow loading website, right?
Whenever Ahrefsbot crawls your site it loads like it would for any other person which uses up bandwidth.
If Ahrefsbot is crawling your website too regularly, it can take up too much bandwidth which means the site will load slower for other visitors.
We all know that speed is a Google ranking factor and affects user experience negatively.
Does Ahrefsbot Really Respect robot.txt?
Ahrefs is very strong on their claim that Ahrefsbot respects robots.txt. They have said multiple times that Ahrefsbot will follow both disallow and allow rules.
Here’s the problem:
There have been a few users that have said this isn’t true. Even after adding robots.txt to their sites, they notice they are receiving traffic from the Ahrefsbot.
Not obeying robots.txt rules is a sign of a bad bot. Red flag!
Ahrefsbot Can Be Used By Spammers
I hate spammers too. Unfortunately, they do exist and as website owners, they are something we need to be aware of.
There is nothing worse than receiving tons of unwanted spammy referral traffic that not only clogs up your Google analytics report but also affects your website performance.
Spammers can immitate the Ahrefsbot user agent to:
- Continuously spam your website.
- Send fake referral traffic.
- Scrape your content or data.
If you notice you are receiving tons of traffic regularly from Ahrefsbot, you might want to review and compare with the official Ahrefs IP Address ranges.
GROWTH HACK: Sign up for my free video training series and get a copy of our internal private SEO tool.
How to Block Ahrefsbot?
You can block or limit AhrefsBot using your robots.txt file or htaccess file. Both methods should work but take a look at each option below to see which works best for you.
Method 1: Block Ahrefsbot With robots.txt
Ahrefs says that Ahrefsbot follows robots.txt rules. You can block Ahrefsbot by adding new rules to your robots.txt file.
You have two options here:
- Change the frequency Ahrefsbot can visit your site
- Completely block Ahrefsbot from visiting your site
To change the frequency of Ahrefsbot visiting your site, add the following code to your robots.txt file:
Crawl-Delay value is the time of the delay recorded in seconds. You will need to change this value to delay the number of seconds between each crawl from Ahrefsbot.
Want to block Ahrefsbot completely?
To do that you will need to add a disallow tag to your robots.txt file. Simply copy and paste the code below:
That’s it! Ahrefsbot should obey the robots.txt rule you just set up.
Method 2: Block Ahrefsbot Using the .htaccess file
This is a very effective method for blocking Ahrefsbot and gives you more control.
Unlike method 1 where you rely on the fact that the Ahrefsbot will obey the rules you set – with method 2, you will be taking complete control at the server level.
You will be adding the range of IP addresses that the Ahrefsbot uses and blocking it based on those IP addresses.
Here is a small example of the code you could add to your .htaccess file to block IP addresses:
Deny from 18.104.22.168
Deny from 22.214.171.124
Deny from 195.154.122
Allow from all
You want to make sure you block all of the current Ahrefs IP addresses that are listed here. That means you will need to edit the code above to make sure it blocks the current IP’s used by Ahrefs.
Follow this tutorial if you need help with your .htaccess file.
Should You Block Ahrefsbot?
This is a good question.
Using the Ahrefs SEO tools?
Then don’t block the Ahrefsbot because you want to ensure your website is being updated in the Ahrefs index.
Don’t use the Ahrefs bot and it’s causing more harm than good?
Then it might be best to block it.
Ultimately – this question will help you decide on the Ahrefsbot: “Does this benefit my website or does it hurt my website?” and you can only figure that out by looking at yours sites access logs.
Wrapping It Up
Ahrefsbot is the second most powerful SEO bot on the planet.
Ahrefsbot can in some cases, cause some issues and even worse be used by spammers that refer unwanted traffic to your site.
Use any of the 2 methods to block the Ahrefsbot if you want to and add a layer of protection for your website and don’t forget to checkout my full Ahrefs review.