Google is aware of about 300T pages on the internet. It’s uncertain they crawl all of these, and a minimum of in response to some paperwork from their antitrust trial we discovered they solely listed 400B. That’s round .133% of the pages they find out about, roughly 1 out of each 752 pages.
For Ahrefs, we select to retailer about 340B pages in our index as of December 2023.
At a sure level, the standard of the online turns into dangerous. There are many spam and junk pages that simply add noise to the information with out including any worth to the index.
Massive components of the online are additionally duplicate content material, ~60% according to Google’s Gary Illyes. Most of that is technical duplication brought on by completely different programs. Nonetheless, should you don’t account for this duplication, it may waste extra sources and create extra noise within the information.
When constructing an index of the online, corporations need to make many selections round crawling, parsing, and indexing information. Whereas there’s going to be quite a lot of overlap between indexes, there’s additionally going to be some variations relying on every firm’s selections.
Evaluating hyperlink indexes is tough due to all of the completely different selections the varied instruments have made. I strive my finest to make some comparisons extra truthful, however even for a number of websites I’m telling you that I don’t wish to put in the entire work wanted to make an correct comparability, a lot much less do it for a complete research. You’ll see why I say this later if you learn what it will take to check the information precisely.
Nonetheless, I did run some checks on a pattern of web sites and I’ll present you find out how to test the information your self. I additionally pulled some pretty massive third celebration information samples for some further validation.
Let’s dive in.
For those who simply checked out dashboard numbers for hyperlinks and RDs in several instruments you may see fully various things.
For instance, right here’s what we rely in Ahrefs:
- Stay hyperlinks
- Stay RDs
- 6 months of information
In Semrush, right here’s what they rely:
- Stay + useless hyperlinks
- Stay + useless RDs
- 6 months of knowledge + a bit extra*
*By a bit extra, what I imply is that their information goes again 6 months and to the beginning of the earlier month. So, for example, if it’s the fifteenth of the month, they might even have about 6.5 months of knowledge as an alternative of 6 months of knowledge. If it’s the final week of the month, they might have near 7 months of knowledge as an alternative of 6.
This will not appear to be so much, however it may enhance the numbers proven by so much, particularly if you’re nonetheless counting useless hyperlinks and useless RDs.
I don’t suppose SEOs wish to see a quantity that features useless hyperlinks. I don’t see a very good motive to rely them, both, apart from to have larger and doubtlessly deceptive numbers.
I solely say this as a result of I’ve referred to as Semrush out on making the sort of biased comparability earlier than on Twitter, however I ended arguing once I realized that they actually didn’t need the comparability to be truthful; they only wished to win the comparability.
There are some methods you’ll be able to examine the information to get considerably related time durations and solely take a look at energetic hyperlinks.
For those who filter the Semrush backlinks report for “Lively” hyperlinks, you’ll have a considerably extra correct quantity to check in opposition to the Ahrefs dashboard quantity.
Alternatively, should you use the “Present historical past: Final 6 months” choice within the Ahrefs backlink report, this would come with misplaced hyperlinks and be a fairer comparability to Semrush’s dashboard quantity.
Right here’s an instance of find out how to get extra related information:
- Semrush Dashboard: 5.1K = Ahrefs (6-month date comparability): 5.6K
- Semrush All Hyperlinks: 5.1K = Ahrefs (6-month date comparability): 5.6K
- Semrush Lively Hyperlinks: 2.9K = Ahrefs Dashboard: 3.5K = Ahrefs (no date comparability): 3.5K
What you shouldn’t examine is Semrush Dashboard and Ahrefs Dashboard numbers. The quantity in Semrush (5.1K) consists of useless hyperlinks. The quantity in Ahrefs (3.5K) doesn’t; it’s solely reside hyperlinks!
Observe that the time durations will not be precisely the identical as talked about earlier than due to the additional days within the Semrush information. You could possibly take a look at what day their information stops and choose that precise day within the Ahrefs information to get an much more correct, however nonetheless not fairly correct comparability.
I don’t suppose the comparability works in any respect with bigger domains due to a problem in Semrush. Right here’s what I noticed for semrush.com:
- Semrush Dashboard: 48.7M = Ahrefs (6 month date comparability): 24.7M
- Semrush All Hyperlinks: 48.7M = Ahrefs (6 month date comparability): 24.7M
- Semrush Lively Hyperlinks: 1.8M = Ahrefs Dashboard: 15.9M = Ahrefs (no date comparability): 15.9M
In order that’s 1.8M energetic hyperlinks in Semrush vs 15.9M energetic in Ahrefs. However as I stated, I don’t suppose it is a truthful comparability. Semrush appears to have a problem with bigger websites. There’s a warning in Semrush that claims, “As a result of measurement of the analyzed area, solely probably the most related hyperlinks can be proven.” It’s potential they’re not exhibiting all of the hyperlinks, however that is suspicious as a result of they may present the full for all hyperlinks which is a bigger quantity, and I can filter these in different methods.
I may type usually by the oldest final seen date and see all of the hyperlinks, however once I do final seen + energetic, I see solely 608K hyperlinks. I can’t get greater than 50k rows of their system to analyze this additional, however one thing is fishy right here.
Extra hyperlink variations
The above comparability wouldn’t be sufficient to make an correct comparability. There are nonetheless plenty of variations and issues that make any type of comparability troublesome.
This tweet is as related because the day I wrote it:
It’s nearly unimaginable to do a good hyperlink comparability
Right here’s how we count links, nevertheless it’s price mentioning that every software counts hyperlinks in several methods.
To recap a few of the details, listed below are some issues we do:
- We retailer some hyperlinks inserted with JavaScript, nobody else does this. We render ~250M pages a day.
- Now we have a canonicalization system in place that others might not, which implies we shouldn’t rely as many duplicates as others do.
- Our crawler tries to be clever about what to prioritize for crawling to keep away from spam and issues like infinite crawl paths.
- We rely one hyperlink per web page, others might rely a number of hyperlinks per web page.
These variations make a good hyperlink comparability practically unimaginable to do.
The way to see the place the most important hyperlink variations are
The best approach to see the most important discrepancies in hyperlink totals is to go to the Referring Domains stories within the instruments and type by the variety of hyperlinks. You should use the dropdowns to see what sorts of points every index might have with overcounting some hyperlinks. In lots of instances, you’re prone to see tens of millions of hyperlinks from the identical website for a few of the causes talked about above.
For instance, once I appeared in Semrush I discovered blogspot hyperlinks that they claimed to have just lately checked, however these are exhibiting 404 once I go to them. Semrush nonetheless counts them for some motive. I noticed this challenge on a number of domains I checked. That is a kind of pages:
A number of hyperlinks counted as reside are literally useless
Seeing the useless hyperlink above counted within the complete made me wish to test what number of useless hyperlinks have been in every index. I ran crawls on the checklist of the latest reside hyperlinks in every software to see what number of have been really nonetheless reside.
For Semrush, 49.6% of the hyperlinks they stated have been reside have been really useless. Some churn is predicted as the online adjustments, however half the hyperlinks in 6 months signifies that quite a lot of these could also be on the spammier a part of the online that isn’t as steady or they’re not re-crawling the hyperlinks typically. For some context, the identical quantity for Ahrefs got here again as 17.2% useless.
It’s going to get extra sophisticated to check these numbers
Ahrefs just lately added a filter for “Finest hyperlinks” which you’ll configure to filter out noise. As an illustration, if you wish to take away all blogspot.com blogs from the report, you’ll be able to add a filter for it.
This implies you’ll solely see hyperlinks you take into account vital within the stories. This can be utilized to the primary dashboard numbers and charts now. If the filter is energetic, folks will see completely different numbers relying on their settings.
You’ll suppose that is simple, nevertheless it’s not.
Fixing for all the problems is quite a lot of work
There are quite a lot of completely different stuff you’d have to unravel for right here:
- The additional days in Semrush’s information that you simply’ll need to take away or add to the Ahrefs quantity.
- Keep in mind that Semrush additionally consists of useless RDs of their dashboard numbers. So you could filter their RD report to only “Lively” to get the reside ones.
- Keep in mind that half the hyperlinks within the check of Semrush reside information have been really useless, so I’d suspect that plenty of the RDs are literally misplaced as properly. You could possibly probably search for domains with low hyperlink counts and simply crawl the listed hyperlinks from these to take away a lot of the useless ones.
- In any case that, you’re nonetheless going to wish to strip the domains all the way down to the basis area solely to account for the variations in what every software could also be counting as a site.
What’s a site?
Ahrefs at the moment reveals 206.3M RDs in our database and Semrush reveals 1.6B. Domains are being counted in extraordinarily alternative ways between the instruments.
In line with the foremost sources who take a look at these sorts of issues, the variety of domains on the web appears to be between 269M–359M and the variety of web sites between 1.1B–1.5B, with 191M–200M of them being energetic.
Semrush’s variety of RDs is greater than the variety of domains that exist.
I imagine Semrush could also be complicated completely different phrases. Their numbers match pretty intently with the variety of web sites on the web, however that’s not the identical because the variety of domains. Plus, lots of these web sites aren’t even reside.
It’s going to get extra sophisticated to check these numbers
A part of our course of is dropping spam domains, and we additionally deal with some subdomains as completely different domains. We come up near the numbers from different third celebration research for the variety of energetic web sites and domains, whereas Semrush appears to come back in nearer to the full variety of web sites (together with inactive ones).
We’re going to simplify our methodology quickly in order that one area is definitely only one area. That is going to make our RD numbers go down, however be extra correct to what folks really take into account a site. It’s additionally going to make for an excellent larger disparity within the numbers between the instruments.
I ran some high quality checks for each the first-seen and last-seen hyperlink information. On each website I checked, Ahrefs picked up extra hyperlinks first and up to date the hyperlinks extra just lately than Semrush. Don’t simply imagine me, although; test for your self.
Evaluating that is biased irrespective of the way you take a look at it as a result of our information is extra granular and consists of the hours and minutes as an alternative of simply the day. Leaving the hours and minutes creates a biased comparability, and so does eradicating it. You’ll need to match the URLs and test which date is first or if there’s a tie after which rely the totals. There can be some completely different hyperlinks in every dataset, so that you’ll have to do the lookups on every set of knowledge for comparability.
Semrush declare,s “We replace the backlinks information within the interface each quarter-hour.”
Ahrefs claims, “The world’s largest index of reside backlinks, up to date with contemporary information each 15–half-hour.”
I pulled information on the identical time from each instruments to see when the newest hyperlinks for some common web sites have been discovered. Right here’s a abstract desk:
| Area | Ahrefs Newest | Semrush newest |
|---|---|---|
| semrush.com | 3 minutes in the past | 7 days in the past |
| ahrefs.com | 2 minutes in the past | 5 days in the past |
| hubspot.com | 0 minutes in the past | 9 days in the past |
| foxnews.com | 1 minute in the past | 12 days in the past |
| cnn.com | 0 minutes in the past | 13 days in the past |
| amazon.com | 0 minutes in the past | 6 days in the past |
That doesn’t appear contemporary in any respect. Their 15-minute replace declare appears fairly doubtful to me with so many web sites not having updates for a lot of days.
Don’t simply belief me, although; I encourage you to test some web sites your self. Go into the backlinks stories in each instruments and type by final seen. Remember to share your outcomes on social media.
Ahrefs now receives information from IndexNow
This may make our information even brisker. That’s ~2.5B URLs / day in March 2024. The web sites inform us about new pages, deleted pages, or any adjustments they make in order that we are able to go crawl them and replace the information. Learn extra here.
Ahrefs crawls 7B+ pages on daily basis. Semrush claims they crawl 25B pages per day. This could be ~3.5x what Ahrefs crawls per day. The issue is that I can’t discover any proof that they crawl that quick.
We noticed that round half the hyperlinks that Semrush had marked as energetic have been really useless in comparison with about 17% in Ahrefs, which indicated to me that they might not re-crawl hyperlinks as typically. That and the freshness check each pointed to them crawling slower. I made a decision to look into it.
Logs of my websites
I checked the logs of a few of my websites and websites I’ve entry to, and I didn’t see something to assist the declare that Semrush crawls sooner. When you’ve got entry to logs of your personal website, you must be capable to test which bots are crawling the quickest.
80,000 months of log information
I used to be curious and wished to take a look at larger samples. I used Web Explorer and some completely different footprints (patterns) to seek out log file summaries produced by AWStats and Webalizer. These are sometimes revealed on the net.
I scraped and parsed ~80,000 log file summaries that contained 1 month of knowledge every and have been generated within the final couple of years. This pattern contained over 9k web sites in complete.
I didn’t see proof of Semrush crawling many instances sooner than Ahrefs for these websites, as they declare they do. The one bot that was crawling a lot sooner than Ahrefsbot on this dataset was Googlebot. Even different engines like google have been behind our crawl price.
That’s simply information from a small-ish variety of websites in comparison with the size of the online. What about for a bigger chunk of the net?
Information from 20%+ of net visitors
On the time of writing, Cloudflare Radar has Ahrefsbot because the #7 most energetic bot on the internet and Semrushbot at #40.
Whereas this isn’t an entire image of the online, it’s a reasonably large chunk. In 2021, Cloudflare was stated to handle ~20% of the web’s traffic, up from ~10% in 2018. It’s possible a lot greater now with that sort of progress. I couldn’t discover the numbers from 2021, however in early 2022 they have been dealing with 32 million HTTP requests / second on common and in early 2023 that they had already grown to dealing with 45 million HTTP requests / second on average, over 40% extra in a single yr!
Moreover, ~80% of websites that use a CDN use Cloudflare. They deal with most of the bigger websites on the internet; BuiltWith reveals that Cloudflare is used by ~32% of the Top 1M websites. That’s a big pattern measurement and sure the biggest pattern that exists.
How a lot do search engine marketing instruments crawl?
A few of the search engine marketing instruments share the variety of pages they crawl on their web sites. The one one within the chart under that doesn’t have a publicly revealed crawl price is AhrefsSiteAudit bot, however I requested our workforce to drag the data for this. Let me put the rankings in perspective with precise and claimed crawl charges.
| Rating | Bot | Crawl Charge |
|---|---|---|
| 7 | Ahrefsbot | 7B+ / day |
| 27 | DataForSEO Bot | 2B / day |
| 29 | AhrefsSiteAudit | 600M – 700M / day |
| 35 | Botify | 143.3M / day |
| 40 | Semrushbot | 25B / day* claimed |
The maths isn’t mathing. How can Semrush declare they’re crawling a number of instances as quick as these others, however their rating is decrease? Cloudflare doesn’t cowl your complete net, nevertheless it’s a big chunk of the online and a greater than consultant pattern measurement.
Once they initially made this 25B declare, I imagine they have been nearer to ninetieth on Cloudflare Radar, close to the underside of the checklist on the time. Semrush hasn’t up to date this quantity since then, and I recall a time period the place they have been within the 60s-70s on Cloudflare Radar as properly. They do appear to be getting sooner, however their claimed numbers nonetheless don’t add up.
I don’t hear SEOs raving about Moz or Sistrix having the very best hyperlink information, however they’re twenty first and thirty sixth on the checklist respectively. Each are greater than Semrush.
Attainable explanations of variations
Semrush could also be conflating the time period pages with hyperlinks, which is definitely talked about in a few of their documentation. I don’t wish to hyperlink to it, however you’ll find it with this quote: “Each day, our bot crawls over 25 billion hyperlinks”. However hyperlinks usually are not the identical factor as pages and there might be a whole bunch of hyperlinks on a single web page.
It’s additionally potential they’re crawling a portion of the online that’s simply extra spammy and isn’t mirrored within the information from both of the sources I checked out. A few of the numbers point out this can be the case.
Y’all shouldn’t belief research executed by a selected vendor when it compares them to others, even this one. I attempt to be as truthful as I might be and observe the information, however since I work at Ahrefs you’ll be able to hardly take into account me unbiased. Go take a look at the information yourselves and run your personal checks.
There are some of us within the search engine marketing neighborhood who attempt to do these checks each now and again. The final main 3rd party study was run by Matthew Woodward, who initially declared Semrush the winner, however the conclusion was modified and Ahrefs was in the end declared to be the rightful winner. What occurred?
The methodology chosen for the research closely favored Semrush and was investigated by a good friend of mine, Russ Jones, might he relaxation in peace. Right here’s what Russ needed to say about it:
Whereas companies like Majestic and Ahrefs possible retailer a single canonical IP tackle per area, SEMRush appears to retailer per hyperlink, which accounts for why there could be extra IPs that referring domains in some instances. I don’t suppose SEMRush is deliberately inflating their numbers, I believe they’re storing the information otherwise than rivals which ends up in a quantity that’s greater and doubtlessly deceptive, however not because of ailing intent.
The response from Matthew indicated that Semrush may need misled him of their favor. Right here’s that remark:
In the long run, Ahrefs gained.
Examine our present stats on our big data page.
Whereas Semrush doesn’t present present hardware stats, they did present some previously once they made adjustments to their hyperlink index.
In June 2019, they made an announcement that claimed that they had the most important index. The check from Matthew Woodward that I talked about occurred after this check, and as you noticed, Ahrefs gained that.
In June 2021, they made one other announcement about their hyperlink index that claimed they have been the most important, quickest, and finest.
These are some stats they launched on the time:
- 500 servers
- 16,128 cpu cores
- 245 TB of reminiscence
- 13.9 PB of storage
- 25B+ pages / day
- 43.8T hyperlinks
The discharge stated they elevated storage, however their earlier launch stated that they had 4000 PBs of storage. They stated the storage was 4x, so I suppose the earlier quantity was purported to be 4000 TBs and never 4000 PBs, and so they simply acquired combined up on the terminology.
I checked our numbers on the time, and that is how we matched up:
- 2400 servers (~5x larger)
- 200,000 cpu cores (~12.5x larger)
- 900 TB of reminiscence (~4x larger)
- 120 PB of storage (~9x larger)
- 7B pages / day (~3.5x much less???)
- 2.8T reside hyperlinks (I’m undecided the full measurement, however to this present day it’s not as huge because the quantity they claimed)
They have been claiming extra hyperlinks and sooner crawling with a lot much less storage and hardware. Granted, we don’t know the main points of the hardware, however we don’t run on dated tech.
They claimed to retailer extra hyperlinks than now we have even now and in much less house than we add to our system every month. It actually doesn’t make sense.
Last ideas
Don’t blindly belief the numbers on the dashboards or the final numbers as a result of they might characterize fully various things. Whereas there’s no good approach to examine the information between completely different instruments, you’ll be able to run most of the checks I confirmed to attempt to examine related issues and clear up the information. If one thing appears off, ask the software distributors for a proof.
If there ever comes a time once we cease successful on issues like tech and crawl velocity, go forward and swap to a different software and cease paying us. However till that point, I’d be extremely skeptical of any claims by different instruments.
When you’ve got questions, message me on X.