Bot Analytics – Preventing Traffic Distortion
One of the emerging challenges for digital analysts in an AI world is bots. Bots are automated, non-human browsers hitting your web site, and so distorting traffic, engagement, and conversion metrics, because they are not real customers. So, bot analytics is an important, “re-emerging” trend for digital analytics.
Why This Matters Now
While this is not a new thing, and managing bot traffic has been an issue for a long time, the growth of AI means this is getting worse. According to a recent report by the analytics software company Piwik, up to 90% – yes, 90% – of Facebook Ads traffic was bot traffic, and Google and Bing also had significant issues. So, this type of fraud is having a direct impact on ad spend.
AI means it’s easier for spammers and ad fraudsters to spin up a new instance or IP address, and so to get round any detection mechanisms one might have put in place. This is emphasised by the fact that even the big AI Search companies, like Perplexity, are using these same masking techniques by using unofficial IP addresses, and even accessing paid-for content, without paying, by ignoring the robots.txt convention. For more on this, see this article.
This growing complexity around bots and reporting is also something we are seeing with our clients for whom we provide analytics and insight services. A common scenario is the sudden emergence of a spike in traffic, often from a different or unusual country or location. A few days later, the traffic has disappeared, either because it’s achieved its goals, or simply to avoid detection, as the bot traffickers set up a new IP addresses. Whilst this may be short-term, it has a significant impact on key traffic figures in that period, and so distorts important KPIs. Moreover, taken altogether, these seemingly one-off spikes add up. Over time, these can start to erode trust in your data, and also take additional resources to spot, analyse, block and discount from data.
Why Bot Analytics Matters To Digital Analysts
As a result, you may say that the perfect solution to an AI-created problem is an AI system to combat it. And certainly, all of the analytics vendors have bot detection systems in place. However, they are often not dynamic, and are in reality not much more than an externally-managed black list. This creates two practical issues for data analysts. Firstly, these lists only update relatively slowly, and certainly only once they can identify that the user agent or IP addresses should be blocked. By which time, of course, the spike has gone. So, the traffickers are always one step ahead.
Which means that this ultimately remains a human, analytical challenge. And it has a big impact on understanding your digital and marketing activities.
Understanding your digital marketing investment
Was yesterday’s spike down to the great work that the marketing campaign has done? Does that mean we should be investing more into that channel?
Or is it bots?
Is that all lower-converting traffic we are seeing?
Or is it down to bots?
And if it is bots, should be we getting money back?
Bot Analytics – What To Look For
And what’s interesting is that you can start to understand broader patterns and analyses around these trends that help you assess them more effectively and insightfully. Did this new spike also behave in a way that looks distinctly less human? Was it hitting lots of pages in a very short space of time? Was the same URL suddenly getting a lot of hits from new visitors, but new visitors who all had suspiciously similar profiles? Indeed, it puts me in mind of some of the classic codebreaking techniques.
Because it’s not just down to IP address. Once you have identified a certain set of suspicious traffic, you can look at other factors. So, monitor resolution, operating system, device type become particularly important characteristics for this. Yes, the ones we used to report on 20 years ago, because we didn’t have much else to go on, are suddenly becoming more important again.
So, we see the management of hyper-agile bot traffic as a re-emerging trend for 2025. If you are having issues with bot traffic, or perhaps more importantly, if you aren’t aware you have a bot traffic problem (because you almost certainly do!), please get in touch and we would be happy to discuss how you can address these challenges.