Did you know that nearly half of your website visitors might not be human? This surge of automated visits is far from harmless. AI bots are flooding websites, devouring resources that can slow down or crash your server, completely falsifying your traffic statistics, and stealing your proprietary content without permission. And if you rely on advertising, your revenue could plummet.
Does this mean you should shut your website out of the market for all AI? Not necessarily. To successfully protect yourself, you must first understand that not all bots behave the same way, and that there are key strategies to stop the ones that are harming you without losing visibility in the future of the internet.
Below, we explain how to differentiate between the types of bots and what practical steps you can take today to maintain control of your website.
Problem: AI bots
According to various studies, between 30% and 50% of internet traffic comes from bots. With the advancement of AI, this is expected to increase. Cloudflare reports that bot traffic has risen by approximately 30% in just one year.
Cloudflare’s Radar tool allows us to view some real-time statistics: https://radar.cloudflare.com/bots?dateRange=52w. These visits cause problems on several levels for websites with high bot traffic:
- Server impact: It can severely affect server performance and, therefore, service availability. AI crawlers make thousands of visits in a short time and consume significant resources (CPU, RAM, and bandwidth), in some cases even crashing the server. This could increase hosting costs.
- Metric distortion: Usage statistics such as visits, session duration, countries, etc., can be distorted by bot behavior.
- Content consumption: Proprietary content can be consumed by these bots, which can lead to plagiarism, price theft, etc.
- Loss of revenue: Websites that rely on advertising may experience a significant drop in monetization (this is a growing trend).
Types of AI bots
Before exploring potential solutions, it’s necessary to differentiate between two types of bots:
- AI Search: A bot that uses specific AI to find content through a direct internet search. This occurs, for example, when ChatGPT displays “Searching the internet”:
In these cases, whether a user finds our website before another on ChatGPT will depend on traditional SEO.
- AI Crawler: An AI bot used to train its Language Model (LLM).
Solutions
- Ignore: On sites with low traffic or content that is not relevant for training, the impact is usually minimal. In these cases, we can ignore AI bots, as they won’t cause any harm. However, limiting access, as we’ll see in the next section, doesn’t cost much and allows us to prevent future problems.
- Restrict: We can restrict bot access using the following strategy: block crawlers, allow searchers. This ensures that if a user types a prompt that triggers an active search by the AI (AI Search), they can find us without issue, but we prevent the AI from using us to train its model (AI Crawler). For most websites, preventing AI models from being trained with their content has virtually no impact, as their individual contribution to the dataset is insignificant compared to the total volume used to train these models. This restriction requires recurring monitoring since new bots are constantly appearing.
- Block everything: In applications such as intranets, private areas, or internal applications, we can block all bots without any disadvantages. This is the default behavior implemented by Cloudflare. Blocking or limiting bots can be done in several ways (firewall, robots.txt, and others). Ideally, Pukka would use Cloudflare, which has an area where you can block specific bots, or all of them if necessary.
Final notes
The method by which AI is trained using publicly available information is currently a subject of debate, and there is no official legislation governing it at present. Large corporations have blocked access to AI, or have granted it only through lucrative deals. The solutions and information presented in this document are likely to change in the near future, and the only way to ensure proper action is to stay informed and actively monitor the situation.
