Default blocking of aggressive bots

A significant proportion of the traffic on many websites comes not from real visitors, but from automated crawlers (bots): SEO analysis tools, robots collecting data to train AI models, foreign search engines and simple scrapers. Some of these crawl every subpage of the site without restriction, often making several requests per second, which slows down the website for genuine visitors and unnecessarily consumes the hosting resource allocation.

For this reason, on the servers hosting our cPanel and WordPress hosting plans, the web server by default returns a 403 Forbidden error to requests whose User-Agent header (the identifier sent by the client) refers to a bot on a blocklist. This rule is configured in the server settings and applies to every website on the hosting plan; you do not need to configure anything yourself. However, if you do wish to allow a particular bot through, you can make an exception in the .htaccess file (see below).

Which bots are prohibited by the rules?

The block list currently contains bots categorised into the following groups:

SEO, marketing and business data collection bots: AhrefsBot and AhrefsSiteAudit, SemrushBot, MJ12bot (Majestic), DotBot (Moz), BLEXBot, BarkRowler, serpstatbot, DataForSeoBot, MegaIndex.ru, RSiteAuditor, seocompany.store, Ezooms, SentiBot, MetaURI, ZoominfoBot, VelenPublicWebCrawler, Dataprovider.com, AwarioBot, IonCrawl, GrapeshotCrawler, CriteoBot, BitSightBot, WellKnownBot, Nimbostratus-Bot, Polaris, brokenlinkcheck.com. They collect data for these external services (backlink analysers, competitor trackers, advertising and company databases). They do not bring visitors to the site, but they crawl it regularly and in great depth.
Bots used to train AI models and collect large volumes of content: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), Amazonbot (Amazon), ImagesiftBot, FriendlyCrawler, GeedoProductSearch. The content of websites is collected either to train large language models or to populate their own databases.
Foreign and smaller search engines: Baiduspider (China), YandexBot (Russia), SeznamBot (Czech Republic), PetalBot and its predecessor, AspiegelBot (Huawei), SeekportBot, Applebot (Siri, Spotlight and Safari suggestions, as well as Apple’s AI models). These belong to legitimate search engines, but typically bring few visitors to a site aimed at a Hungarian audience.
Web archiving bots and crawler frameworks: OSZKBOT (the Hungarian Web Archive of the National Széchényi Library), Archive Team, Nutch (an open-source crawler framework on which many custom bots are based).
HTTP client libraries: python-requests, Go-http-client, axios, GuzzleHttp, fasthttp, GRequests. These are not specific bots, but the default User-Agent values of programming libraries. Browsers never send such values, so in most cases they are used by unidentified scripts, vulnerability scanners or content-stealing programs.
Scrapers disguised as browsers: a few outdated browser identifiers (such as Firefox 62 running on Ubuntu or Firefox 95 running on Linux) used by known scraper networks. Modern browsers do not send such identifiers.

What is not prohibited?

This rule does not apply to Google’s bots (Googlebot, AdsBot-Google), Bing’s and DuckDuckGo’s bots, social media sites’ link-generating bots (Facebook, LinkedIn, X), bots that open a page in response to a user’s query or index pages for AI search engines (ChatGPT-User, OAI-SearchBot, Claude-SearchBot, PerplexityBot), the Internet Archive’s bot, the curl and wget programs, WordPress’s own requests and most uptime monitors (such as UptimeRobot).

Why do we block these bots?

Low load: some bots do not comply with the robots.txt rules and may send several hundred requests per minute. This places a load on the CPU and database, which can lead to a 508 error when the hosting resource limit is reached.
Lower traffic: for websites containing images, videos or numerous subpages, traffic from scrapers can be significant.
Security: Unidentified HTTP clients are often behind automated attack attempts, vulnerability scans or content theft.
Content protection: by blocking AI training bots, the website’s content is prevented from being used as training data for large language models without authorisation.

When might an exception be required?

You may need to use a service that utilises a blocked User-Agent, for example:

analyses their own website using Ahrefs or Semrush, and their crawler is unable to crawl the site;
Does your website also target the Russian, Czech or Chinese markets (Yandex, Seznam, Baidu), or is it important for you to appear in Apple’s search engines (Applebot)?
A payment provider’s callback (IPN), a webhook, an API client or your own script accesses the website and uses an HTTP library written in PHP (Guzzle), Node.js (axios), Python or Go.

If such a service returns a 403 error, the ‘client denied by server configuration’ line associated with the request in the cPanel Dashboard » Errors menu indicates that the server has rejected it.

Exceptions in the .htaccess file

The restriction can be overridden in the .htaccess file located in the website’s root directory, which you can edit via the File Manager menu in cPanel (you can enable the display of hidden files via the Settings button).

It is important to note that the exception only applies within an <If> block: the server rule is evaluated after the standard Require lines in the .htaccess file; thus, for example, a standalone Require all granted line does not override the restriction. The <If> block in the .htaccess file, however, follows the server rule and therefore overrides it.

For WordPress, place the exception outside the section between the # BEGIN WordPress and # END WordPress lines, as WordPress rewrites that section when a backup is taken.

Exceptions do not affect server rules protecting .env files and .git directories; these remain blocked.

Fully enabling a bot

If you wish to allow a particular bot access from all locations, enter its name (or multiple names separated by a |) in the rule:

<If "%{HTTP_USER_AGENT} =~ /(AhrefsSiteAudit|SemrushBot)/i">
    Require all granted
</If>

Anyone can set the User-Agent header as they wish, so any program that pretends to be this bot can gain access using this setting. If the service is accessed from known IP addresses, it is safer to use the IP-based exception below.

Authorisation based on IP address

If a service using a blocked User-Agent (such as a payment provider, monitoring system or your own server) is accessing the service from known IP addresses, only allow traffic from these addresses:

<If "%{HTTP_USER_AGENT} =~ /(GuzzleHttp|python-requests|axios|Go-http-client)/i">
    <RequireAny>
        Require all denied
        Require ip 203.0.113.42
        Require ip 198.51.100.0/24
    </RequireAny>
</If>

Within the <RequireAny> block, it is sufficient for any of the Require lines to be met. The Require all denied line is never met, whereas the Require ip line is met if the request comes from one of the listed addresses. Thus:

Requests originating from the listed IP addresses will still get through even if their User-Agent is on the blocklist;
Any other requests arriving with a prohibited User-Agent will continue to receive a 403 error.

In the Require ip line, you can specify either a single IP address (203.0.113.42) or a CIDR-notated range (198.51.100.0/24), for both IPv4 and IPv6, in any number of lines. If the website is hosted behind Cloudflare, you must still enter the visitor’s actual IP address; the server will retrieve this from the Cloudflare header.

Authorisation based on a secret header

If the client’s IP address is not static (for example, in the case of a CI system or an internal testing tool), a unique HTTP header can be used instead of or alongside the IP address:

SetEnvIf X-Bot-Bypass "^titkos-token-ide$" bot_bypass

<If "%{HTTP_USER_AGENT} =~ /(python-requests|GuzzleHttp)/i">
    <RequireAny>
        Require all denied
        Require env bot_bypass
    </RequireAny>
</If>

In this case, the client must include the header X-Bot-Bypass: titkos-token-ide in every request. As the token appears in plain text in the .htaccess file, this is not a genuine security solution, but merely a simple workaround, for example for testing purposes. Please use a long, random value.

Add your own block

If you wish to block a bot that is not on the list, you can do so in the same way using an <If> block:

<If "%{HTTP_USER_AGENT} =~ /(meta-externalagent|PerplexityBot)/i">
    Require all denied
</If>

For bots that comply with the robots.txt rules (such as those used by major search engines and AI companies), the block specified in the robots.txt file is sufficient.

Testing the rule

After making the change, you can use the curl command to check whether the block and the exception are working as expected:

# 403-as választ kell kapnia
curl -I -A "Mozilla/5.0 (compatible; BLEXBot/1.0)" https://pelda-domain.hu/

# 200-as választ kell kapnia (normál böngésző User-Agent)
curl -I -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" https://pelda-domain.hu/

# Fejléc alapú kivételnél a fejléccel 200-as választ kell kapnia
curl -I -A "python-requests/2.32" -H "X-Bot-Bypass: titkos-token-ide" https://pelda-domain.hu/

In the case of IP-based exceptions, a request sent with a prohibited User-Agent must return a 200 response if sent from an IP address on the exception list, and a 403 response if sent from any other address.

If a service continues to return a 403 error even after the exception has been handled, it may be that it is not this rule but the server’s firewall that is blocking it (see: Firewall IP blocking). In this case, please contact us and we will help you identify the cause.