User Agent Plugin¶

Namespace: \Kanopi\Firewall\Plugins\UserAgent

Analyzes user agent strings to identify bots, devices, browsers, and operating systems.

Configuration Example¶

plugins:
  - plugin: "Kanopi\\Firewall\\Plugins\\UserAgent"
    response: block
    weight: 0
    enable: true
    config:
      # Block all bots
      - "bot:true"

      # Block specific device types
      - "device.type:desktop"
      - "device.type@in:smartphone,tablet"

      # Block specific browsers
      - "client.name:Internet Explorer"
      - "client.type:browser"
      - "client.version@less_than:10"

      # Block specific operating systems
      - "os.name:Windows XP"
      - "os.short_name:WIN"
      - "os.version@less_than:10"

      # Block specific brands or models
      - "brand:Huawei"
      - "model@contains:Galaxy"

      # Complex user agent rules
      - type: AND
        rules:
          - "bot:false"
          - "client.name:Chrome"
          - "client.version@less_than:80"

Catching automated traffic¶

bot:true is backed by matomo/device-detector's curated bot database, which has a blind spot — it does not classify a good deal of the tooling a firewall exists to stop.

automated:true is the union of that database and a broader crawler list, and catches them:

If you wrote bot:true expecting scanners to be stopped, sqlmap is getting through today. One line changes that:

- plugin: "Kanopi\\Firewall\\Plugins\\UserAgent"
  response: block
  enable: true
  config:
    - "automated:true"

It is an ordinary rule variable, so it composes like any other:

config:
  # Anything automated except your own monitoring.
  - type: AND
    rules:
      - "automated:true"
      - "!client.name@contains:StatusCake"

Why not just widen bot:¶

The broader list deliberately counts generic HTTP client libraries as automated. That is usually what a firewall wants — but if a partner integration, a mobile app, or your own monitoring runs on python-requests or Go-http-client, automated:true will block traffic that bot:true let through.

Redefining bot:true would apply that to rules people wrote long ago and have not touched. As a separate variable it is one line to opt into, and one line to leave alone. bot: keeps exactly the meaning it always had.

bot.name and the other sub-keys¶

bot.name, bot.category and bot.producer come from the curated database only. An agent that solely the wider list recognises will satisfy automated:true while exposing no name to match on — the wider list yields a matched pattern, not an identity.

Interaction with client.* rules¶

Worth knowing if you use client.name@contains:sqlmap: it keeps working alongside automated:true. Detection stops as soon as an agent is identified as a bot, and a stopped parse exposes no client at all — so the wider list is deliberately kept out of that decision. Folding it in would have silently broken exactly that rule.

Choosing the bot detection source¶

bot: is answered by device-detector's curated database by default. That is the historical behaviour and it stays the default, because widening what an existing blocking rule matches is not something a minor release should do quietly.

When you want the wider list behind bot: itself, say so:

plugins:
  - plugin: "Kanopi\\Firewall\\Plugins\\UserAgent"
    response: block
    enable: true
    metadata:
      bot_detector: both      # device-detector | crawler-detect | both
    config:
      - "bot:true"

automated: is unaffected — it is always the union of both sources, whatever bot_detector says, because that is the question it exists to answer.

The notice you may see¶

A plugin configured with bot: and no automated:, and no explicit bot_detector, logs this once when it is constructed:

firewall.NOTICE: bot: does not match sqlmap, curl, python-requests or Go-http-client —
                 automated: does. Add "automated:true" alongside it, or set
                 metadata.bot_detector to choose a source explicitly and silence this.

It is not an error and nothing is broken. It exists because the gap is otherwise invisible: the rule is valid, it fires, it simply does not know about half the tooling. Setting bot_detector explicitly — to any value, including the default — counts as having made the choice, and silences it.

What bot.name gives you under each source¶

device-detector's database carries a name, category and producer. The crawler list carries only the substring it matched. So under crawler-detect or both, a bot the curated database knows still reports all three fields, and one only the crawler list knows reports bot.name as the matched string with no category or producer. That is better than the field going empty, but do not expect bot.category to be populated for everything bot:true now matches.

Verifying the crawler is who it says¶

A user-agent rule matches on an assertion the client makes, not a fact. For a block rule that is fine: the cost of being lied to is that an attacker declines to be blocked. For an allow rule it is not, because response: allow short-circuits evaluation entirely — no block, no challenge, no rate limit. Googlebot in a header is one curl flag away.

verify: reverse-dns makes the rule prove it:

plugins:
  - plugin: "Kanopi\\Firewall\\Plugins\\UserAgent"
    response: allow
    weight: -200
    metadata:
      name: verified-search-crawlers
      verify: reverse-dns
      verify_suffixes:
        - .googlebot.com
        - .google.com
        - .search.msn.com
        - .applebot.apple.com
    config:
      - "bot:true"

The rule matches as usual, and then the match has to survive three steps — the same ones Google, Bing and Apple all document:

1. Reverse lookup the client address to a hostname.
2. Check the hostname is inside one of verify_suffixes.
3. Forward-resolve that hostname and confirm it comes back to the address it started from.

Step 3 is the one that matters. Anyone can point reverse DNS for an address they control at crawl-1-2-3-4.googlebot.com; only Google can make that name resolve back.

The lookups are PHP's own by default. A site can send them to a DNS-over-HTTPS provider instead, such as Cloudflare's or Google's, whose lookups have a time limit. That's set once in global.reverse_dns, and a rule can pick a different provider with verify_provider. A provider is a third party that receives the reverse-DNS names of visitors' addresses, so read Reverse DNS before choosing one.

Anything less than a confirmed round trip is no match¶

No PTR record, a hostname outside the list, a forward lookup that does not come back, DNS unreachable — all of them mean the rule does not match, and evaluation carries on to the rules below it.

That is the opposite of the fail-open posture that is right for a reputation source like AbuseIPDB, and deliberately so. A block source that cannot be reached should not start blocking everyone; an allow rule that cannot be verified must not start allowing everyone.

Suffixes are matched on a label boundary¶

googlebot.com accepts crawl.googlebot.com and googlebot.com itself. It does not accept evilgooglebot.com, which anyone can register. Writing the leading dot (.googlebot.com) is clearer and behaves identically.

The cost, and what bounds it¶

Read this before enabling it. Two DNS lookups is far more than a request can afford:

For scale, the firewall's entire evaluation is 3.5–5 ms. A cold verification is ~25× the cost of everything else it does.

Those figures are for PHP's own lookups. Through a DNS-over-HTTPS provider, a verification costs 19–40 ms once the connection is open (66–93 ms for the first, which includes the TLS handshake). More to the point, a provider that stops answering costs timeout_ms (about 300 ms), not the operating system's resolver's 5–10 seconds.

Five things keep that off the request path:

1. It only runs after the rule has already matched. A request matching nothing never pays anything.
2. Verdicts are cached per address, and a cached verdict costs 0.02 ms.
3. Refusals are cached far longer than acceptances (verify_negative_ttl, a day by default). Refusals are what a spoofer generates, and an address that is not Googlebot will not become Googlebot. One lookup per attacking address, then nothing.
4. Concurrent lookups for one address collapse to one. The others fail closed rather than queueing behind it — unless verify_claim_wait_ms says to wait.
5. A slow lookup trips a breaker and DNS is skipped entirely for verify_breaker_cooldown seconds.

Run a local caching resolver¶

Not a suggestion — a prerequisite, with PHP's own lookups. (With a provider, each lookup has a time limit instead, and the host's resolver is never asked.) The 112 ms cold figure drops to ~2 ms once the host's resolver has the answer, so systemd-resolved, dnsmasq or unbound on the host is what makes this affordable at all. Without one, every cache expiry is 112 ms of blocked worker.

A cold cache under concurrency refuses most of them¶

Measured with 25 processes verifying the same uncached address at the same instant (tests/Performance/bin/reverse-dns-probe.php):

The in-flight claim does its job — one address being resolved by 25 workers costs one or two lookups, not 25, and it collapses better as the resolver slows, because the claim is held longer. The breaker does its job too: at 300 ms, one worker absorbed the full latency and the median request took 0.78 ms.

The cost is that the others fail closed. A worker that cannot claim the lookup does not queue behind it — it returns "not verified", so the allow rule does not match and the request falls through to whatever comes next. On a cold cache, with a slow resolver, that is most of a concurrent burst.

For a crawler this is usually survivable: it retries, the cache is warm by then, and the verdict is cached for a day. But it means a genuine crawler arriving in parallel on a cold cache is treated as unverified, and if a rule below the allow blocks it, it is blocked.

Two things reduce it, and both are worth doing whatever else you set:

• Warm the cache before it matters. A single request per crawler address is enough, and crawler address ranges are stable.
• Keep the allow rule scoped, as presets/search-bots.yml does. An unverified crawler then meets the rest of the firewall on public paths only, rather than being blocked outright.

Waiting for the other worker's verdict¶

verify_claim_wait_ms lets a worker that could not claim the lookup wait a bounded time for the holder's answer instead of refusing at once. Same 25-process probe:

The wait has to exceed your resolver's latency to help fully. 100 ms fixes a 50 ms resolver and only reaches 7 of 25 against a 300 ms one, because the holder has not answered yet when the others give up.

And it is a real trade, not a free win. The median request goes from under a millisecond to roughly the resolver's latency, because workers that used to be refused instantly now wait. The worst case barely moves — 57 ms to 60 ms, 307 ms to 318 ms — since somebody was always paying for the lookup. What changes is that the others pay too, and get an answer for it.

metadata:
  verify: reverse-dns
  verify_claim_wait_ms: 100    # slightly above your resolver's typical latency

Default 0, which is what every release before 2.23.0 did. Turn it on when a wrongly refused crawler costs more than the added latency — and if that is true, warming the cache removes the situation entirely rather than mitigating it.

Offline switches it off, unless the rule says otherwise¶

By default, KANOPI_FIREWALL_SOURCES_OFFLINE covers this too, as it covers rule sources and remote configs: includes. verify_offline decides it per rule instead:

  metadata:
    verify: reverse-dns
    verify_suffixes: [.googlebot.com, .google.com]
    verify_offline: false     # verify, even though rule sources stay offline

Why they are separate. The constant exists because a rule-list fetch on the request path makes a visitor wait on someone else's HTTP server, with nothing bounding it. Verification is not that: it already has a verdict cache, a negative cache, a circuit breaker, single-flight claims and a slow threshold. Before 2.33.0 the two were one switch, so a host that keeps rule sources offline — as drupal/basic_firewall does by default — could not offer verified crawler rules at all (#391).

Offline, a verdict already in the cache is still honoured — reading it costs no network. An address with no cached verdict does not verify, so the rule does not match. Unless a shared verify_cache holds verdicts another node wrote, that means the rule verifies nobody new.

Plugin verify is switched off by KANOPI_FIREWALL_SOURCES_OFFLINE - the rule will not verify anyone new

Available on any plugin extending AbstractPluginBase, so IpAddress and the rest accept the same keys.

Caching¶

The plugin's detection is backed by matomo/device-detector, which compiles a 1.7 MB corpus of regex files on the first parse in each PHP process. That costs 110–637 ms depending on the user agent — ordinary mobile browsers are among the worst cases, because brand and model detection walks the largest part of the corpus. Once warm it is roughly 4 ms.

Under PHP-FPM every worker pays that on its first request, and again after each pm.max_requests recycle. The plugin therefore caches the compiled corpus by default:

618 ms   first request (populating the cache)
 23 ms   every subsequent process
618 ms   with caching disabled, every time

No configuration is needed. The cache is written to KANOPI_FIREWALL_CACHE_DIR when that constant is defined, otherwise to a kanopi-firewall-device-detector directory inside the system temp directory — the same convention the AbuseIPDB plugin uses for its verdict cache.

Pointing it somewhere else¶

- plugin: "Kanopi\\Firewall\\Plugins\\UserAgent"
  response: block
  enable: true
  metadata:
    cache:
      dir: /var/cache/firewall
  config:
    - "bot:true"

Using a different backend¶

Name a Memcached or Redis server with a DSN, and the pool is built for you:

    metadata:
      cache:
        adaptor: "memcached://cache.internal:11211"   # or redis://, rediss://
        namespace: device-detector                     # the default
        options: { connect_timeout: 500 }              # over the bounded defaults

cache: "memcached://cache.internal:11211" on its own is shorthand for the same thing. Any other PSR-6 pool is named by its class, with its constructor arguments:

    metadata:
      cache:
        adaptor: "Symfony\\Component\\Cache\\Adapter\\ApcuAdapter"
        args: ['device-detector', 0]

The same shapes work for every cache setting in the firewall: this one, the GeoIP cache, verify_cache, and CacheRateLimitStorage. A DSN pool connects with 1.5-second timeouts, and a server that does not answer is found when the plugin starts, not on every lookup after.

A server that cannot be reached, or a missing ext-memcached or ext-redis, leaves detection running uncached with a warning that names which it was. It does not fall back to the file cache, because that is not what you asked for.

A pool your framework already has can be injected through configuration overrides, since YAML cannot carry an object:

Firewall::create([__DIR__ . '/firewall.yml'], [
    '[plugins][0][metadata][cache][adaptor]' => $myCachePool,
]);

Turning it off¶

    metadata:
      cache: false

Detection is unchanged either way — only the speed differs.

A cache that cannot be created never stops the plugin working: the failure is logged at warning and detection continues uncached. An optimisation should not be able to take a site down.

User Agent regex cache is not writable - every request will re-parse the detection corpus

Only what your rules need¶

Detection runs in four phases — bot, OS, client, then device (brand and model). Since the rules are known up front, the plugin stops at the deepest phase they actually read:

A config that only asks bot:true therefore costs a fraction of one that inspects brand. Nothing needs configuring — the depth is derived from your rules.

Two properties are deliberate:

• Phases are cumulative, not individually selectable. Device detection reads the OS and client results to infer a type — Android plus a browser becomes smartphone. Running it without them would produce a wrong device type, not just a faster one.
• Bot detection always runs. Detection stops early once a bot is identified, so a bot never reaches client or device parsing. Skipping it would let bots through to phases they do not reach today, changing what client.* rules match.

An unrecognised variable or rule shape falls back to running every phase, so the worst case is a lost optimisation rather than a rule that quietly stops matching.

Available Variables¶

• bot - Whether the user agent is a bot ("true" or "false")
• device.type - Device type (desktop, smartphone, tablet, etc.)
• client.name - Browser or client name
• client.type - Client type (browser, mobile app, etc.)
• client.version - Client version number
• os.name - Operating system name
• os.short_name - OS short name (WIN, MAC, LIN, etc.)
• os.version - OS version number
• brand - Device brand (Apple, Samsung, etc.)
• model - Device model