Rule Sources¶

A plugin's rules do not have to be written into your configuration. metadata.sources points a plugin at lists that live somewhere else — a file on disk, a URL, a list your team publishes, a list someone else publishes — and turns whatever shape they arrive in into the rules that plugin expects.

The problem this solves is that almost nothing in the world publishes rules in our format. Cloud providers publish JSON. Monitoring vendors publish newline-delimited text. Threat feeds publish NDJSON. Sources declare what a list is and which part of it you want, so those lists can be consumed as they are rather than reshaped by hand first.

plugins:
  - plugin: "Kanopi\\Firewall\\Plugins\\IpAddress"
    response: block
    enable: true
    metadata:
      sources:
        - name: tor-exits
          upstream: "{config_dir}/lists/tor-exits.txt"
          validate: cidr
    config:
      - 203.0.113.7      # local additions still land last

The pipeline¶

Every source runs through the same seven stages. Only the first two know anything about wire formats; after decode everything is a plain PHP array, which is what lets one pipeline serve text, JSON, NDJSON, YAML, CSV, and TSV alike.

fetch → decompress → decode → select → where → template → validate

Every stage is optional except the fetch. A source that is already a plain list of the values a plugin wants needs nothing but an upstream.

Every option¶

On the source¶

On the upstream¶

A bare string is shorthand for a source with nothing but an upstream, and an upstream with nothing but a URL. These three are the same source:

sources:
  - "{config_dir}/lists/tor-exits.txt"

  - upstream: "{config_dir}/lists/tor-exits.txt"

  - upstream:
      url: "{config_dir}/lists/tor-exits.txt"

Upstreams¶

Most upstreams are a bare location, so upstream takes a string. When the request needs more than a URL, the same key takes a map instead:

- name: private-feed
  upstream:
    url: https://feeds.example.com/v1/blocklist.json
    method: POST
    body: '{"scope":"active"}'
    headers:
      Content-Type: application/json
      X-Account: "12345"
    auth:
      type: bearer
      token: "%env(FEED_TOKEN)%"
    timeout: 10
  format: json
  select: "results.*"
  template: "{value[address]}"
  validate: cidr

Everything about reaching the list lives under upstream; everything else on the source is about the list itself. That split is also why the CSV first-row option is header_row — headers under upstream is unambiguously request headers.

Header values are stripped of newlines, since one would otherwise let an injected value start a header of its own.

Authentication¶

upstream.auth covers the four shapes feeds actually use.

upstream:
  url: https://feeds.example.com/v1/list.json
  auth:
    type: bearer
    token: "%env(FEED_TOKEN)%"

upstream:
  url: https://feeds.example.com/v1/list.json
  auth:
    type: basic
    username: "%env(FEED_USER)%"
    password: "%env(FEED_PASSWORD)%"

upstream:
  url: https://feeds.example.com/v1/list.json
  auth:
    type: header
    name: X-API-Key
    value: "%env(FEED_KEY)%"

upstream:
  url: https://feeds.example.com/v1/list.json
  auth:
    type: query
    name: api_key
    value: "%env(FEED_KEY)%"

Credentials never reach a log¶

Every place the firewall shows an upstream — log context, exception messages, CLI output, the plugin's own debug dump of its metadata — shows a redacted form:

https://reader:hunter2@example.org/list.txt   →  https://***@example.org/list.txt
https://example.org/list?api_key=s3cr3t       →  https://example.org/list?api_key=***

Redaction runs whether or not auth is declared, because a URL pasted straight in can carry a token on its own. Parameters named token, key, api_key, access_token, auth, password, secret, signature and similar are all scrubbed.

Plain http:// is refused¶

A credential sent over plain http travels in clear text, so declaring one on an http:// upstream is an error. An operator on a trusted internal network can say so explicitly:

upstream:
  url: http://internal.example/list.txt
  auth:
    type: bearer
    token: "%env(FEED_TOKEN)%"
  allow_insecure: true

Redirects do not carry credentials off-origin¶

Redirects are followed by hand rather than by PHP's follow_location, which reuses the whole request context on every hop — so a redirect to another host would resend your Authorization header to whoever answered it. When a hop changes scheme, host, or port, both auth and any headers you set are dropped before the next request. An API key header is a credential whatever it is called.

max_redirects: 0 disables following altogether.

Rotating a credential does not invalidate the cache¶

The cache key covers everything that can change what comes back — URL, method, headers, body — but deliberately not the credential. Rotating a token does not change the list it fetches, and re-decoding every source on a key rotation would be pure waste.

Formats¶

txt — newline-delimited¶

The default, and what most published lists actually are. Real lists are rarely tidy, so the decoder handles the mess: # banners, blank separators, trailing whitespace, and trailing labels are all stripped.

# UptimeRobot IPv4
# updated 2026-09-04

216.144.250.150
69.162.124.224/28    # datacenter block

- name: uptimerobot
  upstream: "{config_dir}/lists/uptimerobot.txt"
  validate: cidr

A marker only opens a comment at the start of a line or after whitespace, so a value that legitimately contains one — /path#fragment — survives. Set comment: ";" for a different marker, or comment: "" to disable stripping entirely.

json — a single document¶

- name: cloud-ec2-us
  upstream: https://example.org/v1/ranges.json
  format: json
  select: "prefixes.*"
  where:
    - "service:EC2"
    - "region@starts_with:us-"
  template: "{value[ip_prefix]}"
  validate: cidr

ndjson — one document per line¶

A separate format because the body as a whole is not valid JSON. Several threat feeds publish this.

- name: reputation-feed
  upstream: https://example.org/v1/feed.ndjson
  where:
    - "score@greater_than:80"
  template: "{value[ip]}"

yaml — a single document¶

- name: shared-rules
  upstream: "{config_dir}/lists/rules.yml"

csv and tsv — delimited rows¶

With header_row: true (the default) each row becomes a map keyed by column name. With header_row: false rows stay numerically indexed and you address columns by position.

asn,org,country
13335,CLOUDFLARENET,US
16509,AMAZON-02,US

- name: hosting-asns
  upstream: "{config_dir}/lists/hosting-asns.csv"
  format: csv
  template: "asn:{value[asn]}"

→ ["asn:13335", "asn:16509"]

With header_row: false the same file is template: "asn:{value[0]}", addressing columns by index.

Compression¶

Compression is a separate axis from format, so there is no json-gz to declare:

- name: big-list
  upstream: https://example.org/v1/ranges.json.gz    # format and compression both inferred

Both are inferred from the extension: ranges.json.gz is JSON, gzipped. Declare compression: gzip explicitly when the URL does not say so.

select — narrowing a document¶

select is a dot-path with wildcards and alternation — the same syntax the loader uses for relativePathKeys.

select: "prefixes.*"                    # every record in one collection
select: "{prefixes,ipv6_prefixes}.*"    # every record across two collections
select: "data.regions.*.ranges.*"       # nested collections

Alternation earns its keep on documents that split what is really one list across two keys — AWS publishes IPv4 under prefixes and IPv6 under ipv6_prefixes, and {prefixes,ipv6_prefixes}.* takes both in one source.

With no select, a decoded list is already the record set, and anything else is treated as a single record. So a document that is the list needs no selector at all.

where — filtering records¶

where reuses the plugin conditional-logic engine, so every operator, negation, and group described in Conditional Logic works here unchanged.

where:
  - "service:EC2"
  - "region@starts_with:us-"
  - "!deprecated:true"

where:
  - type: OR
    rules:
      - "service:EC2"
      - "service:LAMBDA"

Filtering is what makes large published documents usable. A cloud provider's range list runs to thousands of prefixes across every service and region; almost nobody wants all of them, and select alone cannot express which — by the time you have descended to ip_prefix you have lost the service and region fields you needed to decide.

Scalar records — text lines — expose themselves as value, so text lists are filterable too:

where:
  - "value@starts_with:10."

template — shaping entries¶

Worth being precise about how much of the plugin surface needs this. IpAddress is the outlier: its config is bare values (10.0.0.0/8), so a list of addresses feeds it directly. Every other plugin takes rule strings — Asn takes asn:13335, Url takes path@starts_with:/admin, UserAgent takes client.name:Chrome — and no upstream feed publishes those. So a template is the normal case, not the exception.

{value} always means the record. Index into it when it is structured:

Write a literal brace as {{ or }}.

Plucking one field¶

select: "prefixes.*"
where:
  - "service:EC2"
template: "{value[ip_prefix]}"

A template that is exactly one placeholder keeps the field's own type, so a numeric column stays numeric rather than becoming the string "443".

Building a rule string¶

- name: bad-agents
  upstream: "{config_dir}/lists/user-agents.txt"
  format: txt
  template: "client.name@contains:{value}"

Building a structured rule¶

Url and UserAgent both accept grouped condition objects, which a string template cannot produce. Interpolate into the leaves instead:

- name: scanner-signatures
  upstream: https://example.org/v1/scanners.json
  format: json
  select: "signatures.*"
  template:
    type: AND
    rules:
      - "client.name@contains:{value[name]}"
      - "bot:{value[is_bot]}"

Passing records through¶

With no template, records reach the plugin untouched — the right default for a source that already publishes rules in our own shape.

When a placeholder cannot resolve¶

The record is dropped, and the count is logged. A rule with a hole in it would match the wrong things, which is worse than one rule fewer.

Guardrails¶

These matter most when upstream is a URL you do not control. A feed that breaks, or is tampered with, otherwise reaches a plugin's rule list intact.

They answer different questions and none substitutes for another: is each entry well formed (validate), is this the publisher's file (checksum/signature), is it a plausible size (max_size, max_entries), and is it a plausible amount of change (max_delta).

validate¶

Individual bad entries are dropped and logged rather than failing the whole source: one malformed line in a 9,000-entry list should not take the other 8,999 with it. Structured entries are rule maps rather than values, so the scalar validators pass them through.

A /0 prefix is accepted but logged as a warning — it covers the entire address space, is essentially never intended, and is catastrophic in either an allow or a block list.

max_size and max_entries¶

upstream.timeout bounds how long a fetch may take and max_redirects bounds where it may go. Until 2.30.0 nothing bounded how much came back, so an upstream that grew without bound was read without bound — and the body and its decoded form are held at once, the decoded form usually being the larger of the two.

upstream:
  url: https://example.org/v1/blocklist.txt
  max_size: 10M        # bytes, or K/M/G. Default 32M. 0 for no limit
max_entries: 250000    # refuse a refresh producing more than this

max_size defaults to 32 MiB rather than to nothing, so "a list that grows to 4 GB should fail, not OOM the refresh job" needs no configuration to be true. It is far above any published list in common use — AWS's ip-ranges.json, the largest most deployments meet, is under 3 MB — and far below the point at which a refresh job dies.

Both refuse rather than truncate. Half a block list is a list whose meaning nobody knows: it would load, match fewer things than it should, and look exactly like a list that is simply shorter this week.

Both fail the source the way a failed fetch does, so on_error applies and last_known_good keeps the previous artifact.

An error page where a list should be¶

The realistic failure is not an attack. A CDN, a captive portal or a misconfigured proxy answers with HTML where a text file should be — and it is small, so max_size says nothing about it. It decodes as txt without complaint and produces a few dozen lines that are not addresses.

With a validate: set, every one of those lines is dropped, and the source is left contributing nothing. That used to be silent apart from a warning: the rule simply stopped matching.

A source that decoded to something and validated down to nothing now fails:

Source failed to load; using last known good copy
  source: abusive-ips
  reason: Source "abusive-ips": every one of its 4 entries failed `validate: cidr`, so it has
          nothing to contribute. That is usually an error page where a list should be.
          First: "<html>", "<head><title>404 Not Found</title></head>", "<body>nope</body>"
  entries: 51204

This is deliberately not the same as dropping bad entries. One malformed line in a 9,000-entry list still drops that line and keeps the other 8,999 — that is what validate is for, and it would be a poor trade to lose it. The escalation is about a source that produced nothing at all, which is a different claim.

It needs a validate:. Without one there is nothing an entry can fail, and an empty result is just an empty list — which is a legitimate thing for a feed to publish on a quiet day.

max_delta¶

Rejects a refresh whose entry count moved further than a healthy update ever would:

max_delta: 0.25    # refuse a change of more than 25% either way

The first load has nothing to compare against, so the check is inert until a source has succeeded once. This is what stops an upstream that starts returning an error page — or an empty document — from quietly emptying your block list.

Entries that match everybody¶

An entry of 0.0.0.0/0, ::/0 or * matches every address there is. On a block rule that refuses every visitor to the site; the firewall is not down, it is working exactly as configured, which is worse to diagnose.

Such entries are refused from a source and logged at error:

Source entries matching every address were refused
  source: abusive-ips
  refused: ["0.0.0.0/0"]

The rest of the source is kept — one bad line should not discard fifty thousand good ones, and a source that fails wholesale is what on_error is for.

If a source genuinely means it — an allow list that opens the site to everyone during a migration, say — declare it:

sources:
  - name: everyone-for-now
    upstream: "{config_dir}/open.txt"
    allow_catch_all: true

The refusal cannot be decided by the rule's response:, because a plugin is never told which bucket it is in — so the source is where the intent has to be declared.

Changing this option changes the source's fingerprint, so the next load re-decodes rather than reusing entries filtered under the old setting.

Verifying what you fetched¶

A list arrives over HTTPS and is used. HTTPS authenticates the host and protects the transport; it says nothing about a repository that was compromised, a CDN object that was replaced, or a publisher who pushed the wrong file. Nothing in the pipeline above notices the difference.

sources:
  - name: abusive-ips
    upstream: https://example.org/v1/abusive-ips.txt
    checksum: sha256            # sidecar at <upstream>.sha256

Both keys are opt-in and stay that way. Most published lists in this ecosystem ship no sidecar at all, so a source that declares neither keeps working exactly as before. firewall-doctor reports how many of your remote sources are in that position, in one line, rather than failing them.

The two tiers, and the gap between them¶

A sidecar from the same host raises the bar without clearing it. It is worth having as the cheap tier, and it is exactly why a pinned-key signature is the one that means something.

checksum:¶

checksum: sha256                        # shorthand: sidecar at <upstream>.sha256

checksum:
  algorithm: sha512                     # sha256 (default), sha384, sha512
  url: https://example.org/SHA256SUMS   # a publisher who keeps every digest in one file

checksum:
  value: "9f86d081884c7d65…"            # pinned here; nothing is fetched

md5 and sha1 are deliberately not offered. A digest a forger can collide is a digest that says nothing, and listing it would invite matching whatever a publisher happens to emit.

The sidecar is read in whichever shape it was published — a bare digest, coreutils text or binary mode, or a multi-file SHA256SUMS, in which case the line naming this file is the one used. A sidecar with exactly one line whose file name does not match is accepted: a publisher renaming their own file is not a reason to refuse a digest that is right there.

signature:¶

signature:
  algorithm: ed25519
  public_key: "%env(FIREWALL_FEED_KEY)%"   # required — base64, hex, or raw
  url: https://example.org/v1/list.txt.sig # default: <upstream>.sig

public_key is required. A signature with no pinned key verifies that the file was signed by whoever signed it, which is not a fact about anything.

Needs ext-sodium. A source declaring signature: on a host without it fails rather than running unverified — that source has been marked as one that matters, and quietly skipping the check is the one outcome nobody asked for.

What happens when it does not match¶

The body is refused and the source takes its ordinary failure policy, which is why last_known_good — the default — means the last verified copy keeps serving and the refused bytes are never used:

Source failed to load; using last known good copy
  source: abusive-ips
  reason: Source "abusive-ips": sha256 checksum mismatch. Expected 9f86d081884c7d65…,
          got 2c26b46b68ffc68f…
  entries: 51204

Mark the source required: true where serving a stale copy is the wrong direction — an allow list, typically — and the mismatch stops the bootstrap instead.

A missing, empty or unreadable sidecar is the same kind of failure. It is not a reason to use the body: a source that cannot be checked has not been checked.

Three details worth knowing:

• The digest covers the artifact as distributed. A .gz list is hashed gzipped, before the pipeline decompresses it, because that is the file the publisher hashed.
• The sidecar is fetched the way the list is, with the same credential and headers — a private feed's digest lives behind the same door — but always as a GET, and never with the list's request body.
• Adding verification invalidates the cache. Entries cached before a source was given a checksum: came from bytes nothing checked, so they are re-fetched and re-verified rather than reused.

What it is not¶

max_delta is the neighbouring guardrail and answers a different question. "Is this the publisher's file" and "is this a plausible amount of change" are independent, and neither substitutes for the other — a signed file can still be a signed mistake.

Failure policy¶

Failure handling is deliberately asymmetric, and choosing the wrong side is the most consequential mistake available here.

required: true forces aborting regardless of on_error.

For a block list, degrading is right. Losing coverage beats losing the site, so the default lets a broken source fall back or drop out while the others keep working.

For an allow list, degrading is wrong. Quietly dropping your CI provider's ranges means deploys start getting challenged or blocked, and the only signal is a log line nobody is watching. Mark those sources required:

- plugin: "Kanopi\\Firewall\\Plugins\\IpAddress"
  response: allow
  weight: -200
  metadata:
    sources:
      - name: ci-ranges
        upstream: https://example.org/v1/ci-ranges.json
        required: true      # fail loudly rather than silently locking out CI
        max_delta: 0.25

Caching, and keeping fetches off the request path¶

What is cached¶

The result, not the bytes. Caching a raw body only saves the download; the expensive part of a large list is decoding it, and a 9,000-entry document parsed, selected, filtered, and rendered on every request costs far more than the transfer ever did.

Entries are stored post-pipeline as a plain PHP array, which opcache holds in memory — so a warm request is one include, not a parse.

The cache key covers upstream plus every option that changes the decoded result, so editing a select or template invalidates it on its own. Renaming a source, or changing its ttl, does not.

How a refresh is avoided¶

1. Inside ttl, the cached entries are used with no fetch at all.
2. Past it, the stored ETag and Last-Modified go back as If-None-Match and If-Modified-Since. A 304 means no body crosses the wire.
3. For upstreams that do not support conditional requests, the body is hashed. An unchanged hash skips the decode even though the download happened.

Where it is written¶

KANOPI_FIREWALL_CACHE_DIR/sources when that constant is defined, otherwise sys_get_temp_dir()/kanopi-firewall-sources.

Syncing out of band¶

Even with all of the above, a cold cache on a live request makes a visitor wait on somebody else's HTTP server, and a TTL expiry under load sends every concurrent request after the same URL at once. Refresh out of band instead:

# At deploy time, or on a cron
vendor/bin/firewall-sources config/firewall.yml

Cache directory: /var/cache/firewall/sources

  ✓ aws-ec2-us                  1284 entries in 412ms
  ✓ uptimerobot                 62 entries in 38ms
  ✓ tor-exits                   1533 entries in 190ms

3 sources refreshed.

It exits 0 when everything loaded, 1 when any source failed, and 2 when the configuration itself could not be read — so a deploy step can fail on a bad list. See Syncing Rule Sources for cron and deploy recipes.

The same upstream declared on several plugin entries is fetched once per run, not once per plugin.

Then stop the runtime reaching the network at all:

define('KANOPI_FIREWALL_SOURCES_OFFLINE', true);

In offline mode a remote source is served from cache, and one with nothing cached is an error rather than a silently empty rule list. Local files are still read normally, which is the intended arrangement: sync to disk out of band, serve from disk.

Attributing a match¶

Every entry remembers which source supplied it, so a match can name the list that caused it rather than reporting only that something matched:

$plugin->entrySource(0);   // "circleci"
$plugin->entrySource(42);  // null — a local entry from inline config

Migrating from metadata.config¶

metadata.config still works, and is deprecated only for the job sources now do better.

For rule lists, move to sources. You gain declared formats, filtering, per-source TTL and failure policy, and validation:

# Before
metadata:
  config:
    - "{config_dir}/lists/blocklist.yml"

# After
metadata:
  sources:
    - upstream: "{config_dir}/lists/blocklist.yml"

Using metadata.config for a list logs a deprecation notice naming the files involved.

For nested configuration documents, keep using it. Sources produce lists of entries — that is the whole shape of the pipeline — so a map-shaped document arrives as one record rather than merging key by key. The nested scoring and risk_levels trees that VulnerabilityScore loads still belong in metadata.config, and no deprecation notice is emitted for them.

The two can be declared together. Sources contribute first, then files loaded through metadata.config, then inline config:.

Gotchas¶

Sources are not config includes. The top-level configs: key merges whole configuration documents; metadata.sources produces a plugin's rule entries. A bare list file added to configs: does nothing useful. See Loading & Includes.

where narrows, config: widens. Covered above, but it is the difference most likely to surprise you.

A large list is scanned linearly. IpAddress walks its rule list per request. That is fine for hundreds of entries; measure before pointing a plugin at many thousands, and prefer where to cut a document down to the part you actually need.

headers and header_row are different things. upstream.headers are request headers. header_row is the CSV/TSV option for whether the first row names the columns. They sit at different levels for exactly that reason.

A sync job needs the credentials too. bin/firewall-sources is a separate process from your application, so a token in your web server's environment is not automatically present in cron. See Syncing Rule Sources.

Declare format when the extension lies. Inference reads the extension and falls back to txt. An endpoint like https://example.org/v1/ranges serving JSON needs format: json spelled out.