DNS Security: DNSSEC, DoH/DoT and How DNS Hijacking Works

A user types the address of their bank, sees a padlock, and logs in. Nothing looks wrong, yet the name resolved to an address controlled by someone else, and the certificate on the page was issued to that someone else a few hours earlier. No malware was involved and no password was guessed. The attacker changed one record, or convinced one resolver to believe a forged answer. DNS sits underneath nearly every connection your organization makes, and it was designed in an era when nobody expected the network to be hostile. This article explains where DNS can be attacked, what each protection actually covers, and how to configure and monitor it.

How Resolution Works, and Where It Breaks

A lookup involves several parties:

1. The stub resolver on the device asks a recursive resolver (your ISP, your company, or a public one).
2. The recursive resolver walks the hierarchy: root servers, then the TLD servers (.org), then the authoritative servers for the zone (example.org).
3. The answer is cached for the duration of its TTL and returned to the client.

Each hop is a place where trust can fail:

Hop	What can go wrong
Device to resolver	On-path attacker or rogue Wi-Fi rewrites the response, or DHCP hands out a malicious resolver
Resolver cache	Forged answers accepted into the cache (cache poisoning)
Resolver to authoritative	Same forgery problem, upstream
Authoritative zone	Records changed through a compromised registrar, DNS provider or API token
Registrar	Nameserver delegation changed, domain transferred or expired

Classic DNS runs over UDP port 53 with no encryption and no authentication. A response is accepted if it arrives from the right address with the right transaction ID and source port. Randomizing those values raised the cost of blind forgery, but it never made the protocol authenticated.

Hijacking in Practice

DNS hijacking is a family of attacks rather than a single technique. Understanding the entry point tells you which control matters.

Registrar or DNS provider account takeover. This is the most damaging variant and the one defenders underestimate. If an attacker logs in to the account that manages your domain, they can change nameservers or records, and every validation you rely on follows. Certificate authorities that use domain validation will issue valid certificates for the attacker’s server, so the padlock appears. Causes are mundane: password reuse, SMS-based recovery, no MFA, a former employee’s access, or social engineering of the registrar’s support desk (see social engineering and phishing detection).

Router and endpoint compromise. Home and small-office routers with default credentials or exposed admin panels have been used to swap the DNS server handed out via DHCP. Malware on an endpoint can do the same locally. Every device behind the router then talks to the attacker’s resolver.

On-path rewriting. On open Wi-Fi or a compromised network segment, an attacker answers DNS queries faster than the legitimate resolver. Captive portals do something similar legitimately, which is why clients are poor at distinguishing the two.

Cache poisoning. An attacker tries to inject a forged answer into a recursive resolver. Modern resolvers randomize source ports and transaction IDs, use 0x20 case randomization in some implementations, and limit what they accept in additional sections. Poisoning is much harder than it was, but it is not impossible against poorly configured or outdated resolvers.

Dangling records and subdomain takeover. A CNAME that still points at a decommissioned cloud resource, bucket or hosting service can be claimed by anyone who registers that resource name. The attacker then serves content under your subdomain, which can be used for phishing or to steal cookies scoped to the parent domain. This is configuration debt, not a protocol flaw.

Expired domains. A lapsed domain can be re-registered by anyone, who inherits whatever still trusts it: email, API callbacks, hardcoded update URLs.

What DNSSEC Does and Does Not Do

DNSSEC adds origin authentication and integrity to DNS data. Zone operators sign their records with a private key. Resolvers that validate can check the signature against a public key, and that key is itself vouched for by the parent zone, up to the root.

The main record types:

• DNSKEY: the zone’s public keys (a zone-signing key and a key-signing key in the classic split).
• RRSIG: the signature over a record set.
• DS: a hash of the child’s key-signing key, published in the parent zone. This is the link in the chain of trust.
• NSEC / NSEC3: authenticated proof that a name does not exist, so an attacker cannot forge a negative answer.

You can inspect the chain with standard tools against domains you manage:

# Ask for DNSSEC records and show the AD (authenticated data) flag
dig +dnssec example.org A

# Show the DS record published at the parent
dig DS example.org +short

# Trace validation from the root down
delv @1.1.1.1 example.org A +rtrace

If a validating resolver sets the ad flag in the response header, it verified the chain. If validation fails, a validating resolver returns SERVFAIL instead of the bad data, which is the point: a forged answer becomes an outage rather than a silent redirect.

What DNSSEC does not provide:

• No confidentiality. Queries and answers are still visible on the wire.
• No protection between the stub and its resolver unless the stub itself validates. Most devices trust the resolver’s ad bit over an unauthenticated channel.
• No protection against registrar compromise. An attacker who controls the account can replace the DS record too. DNSSEC protects the data path, not the account.
• No protection against a malicious but correctly signed zone. A phishing domain can be perfectly signed.

Operational risk is real

DNSSEC fails loudly when mismanaged. Common causes of self-inflicted outages:

• Replacing the key-signing key without updating the DS at the parent, or the reverse.
• Letting signatures expire because an automated signer stopped running.
• Moving a signed zone to a new provider without handling the old DS record first.

Use a provider or software that automates signing and key rollover, enable CDS/CDNSKEY support if both registrar and DNS host offer it, and monitor signature expiry. If your team cannot commit to operating it, a managed DNS service with built-in DNSSEC is safer than a hand-rolled signer.

A zone-signing check you can put in monitoring:

#!/usr/bin/env bash
# Alert if the zone fails DNSSEC validation through a validating resolver
domain="example.org"
if ! dig @9.9.9.9 +dnssec "$domain" SOA | grep -q ' ad'; then
  echo "DNSSEC validation failing or absent for $domain" >&2
  exit 1
fi

DoT and DoH: Protecting the Last Mile

DNS over TLS (DoT) wraps queries between a client and its resolver in TLS on a dedicated port (853). DNS over HTTPS (DoH) carries the same messages inside HTTPS requests on port 443. Both provide:

• Confidentiality against on-path observers on the local network.
• Integrity and server authentication for the client-to-resolver hop, so rogue Wi-Fi cannot rewrite answers.

Neither validates that the data is authentic upstream. The resolver can still be fooled by a poisoned cache, and the resolver itself sees every query. You are choosing whom to trust, not removing trust.

DoT versus DoH trade-offs

Aspect	DoT	DoH
Port	853, easy to identify	443, blends with web traffic
Network visibility	Admins can allow, block or log it	Hard to distinguish without inspection
Typical deployment	OS-level, routers, resolvers	Browsers and apps, sometimes OS
Enterprise control	Simple firewall policy	Needs policy on the endpoint or a canary domain

For organizations, DoH inside browsers can bypass internal resolvers, which breaks split-horizon DNS, internal name resolution and DNS-based filtering and logging. If you rely on DNS visibility, either run your own DoH/DoT endpoint that clients are configured to use, or manage the browser setting centrally. Do not simply block everything and assume you are covered; you also lose the privacy benefit for roaming users.

A minimal DoT forwarder using Unbound on a resolver you operate:

# /etc/unbound/unbound.conf.d/dot-forward.conf
server:
    tls-cert-bundle: /etc/ssl/certs/ca-certificates.crt
    # Validate DNSSEC locally using the root trust anchor
    auto-trust-anchor-file: "/var/lib/unbound/root.key"
    harden-dnssec-stripped: yes
    harden-glue: yes
    harden-referral-path: yes
    qname-minimisation: yes
    hide-identity: yes
    hide-version: yes

forward-zone:
    name: "."
    forward-tls-upstream: yes
    forward-addr: 9.9.9.9@853#dns.quad9.net
    forward-addr: 1.1.1.1@853#cloudflare-dns.com

Here the resolver validates DNSSEC itself and uses TLS to its upstreams, so you get both authenticity and last-mile confidentiality.

Hardening the Authoritative Side

Most real-world incidents come from account and configuration problems, so start there.

Registrar and DNS provider accounts

• Require phishing-resistant MFA (security keys or passkeys, see passkeys vs passwords). Avoid SMS recovery.
• Use role-based accounts with named individuals, not a shared mailbox and password.
• Enable registrar lock (clientTransferProhibited) and, for high-value domains, the registry lock offered by many registries, which requires out-of-band confirmation for changes.
• Restrict and rotate API tokens, scope them to a single zone, and store them in a secrets manager.
• Keep contact emails on domains you control and that are not themselves subject to the same domain.
• Turn on auto-renew and set calendar alerts for expiry.

Zone hygiene

• Audit for dangling CNAME, A and NS records. Remove records when you decommission a resource, and ideally do it in the same change.
• Publish CAA records to restrict which certificate authorities may issue for your domain:

example.org.  3600 IN CAA 0 issue "letsencrypt.org"
example.org.  3600 IN CAA 0 issuewild ";"
example.org.  3600 IN CAA 0 iodef "mailto:[email protected]"

• Keep SPF, DKIM and DMARC records tight, since DNS also carries email authentication.
• Restrict zone transfers (AXFR) to named secondaries, with TSIG if possible.
• Treat DNS configuration as code: keep zones in version control, require review, and apply through a pipeline so changes are attributable.

A quick audit of dangling CNAMEs for a zone you own:

#!/usr/bin/env bash
# Lists CNAME targets in a zone export that no longer resolve
zone_file="example.org.zone"
awk '$4 == "CNAME" {print $1, $5}' "$zone_file" | while read -r name target; do
  if ! dig +short "$target" | grep -q .; then
    echo "Dangling: $name -> $target"
  fi
done

Detection

You usually notice DNS compromise through side effects. Useful signals:

• Certificate Transparency monitoring. Subscribe to CT alerts for your domains. An unexpected certificate is often the first visible sign of hijack or takeover.
• Record change alerts from your DNS provider, and a scheduled job that diffs live records against your source of truth.
• Registrar notifications for nameserver, contact and lock changes, sent to a monitored list.
• Passive DNS and resolver logs. Look for queries to newly registered domains, unusual TXT volume, very long labels or high-entropy names that suggest tunneling, and internal clients talking to external resolvers directly.
• Egress policy checks. Alert on outbound port 53 or 853 traffic that does not come from your sanctioned resolvers.

A simple record-diff job:

#!/usr/bin/env bash
# Compare live NS and A records with the expected baseline
expected="/etc/dns-baseline/example.org.txt"
live=$(for t in NS A MX; do dig +short example.org "$t" | sed "s/^/$t /"; done | sort)
diff <(sort "$expected") <(echo "$live") || echo "DNS drift detected for example.org"

Testing Your Own Setup

Only test domains and networks you own or are explicitly authorized to assess. Useful checks:

• Verify DNSSEC with delv or an online validator, and confirm the DS record at the parent matches the key published in the zone.
• Confirm that AXFR is refused from arbitrary hosts.
• Enumerate your own subdomains and check each target for takeover conditions.
• From a test client, confirm devices use the intended resolver and that rogue DHCP offers are blocked by DHCP snooping on managed switches.

When Not to Rely on These Controls

• DNSSEC will not save you from a stolen registrar login. Spend effort on account security first.
• DoH to a third-party resolver will not stop a compromised endpoint, and it can blind your own monitoring.
• Short TTLs help recovery but increase query volume and load; choose them deliberately, and lower them before planned changes rather than permanently.
• Do not enable DNSSEC if you cannot monitor it. A broken chain is an outage for validating users.

Key Takeaways

• DNS hijacking mostly starts with account compromise, router tampering or stale records, not clever protocol attacks. Lock down registrar and DNS provider access first.
• DNSSEC authenticates the data from the zone to a validating resolver; DoT and DoH protect only the last mile between client and resolver. They complement each other and neither hides your queries from the resolver operator.
• Use registrar lock, phishing-resistant MFA, scoped API tokens and CAA records, and keep zones in version control.
• Monitor Certificate Transparency logs, record drift, nameserver changes and unexpected outbound DNS traffic to detect hijacks early.
• Audit and remove dangling CNAMEs and lapsed domains to prevent subdomain takeover.
• Automate DNSSEC signing and key rollover, and monitor it, or do not deploy it.