Opportunistic Versus Enforced TLS For SMTP

STARTTLS encrypts SMTP between mail servers, but by default it is opportunistic: the sending server uses TLS only if the receiving server advertises support for it, and quietly falls back to plaintext when TLS is absent, fails, or appears to be unsupported. That fallback is the weakness. A network attacker positioned between the two servers can strip the STARTTLS offer out of the handshake, and the sender, seeing no TLS on offer, downgrades to cleartext and delivers the message in the open. Enforced TLS closes this gap by telling senders, out of band, that encryption is mandatory and which certificate to expect. The two mechanisms that do this are MTA-STS and DANE.

The reason opportunistic TLS exists at all is reach. SMTP between independent organizations has no prior trust relationship and no way to require encryption without risking lost mail, so the protocol chose "encrypt if you can" over "encrypt or fail." That tradeoff was reasonable for adoption and is exactly what enforcement now has to fix.

How STARTTLS actually works

STARTTLS is not a separate port or a separate protocol. It is an SMTP command. The connection opens in plaintext on port 25, the receiving server lists its capabilities in response to EHLO, and if STARTTLS is among them the sender issues the command to upgrade the existing connection to TLS before sending any mail data.

220 mx.example.com ESMTP
EHLO sender.example.net
250-mx.example.com
250-STARTTLS
250 SIZE 52428800
STARTTLS
220 2.0.0 Ready to start TLS
... TLS handshake, then the encrypted session continues ...

The vulnerability is structural. Everything up to the TLS handshake is in cleartext, including the 250-STARTTLS line. An attacker who can see and modify the traffic, through BGP hijacking, DNS spoofing, or any on-path position, simply removes 250-STARTTLS from the capability list or rewrites the STARTTLS command so it fails. The sender observes a server that does not support TLS and, under opportunistic policy, sends in plaintext. Neither side logs an error, because from each end this looks like an ordinary unencrypted delivery. This is STARTTLS stripping, and it is undetectable to the participants without an external policy to compare against.

Opportunistic versus enforced

Opportunistic TLS asks "is TLS available?" Enforced TLS asserts "TLS is required, and here is how to verify the certificate." The difference is whether the sender has an authenticated, tamper-resistant signal that the destination must be reached over TLS. Two standards provide that signal.

MTA-STS

MTA-STS (RFC 8461) lets a domain publish a policy saying its MX hosts support TLS and that senders should refuse to deliver if a valid TLS connection cannot be established. It is anchored in the Web PKI, so it does not require DNSSEC, which is the main reason it is easier to deploy.

It has two parts. A DNS TXT record advertises that a policy exists and carries an id that changes when the policy changes:

_mta-sts.example.com.   IN   TXT   "v=STSv1; id=20260613T120000;"

The policy itself is served over HTTPS from a well-known URL on a mta-sts subdomain, and the validity of that HTTPS certificate is what anchors trust:

# https://mta-sts.example.com/.well-known/mta-sts.txt
version: STSv1
mode: enforce
mx: mx1.example.com
mx: mx2.example.com
max_age: 604800

In enforce mode, a sending MTA that supports MTA-STS will refuse to deliver to a host that is not in the mx list or whose certificate is not valid and PKI-trusted. The sender caches the policy for max_age seconds, so an attacker cannot strip it on a single connection. The pair mode: testing plus a TLS-RPT reporting record is the standard way to roll this out without bouncing mail while you confirm coverage.

DANE and TLSA

DANE (RFC 7672 for SMTP) takes a different anchor: DNSSEC. The destination publishes a TLSA record that pins the expected certificate or public key, and because the DNS answer is signed with DNSSEC it cannot be forged or stripped without breaking the signature. A sender that supports DANE validates DNSSEC, reads the TLSA record for the MX host, and treats TLS as mandatory with that exact key.

A TLSA record for SMTP lives under the port and protocol of the MX host. The recommended form for SMTP is usage 3, selector 1, matching type 1:

_25._tcp.mx1.example.com.   IN   TLSA   3 1 1 <64-hex-char SHA-256 of the public key>

The four fields:

• Usage 3 (DANE-EE): trust the server's own certificate or key directly, as named by this record, with no requirement that a public CA chain validates. This is the standard choice for SMTP.
• Selector 1 (SubjectPublicKeyInfo): pin the public key rather than the full certificate, so the pin survives certificate renewal as long as the key pair stays the same.
• Matching type 1 (SHA-256): the association data is a SHA-256 hash, which is exactly 64 hexadecimal characters.

Because DANE is anchored in DNSSEC, the presence of the TLSA record alone tells the sender that TLS is mandatory. There is no plaintext fallback for a domain that publishes DANE, and a stripping attacker cannot remove the signal without invalidating the DNSSEC chain.

Choosing, and using both

MTA-STS and DANE solve the same problem with different roots of trust, and they coexide. Many operators publish both for defense in depth: senders that validate DNSSEC prefer DANE because the cryptographic binding is stronger, and fall back to MTA-STS where DANE is not available. The practical decision usually comes down to whether you can run DNSSEC. If your zone is signed, DANE gives you the strongest enforcement; if it is not, MTA-STS gives you enforcement anchored in the Web PKI without the DNSSEC dependency. Pair either with a TLS-RPT record so receiving domains send you daily reports of TLS failures, which is how you catch a stripping attempt or a broken certificate before it costs you mail.

Asked to explain SMTP encryption out loud, the answer that lands is: STARTTLS is opportunistic, so an on-path attacker can strip the TLS offer and force a silent downgrade to plaintext; MTA-STS publishes an HTTPS-anchored policy that makes TLS mandatory, and DANE publishes a DNSSEC-signed TLSA record that pins the key and makes TLS mandatory; you deploy one or both, plus TLS-RPT, to turn "encrypt if convenient" into "encrypt or do not deliver."