MTA-STS and TLS-RPT in 2018: Enforceable Transport Security and Reporting

Anil Jalela · Published Sep 26, 2018 · 12 min read

In September 2018, the IETF published RFC 8461 for SMTP MTA Strict Transport Security and RFC 8460 for SMTP TLS Reporting. Together they addressed two weaknesses in ordinary opportunistic STARTTLS. MTA-STS lets a receiving domain declare which MX hosts are valid and require certificate-authenticated TLS from supporting senders. TLS-RPT gives the domain aggregate evidence about successful sessions and failures involving routing, STARTTLS, certificates, MTA-STS or DANE. One mechanism sets transport policy; the other makes transport failure observable. Neither authenticates message authorship, guarantees inbox placement or replaces SPF, DKIM and DMARC.

The dated mailbox-provider change

SMTP was designed to deliver mail across independently operated systems. STARTTLS later added encrypted transport, but ordinary opportunistic behavior prioritizes delivery. If TLS is unavailable or appears broken, a sending MTA may fall back to cleartext. That protects reachability but creates an opening for downgrade attacks and undetected configuration failure.

An active attacker can strip the STARTTLS advertisement or redirect DNS resolution toward an unauthorized MX. Routine operational problems can look similar: an expired certificate, incomplete chain, hostname mismatch, missing STARTTLS support or load balancer presenting the wrong identity. Without a declared policy, the sender has limited grounds to distinguish temporary failure from permitted fallback.

MTA-STS introduced a recipient-domain policy discovered through DNS and fetched over HTTPS. A cached enforcing policy tells a conforming sender not to deliver when the negotiated TLS path fails the policy. TLS-RPT introduced an aggregate JSON report so a receiving domain can learn how supporting senders experienced its transport controls.

The RFC record verifies September 2018 but not one shared day for both publications. This article therefore preserves month precision. The CMS date is an editorial scheduling choice and is not presented as the exact standards event.

How the system worked before the change

Before these standards, many transfers used STARTTLS successfully, but the receiver could not publish a broadly interoperable Web PKI policy that required supporting senders to authenticate its MX hosts. Successful encryption statistics also did not reveal whether another sender was falling back after a certificate or routing error.

Operators often tested with a single command against the primary MX. That could miss backup MX hosts, IPv6 listeners, regional load balancers, certificate rotation and intermittent DNS answers. One successful TLS handshake did not prove the full inbound estate was safe for enforcement.

Certificate monitoring focused on web endpoints while mail certificates expired unnoticed. A shared proxy could serve the right certificate on port 443 but the wrong chain on port 25. Because opportunistic delivery could continue, the incident might remain invisible to recipient operations.

DANE offered a DNSSEC-based transport-authentication path, but deployment required signed DNS and correct TLSA operation. Domains without that architecture needed another interoperable option. MTA-STS uses Web PKI and HTTPS, while TLS-RPT can report failures associated with either MTA-STS or DANE.

What changed on the provider side

RFC 8461 defined the DNS marker at _mta-sts.example.com, using v=STSv1 and an id value that changes when policy content changes. The actual policy is served from https://mta-sts.example.com/.well-known/mta-sts.txt. It declares a version, mode, one or more acceptable MX patterns and a cache lifetime.

The modes have different intent. testing asks supporting senders to evaluate and report policy failures without refusing delivery because of MTA-STS. enforce asks them to require a valid policy-matching TLS path. none signals policy removal, but cached-policy behavior and max-age still make planned transitions essential.

RFC 8460 defined _smtp._tls.example.com with v=TLSRPTv1 and one or more rua destinations using mailto or HTTPS. Supporting sending systems can deliver aggregate success and failure counts with policy, MX and result-type context.

The standards are complementary but independent. A domain can collect TLS reports before enforcing MTA-STS, and reporting does not make a broken policy safe. Deployment succeeds only when every advertised production MX presents a trusted, unexpired certificate whose identity matches policy, supports STARTTLS consistently and remains reachable.

Message path before and after

Before

Sending MTA resolves recipient MX
    |
    v
Connects and looks for STARTTLS
    |
    +----> TLS succeeds -> encrypted delivery
    +----> STARTTLS stripped or TLS fails -> opportunistic fallback may occur
    |
    v
Receiving domain may have little aggregate evidence of failure

After

Sending MTA checks _mta-sts DNS id and cached policy
    |
    +----> fetches HTTPS policy when required
    |
    v
Resolves MX and validates policy match + STARTTLS + certificate
    |
    +----> compliant path -> TLS delivery
    +----> testing failure -> delivery behavior plus TLS-RPT evidence
    +----> enforce failure -> defer, retry and do not downgrade
    |
    v
Aggregate JSON report sent to _smtp._tls rua destination

Who and what the change affected

Effect on delivery, placement and recipient visibility

MTA-STS can intentionally reduce immediate deliverability when the receiver’s secure path is broken. Under enforcement, a conforming sender should defer rather than downgrade to an unauthorized, unencrypted or certificate-invalid route. That delay is the security property, not evidence that filtering placed the message in spam.

A malformed policy can create a self-inflicted outage. Common causes include listing only the primary MX, using a pattern that does not match the actual host, presenting an incomplete certificate chain, failing IPv6, serving the policy with an invalid web certificate, or publishing enforcement before all regions are tested.

Marketing teams must understand queue lifetime. A short certificate incident can delay a time-sensitive campaign until after the offer has expired even if mail eventually delivers. Operational readiness therefore includes campaign pause authority, corrected landing-page terms and a decision about whether delayed messages remain appropriate.

The standards do not determine recipient wantedness. A perfectly authenticated TLS session can carry unsolicited mail, and a compliant receiving path can still place mail in spam. Keep transport security dashboards separate from complaint rate, IP and domain reputation, authentication alignment and folder evidence.

When diagnosis shows certificate-host-mismatch, certificate-expired, starttls-not-supported, sts-policy-invalid or policy-fetch failure, repair the exact transport layer. Rotating sending IPs or rewriting content does not correct the receiver’s certificate or policy.

Effect on measurement and diagnosis

TLS-RPT is aggregate telemetry. Count total-successful-session-count and total-failure-session-count within each policy and date range. Do not count JSON files as sessions, and do not assume reports cover every sender. Participation, aggregation windows and implementation details vary.

Failure result types can overlap, so totals require schema-aware interpretation. Preserve reporting organization, report ID, date range, policy domain, policy type, policy string, MX hostname, sending MTA IP and failure reason. Deduplicate before trending.

Use an implementation baseline before publishing testing mode. Measure STARTTLS availability, certificate validity, hostname coverage, TLS versions, all MX routes and all address families. After publishing TLS-RPT, compare external reports with internal handshakes and certificate monitors.

A report destination is an ingestion service, not an ordinary shared mailbox. Set message-size limits, gzip protection, JSON schema validation, authentication and safe parsing. Treat URLs and free-text failure details as untrusted evidence. Restrict retention because reports can expose mail-flow relationships and infrastructure identifiers.

Define rollout gates. For example, require a sustained interval with zero unexplained policy failures across material senders and every production MX before enforcement. State the denominator, observation window and exceptions. A green test from one large sender is not sufficient evidence for the full ecosystem.

Advantages for email marketers

Disadvantages and operational risks

What email teams needed to do at the time

1. Inventory every inbound MX. Include backups, regions, third parties, IPv4 and IPv6.
2. Validate SMTP TLS. Test STARTTLS, trust chain, expiry and hostname matching on every route.
3. Create the policy HTTPS host. Serve only the required well-known policy with reliable Web PKI.
4. Publish TLS-RPT first. Build external evidence before changing delivery requirements.
5. Begin in testing mode. Observe policy failures without making MTA-STS the cause of deferral.
6. Reconcile reports. Explain every material failure and reporter difference.
7. Set enforcement gates. Require sufficient duration, coverage and rollback readiness.
8. Change the policy id deliberately. Tie every id to reviewed content and a change record.

What email teams should do now

1. Maintain policy as code. Generate MX entries from a reviewed receiving-estate inventory.
2. Monitor both certificates. Check SMTP MX certificates and the mta-sts HTTPS certificate.
3. Test all network paths. Include IPv6, backup MX and regional endpoints after each change.
4. Operate a hardened report pipeline. Validate MIME, gzip, JSON size and schema safely.
5. Correlate reports with deployments. Annotate DNS, load balancer and certificate changes.
6. Preserve raw and normalized evidence. Make parser corrections auditable.
7. Separate transport from placement. Use dedicated reputation and inbox diagnostics.
8. Review max-age as risk. Long caches strengthen continuity but lengthen recovery from mistakes.
9. Exercise rollback. Test policy change, id update, DNS and queued-mail recovery before incidents.
10. Evaluate DANE separately. Use DNSSEC and TLSA expertise rather than assuming MTA-STS equivalence.

Worked deliverability scenario

A retailer hosts inbound mail on two primary MX systems and one disaster-recovery MX. The primary routes pass STARTTLS tests, so the team publishes an enforcing policy that lists only those two hosts. A week later routing shifts to disaster recovery during maintenance.

Supporting senders resolve the advertised backup, but its hostname is absent from policy and its certificate covers an internal name. Under enforcement, mail is deferred. Marketing initially calls the event a sender-reputation block because a campaign confirmation is late.

TLS-RPT data shows policy and certificate failures against the backup MX. SMTP logs confirm that spam filtering never accepted the messages. The receiving team installs the correct certificate, adds the verified backup pattern, changes the policy id and tests both address families. Queued mail then succeeds on a policy-compliant TLS path.

The post-incident control generates MTA-STS policy from the same reviewed inventory used for MX changes. Certificate expiry and hostname coverage are checked continuously. Maintenance cannot advertise a new route until testing mode evidence and active probes pass. Marketing receives a transport-delay status distinct from inbox-placement reporting.

Evidence and diagnostics

• DNS marker: exact _mta-sts TXT version, id, TTL and authoritative response.
• HTTPS policy: URL, status, redirects, MIME, certificate, content, fetch time and cache age.
• Policy fields: version, mode, every MX pattern and max-age.
• MX inventory: preference, hostname, addresses, region, owner and change ticket.
• SMTP TLS: STARTTLS advertisement, negotiated version, cipher, chain, trust and name match.
• TLS-RPT DNS: _smtp._tls version, rua destinations and authorization where applicable.
• Report evidence: reporter, report ID, interval, policy, successful and failed sessions, result type.
• Queue evidence: complete SMTP response, first failure, next retry, age and eventual outcome.

Failure modes and incorrect conclusions

• Calling MTA-STS end-to-end encryption. It protects supporting server-to-server hops, not message storage or every downstream path.
• Calling TLS-RPT an enforcement mechanism. It reports observations; it does not require secure delivery.
• Publishing enforce after one MX test. Backup and IPv6 routes can still fail.
• Listing recipient domains instead of MX patterns. Policy matching applies to expected mail exchangers.
• Changing policy without changing id. Senders can continue using cached content.
• Using report absence as success. The sender may not report or ingestion may be broken.
• Repairing transport with reputation changes. IP rotation cannot fix an expired receiver certificate.
• Equating MTA-STS with DANE. Their trust and DNS security models differ.

Current status and superseding changes

RFC 8460 and RFC 8461 remain active IETF Proposed Standards. Major hosted-mail and security platforms support deployment, but reporting and enforcement coverage is not universal. Operators must retain internal monitoring and cannot outsource proof entirely to aggregate reports.

MTA-STS still uses the DNS id plus HTTPS policy architecture defined in 2018. The initial discovery model has known limitations when an attacker is present before a sender has cached a valid policy; long-lived valid cache and secure operations improve continuity. DANE provides a different DNSSEC-based model.

TLS-RPT remains valuable beyond MTA-STS because it can describe DANE and STARTTLS failures. Report processing needs the same discipline applied to DMARC aggregate ingestion: validate untrusted input, retain raw evidence, normalize counts correctly and record coverage.

The current operating principle is staged enforcement. Build a complete MX and certificate inventory, publish reporting, observe testing mode, repair every meaningful path, rehearse recovery, and only then enforce. Treat secure deferral as a designed outcome while maintaining communication with business teams whose time-sensitive mail may be delayed.

Operator checklist

• Record both RFCs at September 2018 precision without inventing a common day.
• Publish _mta-sts and _smtp._tls as separate correctly scoped TXT records.
• Serve the policy at the exact HTTPS well-known path.
• List every valid production MX pattern.
• Validate trusted matching SMTP certificates on every route and address family.
• Collect and normalize TLS reports before enforcement.
• Define measurable testing-to-enforce gates.
• Change the policy id whenever policy content changes.
• Monitor SMTP and HTTPS certificates independently.
• Document max-age-aware rollback and queued-mail recovery.
• Keep transport-security and inbox-placement dashboards separate.

Primary and contemporaneous references

• RFC Editor: RFC 8460 SMTP TLS Reporting: Primary September 2018 standard.
• RFC Editor: RFC 8461 SMTP MTA Strict Transport Security: Primary September 2018 standard.
• Google Workspace: About MTA-STS and TLS reporting: Current implementation overview.
• Google Workspace: Set up MTA-STS: Current deployment and validation guidance.
• Google Workspace: Turn on MTA-STS and TLS reporting: Current DNS and reporting guidance.

Related NitWings guides

• MTA-STS Email Security
• TLS-RPT Email Reporting
• Email Authentication Guide
• DMARC Aggregate Reports
• Email Blocklists and Delisting