MTA-STS Explained: How to Force TLS on Inbound Email and Why It Matters

The Vulnerability of Opportunistic TLS: Why STARTTLS Isn't Enough

To understand MTA-STS, you must first understand the fundamental weakness of standard SMTP. When one mail server connects to another, they perform a "Handshake." By default, SMTP is a plain-text protocol. To encrypt the connection, the receiving server says "I support STARTTLS." The sending server sees this and upgrades the connection to encrypted TLS. However, because this happens mid-stream, it is vulnerable to a "Man-in-the-Middle" (MITM) attack.

An attacker sitting on the network between the two servers can "strip" the STARTTLS command from the conversation. The sending server then believes that the receiver doesn't support encryption and falls back to plain-text delivery. This is known as a Downgrade Attack. The email is delivered successfully, so the user sees no error, but the attacker has successfully intercepted the message in clear text. This vulnerability exists because there is no "Pre-Verified" list of servers that require encryption.

• Passive Eavesdropping: Without forced TLS, any government or malicious actor with access to internet backbone routers can read your sensitive business communications in bulk.
• DNS Spoofing: An attacker can redirect your MX traffic to their own server. Without MTA-STS, the sending server might accept a fake certificate and deliver the mail to the attacker.
• Legacy Fallback: Many mail servers emphasize "Delivery at all costs" over "Security," meaning they will always choose unencrypted delivery over a failed handshake unless explicitly told otherwise.

MTA-STS (RFC 8461) was designed specifically to eliminate this "Opportunistic" behavior. It provides a mechanism for a domain owner to publish a policy that officially declares: "My mail servers always support TLS, and you should reject the email if you cannot establish a secure, verified connection." This turns the "Should we encrypt?" question into a "Must we encrypt?" mandate, protecting your organization from the vast majority of transport-layer intercept attacks.

Implementation of MTA-STS is now a baseline requirement for high-security environments and highly regulated industries. As major providers like Google and Microsoft have fully adopted the standard, the "Shield" provided by MTA-STS now covers an increasingly large portion of global email traffic. By deploying it, you ensure that your inbound communications are never exposed to legacy plain-text vulnerabilities that have plagued the SMTP protocol for decades.

The Architecture of MTA-STS: Closing the Downgrade Gap

MTA-STS works by providing a "Trusted Source of Truth" outside of the SMTP handshake itself. It uses two distinct components to verify a connection: a DNS TXT Record and an HTTPS Policy File. This dual-layer approach is critical because it prevents an attacker from simply spoofing your DNS to disable your security. The DNS record signals that a policy exists, but the actual security rules are fetched over a secure HTTPS connection to a specific subdomain (mta-sts.yourdomain.com).

When a sending mail server (like Gmail) wants to send you an email, it first looks for your _mta-sts TXT record. This record contains a version ID. The sender compares this ID to its local cache. If the ID is new, or if it doesn't have a cached policy, it fetches the policy from your web server. This policy lists your authorized MX hostnames and tells the sender whether to enforce the encryption or just "test" it. Once the policy is cached, the sender will refuse to talk to your mail server unless the TLS certificate matches your policy.

; The MTA-STS Verification Flow:
1. Sender queries DNS for _mta-sts.example.com.
2. Sender fetches policy from https://mta-sts.example.com/.well-known/mta-sts.txt.
3. Sender verifies your MX server's TLS certificate against the policy.
4. If the certificate is invalid or missing, the sender blocks the delivery.

The "Strict" part of MTA-STS comes from the Max Age parameter in the policy file. This tells the sender how long to remember your security rules. Once a sender like Google Workspaces caches your "Enforce" policy, if an attacker tries to perform a downgrade attack tomorrow, Google will remember that you must have TLS. It will see the missing STARTTLS command, realize it's an attack, and stop the delivery. The attacker cannot "un-ring the bell" once the policy has been cached.

This architecture solves the core problem of "Initial Trust." By using the existing, highly secure infrastructure of the Web (HTTPS) to secure the legacy infrastructure of Email (SMTP), MTA-STS allows the two systems to work in tandem. This cross-pollination of trust is why MTA-STS has been so successful where previous attempts at SMTP security failed—it leverages the ubiquitous security of SSL certificates to protect every hop of the email delivery path.

Policy Configuration: Modes, Syntax, and Host Verification

The heart of your deployment is the mta-sts.txt policy file. This is a simple, plain-text file that must be hosted at a very specific "Well-Known" URL: https://mta-sts.yourdomain.com/.well-known/mta-sts.txt. The use of the mta-sts subdomain is mandatory; you cannot use a different name. The file must be served with a text/plain MIME type and must be accessible over a valid, CA-signed HTTPS connection.

There are three primary Modes you can set in your policy. Selecting the right mode is the difference between a successful rollout and a catastrophic mail outage. You should always move through these modes in sequence. Never jump straight to "Enforce" without first observing your traffic in "Testing" mode. This allows you to catch certificate mismatches or legacy sender issues before they cause actual bounces.

As shown in the table, the testing mode is your safety net. It allows you to "Dry Run" your security. Senders will behave as if encryption is mandatory, but if the handshake fails, they will deliver the mail anyway and send you an automated report (via TLS-RPT) explaining why it failed. This is how you discover that your old backup mail server is still using an expired certificate or that your policy doesn't include a specific MX hostname that you forgot about.

The policy syntax also requires an mx tag for every authorized mail server. These are the hostnames that the sender will look for in your TLS certificate's "Subject Alternative Name" (SAN) field. If your MX is mail.example.com but your certificate is only for example.com, MTA-STS will fail. You can use wildcards (like *.example.com) to cover multiple servers, but you must ensure that your physical certificate matches whatever pattern you publish in your policy.

TLS-RPT: Establishing a Reporting and Feedback Loop

MTA-STS is practically blind without its companion record: TLS-RPT (TLS Reporting). While MTA-STS tells senders what to do, TLS-RPT tells them where to report failures. If a sender like Gmail tries to deliver an encrypted email to you but your certificate fails the MTA-STS check, you would never know why the mail was missing without TLS-RPT. The sender will send you a daily JSON report detailing the successful and failed connections to your infrastructure.

The TLS-RPT record is a simple DNS TXT record placed at _smtp._tls.yourdomain.com. It contains a mailto: or https: address where these reports should be delivered. Because reading thousands of JSON reports manually is impossible, most organizations use a "DMARC and TLS Reporting Service" to aggregate this data into a visual dashboard. This allows you to see at a glance if your global deliverability is healthy or if a specific geography is experiencing network attacks.

• Pre-Enforcement Visibility: Deploy TLS-RPT before you even create your MTA-STS policy. It will show you your current "Opportunistic TLS" success rate immediately.
• Error Breakdown: Reports identify exactly why a connection failed: was it a certificate.expired, a hostname-mismatch, or a negotiation-failure?
• Global Coverage: TLS-RPT reports include data from every provider that supports the standard, giving you a worldwide view of your mail server's accessibility.

Monitoring these reports is especially critical for brands with multiple MX providers. If you use a cloud provider like Microsoft 365 as your primary but have a legacy on-premise server for specific apps, TLS-RPT will show you if one of those endpoints is "weaker" than the other. It acts as an early warning system; if you see a spike in "Expired Certificate" reports, you have a few days to fix the issue before the "Max Age" of the cached policy causes mail to start bouncing for real.

Without TLS-RPT, an enforce policy is a "Black Box." If your mail flow drops, you have no way to distinguish between a routing problem, a network outage, or a security block. By coupling the two standards, you gain "Observability" into the transport layer of your email system—a level of transparency that was previously only available to the largest ISPs in the world. For a complete look at these reports, see our Guide to TLS-RPT Analytics.

MTA-STS vs. DANE: Comparing SMTP Security Standards

Before MTA-STS became the standard, another protocol called DANE (DNS-Based Authentication of Named Entities) was the primary attempt at securing SMTP TLS. DANE is a powerful standard that allows you to store your certificate "fingerprint" directly in your DNS (via TLSA records). However, DANE has a very high barrier to entry: it requires DNSSEC to be perfectly implemented across your entire domain. Because DNSSEC is complex and error-prone, DANE adoption has lagged behind.

MTA-STS was created as a "Web-Friendly" alternative to DANE. It relies on the existing Certificate Authority (CA) system—the same one that secures your website—rather than relying on DNSSEC signatures. This makes it much easier for most organizations to deploy. While DANE is technically "more secure" because it doesn't rely on the CA system, MTA-STS is more "practically secure" because it is actually supported by the major email providers that handle the bulk of the world's traffic.

As shown in the table, the "Support" column is the deciding factor for most brands. Google and Microsoft both fully support MTA-STS for their hosted mail products, but neither currently supports DANE for inbound mail. If your goal is to protect your communication with the largest number of partners, MTA-STS is the logical choice. DANE is increasingly popular in highly regulated government sectors, particularly in Europe, where full DNSSEC sovereign control is a requirement.

It is possible (and often recommended) to implement both. They are not mutually exclusive. If you have a perfectly signed DNSSEC zone, you should publish TLSA records for DANE and an MTA-STS policy for the rest of the world. This "Defense in Depth" strategy ensures that whether your partner supports the DNSSEC model or the HTTPS model, your communication remains encrypted and protected against downgrade attacks.

Implementation and Verification Workflow

Deploying MTA-STS is a four-step process. First, you must ensure your MX server's TLS certificate is valid and covers its own hostname. Second, you create the mta-sts subdomain and host your mta-sts.txt policy file. Third, you publish your _mta-sts DNS TXT record with a unique id. Finally, you observe your TLS-RPT reports to ensure that no legitimate traffic is failing before you lock the door with "Enforce" mode.

Verification is the most critical part of the workflow. You should use a combination of local CLI tools and external auditing services. A manual curl command to your policy URL will confirm that the file is reachable and has the correct line endings (CRLF is required by some parsers). A dig command will confirm that your DNS record is live. However, the most authoritative test is our MTA-STS Lookup Tool, which performs a full end-to-end "Handshake Simulation" to ensure all three layers (DNS, HTTPS, and SMTP) are in perfect alignment.

# Verify your policy file from the command line
curl -v https://mta-sts.yourdomain.com/.well-known/mta-sts.txt

# Expected Output:
# < HTTP/1.1 200 OK
# < Content-Type: text/plain; charset=utf-8
# version: STSv1
# mode: testing
# ...

• Avoid Redirects: Ensure your mta-sts subdomain doesn't redirect to your main site home page. The sender's proxy will not follow redirects that move to a different hostname for security reasons.
• Check Certificate Chain: When testing your MX server (on port 25), ensure your server is sending the Full Certificate Chain. If you miss the "Intermediate" certificate, many MTA-STS senders will fail the validation check even if your primary certificate is valid.
• Match Hostnames Exactly: If your MX is mx1.emails.com, your policy must say mx: mx1.emails.com. It cannot just say mx: emails.com.

Common deployment failures usually stem from the "Bootstrap" nature of the standard. If your mta-sts subdomain has an expired SSL certificate, the sender cannot fetch the policy to know that your mail server's certificate is valid. This "Certificate Inception" means you must be diligent about monitoring the SSL for your web server as much as your mail server. We recommend using our SSL Expiry Monitoring tool specifically for your mta-sts subdomain to prevent a single $10 web certificate from blocking all of your corporate email.

For brands using cloud providers like Google Workspace or Office 365, the process is simplified. The mail servers are already secured; you just need to point your policy hostnames to the respective cloud endpoints (e.g., *.aspmx.l.google.com). This allows small businesses to gain "Enterprise-Grade" transport security without needing to manage their own local mail server infrastructure. Once deployed, leave your policy in testing mode for at least 14 days of clean reports before committing to enforce.

Frequently Asked Questions

Next Steps

Verify your current MTA-STS status and analyze your inbound TLS health using our MTA-STS Lookup tool. This will give you a complete picture of your DNS record, policy file availability, and MX certificate alignment.

If you don't yet have reporting configured, deploy a TLS-RPT record today. This is a low-risk, high-reward step that provides immediate visibility into your mail flow's security. For more on the foundational security layers, see our guides on DMARC Enforcement and DKIM Management.

Browse all Email Guides on DNSnexus for related topics on transport security and certificate chain validation.

Related Guides

• DMARC Policy Enforcement: Moving from none to quarantine to reject Safely
• TLS Certificate Chain Validation: How to Fix Incomplete Chains
• SSL Certificate Expiry Monitoring: Preventing Unexpected Outages
• SPF Record Explained: Setup, Syntax, and the 10-Lookup Limit
• DNS TXT Records Explained: Why Every Standard Uses Them