Lesson 001 · Linux MTA Operations Learning Path

Linux Mail Server Architecture: SMTP, DNS and the Complete Message Path

Anil Jalela · Published Sep 3, 2026 · 10 min read

A mail server is not one daemon and a successful SMTP banner is not a working mail platform. Delivery crosses public DNS, network policy, Postfix SMTP services, relay controls, identity checks, filters, queues, mailbox delivery and Dovecot access. This first lesson builds a testable map of that path. Every later configuration will attach to a named boundary in this model, so an operator can diagnose the first failing layer instead of restarting services or changing unrelated records.

Prerequisites and inherited lab checkpoint

Use two disposable RHEL 9 or RHEL 10 virtual machines on an isolated network. Name them mail1.example.test and client1.example.test. The reserved example.test namespace prevents accidental public delivery. Keep console access, a snapshot and a second privileged shell before changing network or mail policy.

No Internet-reachable SMTP service is required in this lesson. If the lab later receives public mail, obtain an approved domain, static address, reverse-DNS delegation and abuse contact first. Never copy production credentials, DKIM private keys or customer messages into the lab.

Install and prepare the required components

sudo dnf install -y bind-utils swaks tcpdump
mkdir -p "$PWD/mta-lab-evidence"
date -u --iso-8601=seconds
hostnamectl
cat /etc/redhat-release
ip -brief address
ip route show

• bind-utils supplies dig; swaks is a controlled SMTP test client when available from an approved repository. If dnf info swaks finds no package, use openssl s_client and Postfix tools later rather than downloading an unidentified script.
• tcpdump is installed for a later packet-level lab. Captures can contain addresses, credentials and message content, so restrict permissions and capture only the required interface, host and port.
• The evidence directory is unprivileged in this introductory lab. Later lessons use a root-owned directory because configuration and logs can expose sensitive operational data.

Write the architecture contract before touching Postfix

Create a short design record containing the accepted domains, outbound identities, mailbox format, authentication source, TLS names, delivery method, filtering boundary, monitoring owner and recovery objectives. A component is selected only after its responsibility is clear.

Draw separate trust zones for the public Internet, SMTP edge, local policy daemons, database, mailbox storage and administration network. Mark every socket as TCP or Unix, every process owner, and whether failure should temporarily defer mail, reject it, or permit it. “Accept on error” is not a harmless default when the failed service enforces identity or malware policy.

Verify the working path

dig +short MX example.test
dig +short A mail1.example.test
getent hosts mail1.example.test
ss -lntup
sudo firewall-cmd --get-active-zones
sudo firewall-cmd --list-all
sudo getenforce
sudo systemctl --failed

• The reserved domain should not unexpectedly resolve through public DNS. Local lab resolution must be explicitly documented through a private DNS zone or reviewed hosts entries.
• At this stage there should be no unexplained SMTP, submission, IMAP or POP listener. Record the baseline so later openings are attributable.
• SELinux should remain enforcing. A mail service that works only after disabling SELinux has an unresolved labeling, boolean or policy problem.
• A clean command does not prove end-to-end mail flow. These checks establish the starting state against which later changes are measured.

Production decisions before continuing

Separate protocol roles and trust decisions

An MTA accepts or relays messages using SMTP. Postfix performs that role and also manages queues. An MDA or local delivery service places accepted mail in the final mailbox; this course uses Dovecot LMTP. Dovecot also provides IMAP access and SASL authentication for message submission. DNS publishes routing and identity assertions, but DNS alone does not authorize relay or prove that a message is wanted.

SMTP has two different public-facing jobs. Port 25 is server-to-server transport and normally does not offer arbitrary user relay. Port 587 is authenticated message submission, requires encryption before credentials and applies sender authorization. Port 465 is implicit-TLS submission when the service is deliberately enabled. Treating all three listeners as one policy is a common route to credential exposure or open relay.

SPF evaluates the SMTP envelope domain against the connecting IP. DKIM verifies a cryptographic signature over selected headers and body content. DMARC asks whether the visible From domain aligns with a passing SPF or DKIM identity and applies the domain owner policy. These controls do not replace recipient validity, rate limits, malware filtering, complaint handling or reputation management.

Understand the component before configuring it

Build it step by step

1. Start with an envelope. Record connecting IP, HELO, MAIL FROM and RCPT TO separately from visible message headers.
2. Resolve the destination. Follow MX preference to A or AAAA records and test reachability to TCP/25 from the actual source network.
3. Classify the listener. Decide whether the connection is remote transport or authenticated submission before applying policy.
4. Make relay ownership explicit. Accept final local domains, authenticated submission and approved relay clients only; reject every unintended third-party path.
5. Attach policy in order. Preserve trustworthy upstream identity, calculate local results, filter content and record which component made each decision.
6. Follow the queue ID. Correlate smtpd acceptance, cleanup, queue manager, outbound smtp or LMTP delivery using one identifier.
7. Prove final state. Verify the recipient mailbox or remote SMTP response and retain evidence for delay, bounce and complaint analysis.

Operate and inspect the component

dig +noall +answer MX example.net
dig +noall +answer A mail.example.net
dig +noall +answer -x 192.0.2.25
nc -vz mail.example.net 25
openssl s_client -starttls smtp -connect mail.example.net:25 -servername mail.example.net -crlf
sudo tcpdump -ni any 'tcp port 25 or tcp port 587 or tcp port 993'

• Use documentation address ranges such as 192.0.2.0/24 in examples. Replace values only after the production change is reviewed.
• openssl s_client validates the TLS negotiation and presented chain; it does not prove relay policy, mailbox delivery or reputation.
• Unsafe: packet capture can collect message content and authentication material. Narrow the filter, duration and access, and securely remove the capture after approved retention.
• An SMTP transcript should retain commands, reply codes and queue ID but redact credentials, private message content and customer addresses.

Evidence and acceptance criteria

Trace one message instead of guessing

A remote system reports a temporary failure for [email protected]. DNS resolves correctly and TCP/25 reaches Postfix. The transcript shows recipient acceptance and returns a queue ID. Postfix logs then show LMTP returning a temporary storage error because the mailbox filesystem is read-only. Restarting Postfix would not repair that condition. The correct response is to protect the queue, investigate the filesystem, restore safe writes, retry one message and verify final delivery before flushing broader traffic.

The same method works outbound. Start with the submitted queue ID, inspect active or deferred state, read the remote enhanced status code, verify the next-hop DNS answer and only then decide whether the cause is local policy, network reachability, remote throttling or a permanent recipient failure.

Troubleshooting by symptom

Unsafe operations and recovery boundaries

• Unsafe: adding a broad subnet to trusted networks can create an open relay. Trust only owned, enumerated clients and prove a negative relay test.
• Unsafe: opening SMTP, submission or IMAP to the Internet before authentication, TLS and rate controls are verified exposes an unfinished service.
• Unsafe: deleting a deferred queue hides the remote response and can lose legitimate mail. Preserve representative evidence and classify messages first.
• Unsafe: accepting forged inbound Authentication-Results lets a sender claim its own SPF, DKIM or DMARC success. Trust only results from identified internal authorities.

Rewritten knowledge checks

What is the difference between an MTA and an MDA?

The MTA transports and queues mail using SMTP; the MDA performs final delivery into the recipient mailbox.

Why is port 587 not interchangeable with port 25?

587 is authenticated user submission with submission policy; 25 is primarily server-to-server transport and must not offer arbitrary relay.

Does an MX record authenticate a sender?

No. It routes inbound mail for a domain. Sender authentication uses SPF, DKIM and DMARC with distinct identities.

Which identity does SPF normally evaluate?

The RFC 5321 envelope sender domain, with HELO handling for a null reverse path.

What does DMARC alignment compare?

The visible RFC 5322 From domain with a passing SPF-authenticated or DKIM signing domain under relaxed or strict alignment.

Why retain the Postfix queue ID?

It correlates acceptance, cleanup, policy, queue movement and final delivery or remote response.

What proves a server is not an open relay?

A controlled unauthenticated attempt from an untrusted client to a non-local recipient is denied while legitimate paths continue to work.

What does a TLS handshake prove?

It proves properties of that encrypted connection and certificate validation performed by the client, not message authorization or inbox placement.

Why use a reserved domain in the lab?

It prevents examples from accidentally routing mail or publishing policy for a real organization.

What is the first step when a message is missing?

Obtain timestamps, envelope identities and queue ID, then find the first expected stage for which evidence is absent.

Cumulative lab checkpoint

1. Draw both inbound and submission paths with every DNS record, port, process, Unix socket, queue and mailbox boundary labelled.
2. Create a relay matrix for local domains, authenticated users, approved systems and untrusted Internet clients.
3. Record baseline listeners, firewall zones, SELinux mode and resolver behavior on both lab hosts.
4. Write positive and negative acceptance tests for SMTP transport, submission and mailbox access before installing services.
5. Define what evidence will be retained for a queue delay and how addresses or content will be redacted.
6. Snapshot both VMs and save the architecture contract as the inherited checkpoint for Lesson 2.

Primary references

• Postfix architecture overview
• SMTP specification RFC 5321
• Internet Message Format RFC 5322