+91 9619904949

05-Email Optimizing Bounce Handling For Marketers

In email marketing, maintaining a low bounce rate is crucial for deliverability, sender reputation, and the overall success of email campaigns. Bounces occur when an email fails to reach the intended recipient, leading to lost opportunities for engagement.

There are two types of bounces: hard bounces (permanent failures) and soft bounces (temporary issues). If these bounces aren’t handled properly, they can significantly affect email deliverability, damage IP and domain reputation, and reduce the effectiveness of marketing efforts.

This case study explores a scenario where an email marketing team at an e-commerce company struggled with high bounce rates, particularly after launching a series of new promotional campaigns. The goal was to improve bounce handling practices, reduce bounce rates, and enhance overall deliverability.

Objective:

To reduce bounce rates and improve email deliverability by optimizing the management of bounced emails, ensuring list hygiene, and enhancing email campaign strategies.

Initial Challenges:

The marketing team faced several issues that contributed to high bounce rates:

Poor List Hygiene: The team had not cleaned their email list regularly, resulting in a significant number of invalid email addresses.
Insufficient Bounce Management: Hard bounces were not removed promptly, and soft bounces were being retried too frequently, leading to repeated delivery failures.
Lack of Authentication: The emails lacked proper authentication protocols, such as SPF and DKIM, causing many ISPs to reject them or flag them as suspicious.
Content Triggers: Certain campaigns had high bounce rates due to content flagged by spam filters, such as overly promotional language and excessive use of images.

Step-by-Step Approach to Optimize Bounce Handling:

Error Identification Process:

  • Bounce Codes: Analyze bounce codes provided by ISPs. These codes indicate specific issues, such as invalid addresses (hard bounce), temporary server issues (soft bounce), or content-related rejections.
  • Authentication Failures: Monitor failures related to SPF, DKIM, and DMARC authentication. Failures in these protocols may result in delivery rejections.
  • Feedback Loops (FBL): Set up FBLs with major ISPs to receive spam complaint data. High complaint rates are a warning sign of errors in targeting or list quality.
  • ISP-Specific Issues: Monitor inbox placement reports from ESPs and tools like Return Path to check if any ISPs are particularly resistant to your emails.
  • Domain Reputation: Use reputation monitoring tools (e.g., Google Postmaster, SenderScore) to identify domain or IP reputation issues.
  • Spam Filters: Check if your emails are being flagged by spam filters due to content triggers, poor formatting, or overly promotional language.

1. Email List Hygiene

Problem: The marketing team was sending emails to an outdated list containing inactive, invalid, and misspelled addresses.
Solution:
=> List Cleaning: The team conducted an in-depth list cleaning process using email verification tools like ZeroBounce and NeverBounce to remove invalid and undeliverable addresses. This helped reduce hard bounces immediately.

=> Double Opt-In: They implemented a double opt-in process for new subscribers to ensure email validity from the start, reducing the chances of fake or incorrect email addresses entering the list.

Outcome: Hard bounce rates dropped by 50% after the first round of list cleaning, leading to an immediate improvement in sender reputation.

2. Hard and Soft Bounce Management

Problem: The team was retrying failed email deliveries excessively, especially for soft bounces, which annoyed some ISPs and further damaged sender’s reputation.

Solution:
=> Hard Bounce Removal: They set up automated processes to remove hard bounces immediately after the first occurrence, ensuring they weren’t included in future sends.
=> Soft Bounce Handling: Soft bounces were monitored more closely, with a threshold set to retry emails only twice. After three soft bounces over consecutive campaigns, the email addresses were moved to a suppression list.

    Threshold: Depending on your email sending frequency and strategy, the exact number may vary, but 3 to 5 soft bounces is a common rule of thumb.
ISP Guidelines: Some ISPs may have their own soft bounce limits, and monitoring the responses can help you fine-tune your threshold.

Best Practices:

  • Use a Bounce Management System: Centralized bounce handling can be done using email service provider (ESP) tools that automatically capture, analyze, and report bounce codes.
  • Real-Time Bounce Tracking: Implement real-time tracking for bounces so that your system can immediately process and react to bounce types (soft vs. hard).
  • Create Suppression Lists: Set up centralized suppression lists for hard bounces, invalid addresses, and users who have marked emails as spam. This ensures that bounces are managed across all campaigns and tools.
  • Consolidate Bounce Logs: Integrate your bounce logs from multiple sending platforms to avoid duplication and ensure every email address is handled properly across campaigns.
  • Monitor Feedback Loops (FBL): Centralize spam complaint data from ISPs to ensure prompt removal of flagged addresses.

Outcome: The team saw a significant reduction in the volume of emails sent to non-deliverable addresses, and overall soft bounce rates fell by 30%.

3. Improving Authentication Protocols

Problem: A lack of proper email authentication led ISPs to reject or filter out legitimate emails.
Solution:
SPF and DKIM: The team implemented SPF and DKIM authentication to prove that emails were being sent by authorized servers and hadn’t been altered during transit.
DMARC Policies: They also configured DMARC policies to further enhance security and provide reporting on email authentication issues.

Outcome: With SPF, DKIM, and DMARC properly set up, bounce rates related to authentication failures were minimized, and deliverability improved by 15%.
4. Content Optimization

Problem: Some campaigns triggered spam filters due to poor content choices, such as using all caps in subject lines, too many promotional keywords, and unoptimized images.
Solution:
Subject Line Testing: The team ran A/B tests to find more balanced and effective subject lines that didn’t trigger spam filters.
HTML Optimization: They optimized the HTML structure of their emails, reducing image-heavy content and ensuring the code was clean and responsive.
Avoiding Spammy Language: The team reduced the use of overly promotional words and phrases like “FREE,” “BUY NOW,” and “LIMITED TIME,” which often triggered spam filters.

Outcome: After optimizing content, spam complaints decreased, and soft bounces related to content issues dropped by 20%.
5. Throttling and Sending Practices

Problem: Sending too many emails at once was overwhelming some ISPs, resulting in delivery blocks.
Solution:
Throttling: The team introduced email throttling to gradually send emails, preventing large volumes from being sent in a short period.
Segmentation: By segmenting their audience, they prioritized sending emails to the most engaged users first, which improved their overall sender reputation.

Segmentation Rating System & Strategies:

  • Engagement Metrics: Use engagement rates like open rates, click-through rates, and conversion rates to assign scores to each email address.
    • High-engagement users (e.g., those who open or click frequently) should be rated highly.
    • Low-engagement or inactive users can be assigned a lower score or flagged for a re-engagement campaign.
  • Bounce and Complaint History: If an email address has a history of bounces or spam complaints, it should be given a low rating or suppressed altogether.
  • Segmentation: Create separate ratings for each list segment based on factors such as the age of the list, source (organic vs. purchased), and engagement history.
  • List Age: Older lists that haven’t been cleaned or engaged with in a while may warrant a lower rating.
  • Assign Ratings:
    • A-Rating: Highly engaged and active users.
    • B-Rating: Moderately engaged, possible re-engagement candidates.
    • C-Rating: Inactive or unengaged users, likely to be pruned or re-engaged.

Segmentation Strategies:

  • Engagement-Based Segmentation:
    • Active Subscribers: Create a segment of users who frequently open or click emails. These are your most valuable subscribers and should be targeted with more frequent or personalized content.
    • Inactive Subscribers: Segment users who haven’t opened or clicked an email in a defined time frame (e.g., 6 months). You can send them a re-engagement campaign or move them to a suppression list if they remain inactive.
  • Demographic-Based Segmentation:
    • Use demographic data like location, gender, or age to send targeted offers. For example, if you’re marketing a retail brand, you can send location-specific promotions.
  • Behavioral Segmentation:
    • Purchase History: Segment users based on their purchase behavior. Send follow-up emails, upsell offers, or loyalty rewards based on past purchases.
    • Browsing Activity: For e-commerce businesses, segment users based on their browsing behavior on your website (e.g., sending product recommendations based on recently viewed items).
  • Content Preferences: Based on user preferences (from past interactions or surveys), send segmented content that matches their interests, whether it’s product-focused, informative, or educational.
  • Re-Engagement Segmentation: Create segments specifically for re-engagement campaigns targeting users who have not interacted in a certain period.

 By using segmentation, you’ll improve the relevance of your emails, reduce bounce rates, and enhance engagement. Well-segmented campaigns also help ISPs recognize your emails as valuable and legitimate, boosting your deliverability.

Outcome: Throttling reduced ISP-related blocks, and segmentation ensured better engagement, further improving sender reputation and reducing bounces.

Results:
By implementing these bounce-handling optimizations, the email marketing team was able to achieve the following:
Reduced Hard Bounce Rates by 50%: Thanks to improved list hygiene and prompt hard bounce removal.
Lowered Soft Bounce Rates by 30%: Through better handling of soft bounces and limiting retries.
Increased Deliverability by 20%: Authentication improvements and content optimization helped emails bypass spam filters and reach the inbox.
Boosted Sender Reputation: Throttling, proper segmentation, and feedback loop monitoring led to fewer complaints and higher engagement rates.

Conclusion:

This case study demonstrates the importance of effective bounce handling and how it can significantly impact email marketing performance. By focusing on list hygiene, proper bounce management, authentication, and content optimization, the marketing team not only reduced bounces but also improved deliverability and engagement. Email marketers should continually monitor their bounce rates, sender reputation, and email content to maintain a healthy email marketing program.

Key Takeaways:

List hygiene is critical to reducing bounces and maintaining a clean email list.
Hard and soft bounce handling should be automated and managed carefully to avoid damage to sender’s reputation.
Authentication protocols (SPF, DKIM, DMARC) are essential for gaining ISP trust and improving email deliverability.
Content optimization helps prevent spam filtering and keeps bounce rates low.
Throttling and segmentation can reduce delivery blocks and improve engagement.

By following these best practices, email marketers can ensure a more effective, high-deliverability email strategy.

Great DB Mongo

Great DB Mongo

There are more than 379 database servers in use around the world today. Among them, MongoDB stands out as a top performer, surpassing databases like HBase, Neo4j, Riak, Memcached, RavenDB, CouchDB, and Redis. Tech giants like Google, Yahoo, and Facebook rely on MongoDB in their production environments.In the DB-Engines ranking, MongoDB holds the 5th position overall, following:

Oracle
MySQL
Microsoft SQL Server
PostgreSQL
MongoDB

Notably, MongoDB is also ranked as the number one NoSQL database.

RDBMS Mongo
Database database
table Collection
Record Document
Joins Embedded Object/Document

Mongo works perfectly with most all programming languages.
Also, mongo works with Windows and Linux with the same performance and without issues.
Mongo provides Replication, Sharding, Aggregation, and indexing features.
Mongo is an object-oriented and schema-less database.

Mongo is based on JavaScript, and all documents (Records) are presented in JSON format. Also, in Mongo, everything is an object. In Mongo 1st field, the compulsory field is _id which is not skippable.

{
_id: numeric or numeric-alphabetical string or,it set automatically. Mongo id is a 12-byte Object id that is a 4-byte time-stamp with 5 bytes of any random value and 3 bytes of a counter value.
}

mongo document stores like
the document content single field, array, sub-document (join), or, an array of sub-document.
{
_id: 1
First name: Anil,
middle name : Jasvantray,
last name: Jalela
Mobile: [9619904949,2573335,2567580]
}

install mongo:-

1 add Repo echo “deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/6.0 multiverse” > /etc/apt/sources.list.d/mongodb-org-6.0.list
2 add Key wget -qO – https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add –
3 update package list apt-get update
4 install mongo server apt-get install -y mongodb-org && apt-get install mongodb-org-server
5 change dir for modification cd /etc/
6 rename conf mv mongod.conf mongod.conf_org
7 vi mongod.conf  and add the below content into this.
8 # mongod.conf

# for documentation of all options, see:
# http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
# engine:
# wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0

# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

security:
  keyFile: /etc/keyfile-mongo
authorization: enabled

#operationProfiling:

#replication:
#  replSetName: “election01”

#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

9 kerate key file on master for replica set openssl rand -base64 756 > /etc/keyfile-mongo
10 change permission chmod 600 /etc/keyfile-mongo && ll /etc/keyfile-mongo
11 change onership of files chown mongodb:mongodb /etc/mongod.conf /etc/keyfile-mongo
12 start  mongo sudo systemctl start mongod
13 start mongo on system boot sudo systemctl enable mongod
14 check mongo status sudo systemctl status mongod
15 mongo login command mongosh
16 use mongo database use admin
17 set mongo password db.createUser(
{
user: “mongoadmin”,
pwd: passwordPrompt(),
roles: [ { role: “root”, db: “admin” }, “readWriteAnyDatabase” ]
}
)
18   mongosh –username=mongoadmin –password=yourpass –authenticationDatabase admin
19 create database  use nitwings
20 drop database “use nitwings” and then “db.dropDatabase()”
21 create database-specific user db.createUser({
user: “blackpost”,
pwd: passwordPrompt(),
roles: [
{ role: “readWrite”, db: “nitwings” }
],
mechanisms: [“SCRAM-SHA-256”],
authenticationRestrictions: [
{
clientSource: [“0.0.0.0/0”]
}
]
})
22 drop user “use admin”  and then db.dropUser(“blackpost”);
23 show users
db.getUsers()
24 create collection use nitwings
db.createCollection(“testCollection”) 
25
Insert One Document into collection
db.testCollection.insertOne({ name: “test”, value: 123 })
26 Insert many Document into collection
db.testCollection.insertMany([{ name: “blackpost”, value: 789 }, { name: “nitwings”, value: 456 }])
27
create Index
db.testCollection.createIndex({ name: 1 });
28 check index created or not for collection
db.testCollection.getIndexes()
29 find document
nitwings> db.testCollection.findOne({ name: “blackpost” })
{
_id: ObjectId(’66d485a63c3f4eb31f5e739c’),
name: ‘blackpost’,
value: 789
}
30 update document
nitwings> db.testCollection.updateOne({ name: “blackpost” }, { $set: { name: “aniljalela” } })
{
acknowledged: true,
insertedId: null,
matchedCount: 1,
modifiedCount: 1,
upsertedCount: 0
}
nitwings> db.testCollection.findOne({ name: “aniljalela” })
{
_id: ObjectId(’66d485a63c3f4eb31f5e739c’),
name: ‘aniljalela’,
value: 789
}
nitwings>
31 delete document
nitwings> db.testCollection.deleteOne({  name: “aniljalela” })
{ acknowledged: true, deletedCount: 1 }
nitwings>
  show collections and drop collection
nitwings> show collections

nitwings> db..drop()

32 backup database mongodump –db nitwings –out /opt/backup/ –username mongoadmin –password yourpass –authenticationDatabase admin
33 backup database from remote mongodump –host 10.10.10.10 –port 27017 –db nitwings –out /opt/backup/ –username mongoadmin –password yourpass  –authenticationDatabase admin
34 mongo all database mongodump –out /backups/all_databases_backup –username mongoadmin –password yourpass –authenticationDatabase admin

 

35 mongo all database from remote  mongodump –host 10.10.10.10 –port 27017 –out /path/to/backup /opt/backup/ –username mongoadmin –password yourpass –authenticationDatabase admin
36 restore database dump
mongorestore –host 10.10.10.10 –port 27017 –db nitwings –username mongoadmin –password yourpass –authenticationDatabase admin /opt/backup/nitwings
37  drop existing DB and restore db
mongorestore –host 10.10.10.10 –port 27017 –db nitwings –username mongoadmin –password yourpass –authenticationDatabase admin –drop /opt/backup/nitwings
38  restore all databases
mongorestore –host 10.10.10.10 –port 27017 –username mongoadmin –password yourpass –authenticationDatabase admin /opt/backup/
39
drop and restore all databases
mongorestore –host 10.10.10.10 –port 27017 –username mongoadmin –password yourpass –authenticationDatabase admin –drop /opt/backup/
40  restore specific collection
mongorestore –host 10.10.10.10 –port 27017 –db nitwings –collection –username mongoadmin –password yourpass –authenticationDatabase admin /opt/backup/nitwings/.bson

 

If the –drop option is not used with mongorestore, MongoDB restores data without dropping existing collections. If a collection already exists, mongorestore merges the backup with the current data. Documents with the same _id are not overwritten; instead, they are skipped to avoid duplicates. New collections from the dump are created if they don’t exist. This approach can lead to inconsistent or duplicated data, especially if the data structure has changed since the backup was created, potentially causing incorrect query results. Indexes are restored as in the dump, but existing indexes are not recreated, and mismatched index specifications may cause the restore to fail.

Replica set:-
1.1.1.1 production-mongodb-01 master-node1
2.2.2.2 production-mongodb-02 slave-node1
3.3.3.3 production-mongodb-03 slave-node2

(1) Install Mongo on all  servers using the above steps from 1 to 17
(2) un-comment below lines from conf

replication:
  replSetName: “election01”

(3) keyFile: This is used for internal authentication between MongoDB instances in a replica set or sharded cluster.
It ensures that only authorized MongoDB instances can communicate with each other.

scp  /etc/keyfile-mongo  2.2.2.2: /etc/keyfile-mongo
scp  /etc/keyfile-mongo  3.3.3.3: /etc/keyfile-mongo

(4) restart  master and slave and log in to Mongo to start replication


1 login master mongosh –username=mongoadmin –password=yourpass –authenticationDatabase admin
2 Initiate replication rs.initiate()
3 add replica rs.add(“2.2.2.2”)
rs.add(“3.3.3.3”)
4 check replication status rs.status()
5 remove the replica from the replication rs.remove(“hostname:port”)
6 rs.reconfig({})
7 db.serverStatus()
8 db.currentOp()
9 db.repairDatabase()
10 db.stats()
db.collection.stats()

04-Email E-mail Security and privacy

Email Security and Privacy Considerations

Email security is crucial for managing unwanted events by preventing them or mitigating potential damage and loss. Ensuring email security involves addressing the entire process, considering the environment and risk conditions.

Vulnerabilities in Email Security

The infrastructure of internet email originates from the ARPAnet, where the primary concern was reliable message delivery, even during partial network failures. Confidentiality, endpoint authentication, and non-repudiation were not priorities, leading to significant vulnerabilities in modern email communication. As a result, an email message is susceptible to unauthorized disclosure, forgery, and integrity loss.

While these vulnerabilities stem from lower-level internet protocols (such as TCP/IP), they could have been mitigated by email protocols like SMTP and MIME. However, during their design, email was primarily used within the scientific community, where security concerns were minimal. The S/MIME standard now addresses these issues by providing cryptographic security services, including authentication, message integrity, non-repudiation of origin, and confidentiality. Despite widespread commercial support for S/MIME, interoperability issues persist, preventing it from becoming a universal standard.

Message Forgery

Message forgery is a significant concern in email security, where an attacker can manipulate an email to appear as though it was sent by someone else. This can be done by altering headers such as the “From” or “Date” fields. A forged email can deceive recipients into believing the message is legitimate, leading to potential security breaches. Detecting forged emails requires analyzing the email headers and understanding the underlying data, but most users lack the expertise to do so. Although email systems have mechanisms to detect and prevent forgery, they are not foolproof, and the risk remains significant.

The Role of DMARC, SPF, and DKIM

To combat email forgery, three key technologies are widely used: DMARC, SPF, and DKIM.

  • SPF (Sender Policy Framework): SPF is an email authentication method that allows domain owners to specify which IP addresses are authorized to send emails on behalf of their domain. This is done through DNS records. When an email is received, the recipient’s mail server checks the SPF record to ensure the email is coming from an authorized source. If it isn’t, the email can be flagged as potentially fraudulent.
  • DKIM (DomainKeys Identified Mail): DKIM provides a way to validate that an email was sent from the domain it claims to be sent from. It uses cryptographic signatures to verify that the email content hasn’t been altered in transit. The signature is generated by the sender’s mail server and verified by the recipient’s mail server using public keys published in the sender’s DNS records.
  • DMARC (Domain-based Message Authentication, Reporting, and Conformance): DMARC builds on SPF and DKIM by allowing domain owners to publish a policy in their DNS records that instructs receiving mail servers on how to handle emails that fail SPF or DKIM checks. DMARC also provides a mechanism for domain owners to receive reports on how their email domain is being used, which helps in identifying and stopping fraudulent activities.

By implementing SPF, DKIM, and DMARC, organizations can significantly reduce the risk of email spoofing and improve the overall security of their email communications.

Brand Indicators for Message Identification (BIMI)

BIMI is a newer email specification that works alongside DMARC to provide visual verification of an email’s authenticity. With BIMI, organizations can display their brand logos in the recipient’s inbox, next to the email message, as a sign of authenticity. This visual indicator helps recipients quickly identify legitimate emails from trusted brands and enhances email security by making it more difficult for attackers to impersonate a brand. However, BIMI adoption is still in its early stages, and its effectiveness relies on widespread adoption by both senders and email clients.

Phishing

Phishing is a form of cyber fraud that uses deceptive emails to acquire confidential information, such as usernames, passwords, and credit card details. Phishing emails often masquerade as legitimate communications from trusted entities, tricking recipients into providing sensitive information. The impact of phishing can be severe, leading to financial loss and compromised personal information. Phishing attacks are becoming increasingly sophisticated, making it essential for users to be vigilant and for organizations to implement robust security measures.

Email Spam

Spam is the unsolicited flood of emails that clogs inboxes and hampers effective communication. It serves as a form of noise that obscures meaningful messages. The volume of spam has grown so significantly that it often surpasses the number of legitimate emails. While spammers typically aim to promote products or services, the sheer volume of spam can overwhelm email systems, leading to potential denial of service.

Anti-Spam Filtering

Anti-spam filters are essential tools in combating spam. These filters analyze incoming emails and identify characteristics typical of spam, such as suspicious subject lines, content, or sender information. Depending on the filter’s configuration, suspected spam can either be marked and moved to a special folder or discarded entirely. However, setting up anti-spam filters is a delicate process. An overly aggressive filter may result in false positives, where legitimate emails are mistakenly classified as spam, leading to potential loss of important communication.

Anti-spam filtering is an ongoing challenge, as spammers continually adapt their techniques to bypass filters. Advanced filtering technologies, such as machine learning algorithms, have improved the accuracy of spam detection, but no system is entirely foolproof.

Ensuring Message Authenticity with GPG

GPG (GNU Privacy Guard) is a popular tool used for encrypting and signing emails, ensuring that the contents are secure and the sender is authenticated. By using GPG, both the sender and recipient can verify the authenticity of the email and ensure that it has not been tampered with during transmission. GPG works by using a pair of cryptographic keys – one public and one private. The sender uses the recipient’s public key to encrypt the message, and the recipient uses their private key to decrypt it. Additionally, the sender can sign the email with their private key, allowing the recipient to verify the sender’s identity with the corresponding public key.

Ensuring Message Authenticity

Message authenticity refers to the assurance that an email originates from the claimed sender and has not been tampered with during transmission. According to RFC 2822, email headers like Date and From are crucial in establishing authenticity, but these can be easily manipulated, making it challenging to verify the true origin of an email.

In business, email authenticity is generally assumed unless there are clear signs of forgery. However, in archival processes, verifying authenticity is more complex, and additional measures, such as electronic signatures or certified email services, can help ensure the integrity and authenticity of messages.

Certified Email Services

Certified email services, like Italy’s Posta Elettronica Certificata (PEC), provide a legal guarantee of message authenticity and integrity. These services require users to be registered with certified providers who authenticate the sender and issue electronic receipts proving the message’s dispatch and delivery. Such services offer a higher level of security and can be legally binding in disputes.

Privacy Concerns

Email messages can easily be disclosed without authorization, posing privacy risks such as identity theft. To mitigate these risks, sensitive information should either be excluded from emails or protected through encryption. Privacy concerns are often more focused on unauthorized mailbox access rather than message interception during transmission.

In many countries, email is afforded the same privacy protections as traditional mail, with strict regulations governing who can access a user’s mailbox. These regulations vary by country and can significantly impact email recordkeeping policies, balancing the need to preserve potentially legally relevant information with privacy considerations.

Some organizations address privacy concerns by obtaining explicit consent from employees to access their company mailboxes or by allowing users to tag messages as public or private. However, these practices may not always align with national privacy laws.

02-Email e-mail Infrastructure

02-Email e-mail Infrastructure

How Email Works

Email is a store-and-forward method of exchanging messages on the Internet. This means a message sent by a user goes through an asynchronous process of delivery, typically involving a series of steps. In each step, the message is stored by an intermediate server on the network to be forwarded at a later time until it finally reaches its destination. The timing of delivery depends on the availability of network connections.

Figure 1 illustrates the delivery process, which involves a sender, Alice, and a recipient, Bob. Both Alice and Bob use specific applications called email clients, which run on their PCs to send and receive emails. These clients do not communicate directly but connect to email servers, which are specialized applications operated by Alice’s and Bob’s organizations or ISPs that manage the delivery process.

Figure 1 – Basic Email Infrastructure

 

The email delivery process involves the following steps:

  1. Alice composes the message using her email client.
  2. The message is formatted by Alice’s email client in a specific Internet email format and then sent to her local email server.
  3. Alice’s email server locates the address of Bob’s email server using the Domain Name System (DNS), the distributed directory of the Internet.
  4. The two email servers exchange the message, which may pass through a series of intermediate servers on the network, until it is finally stored in Bob’s personal mailbox on Bob’s email server.
  5. The message remains in Bob’s mailbox until he reads or downloads it using his email client.

The procedure is quite similar to the process Alice and Bob follow when exchanging letters. Local post offices play a role similar to that of local email servers, and letter delivery may go through additional post offices (intermediate servers). In both cases, delivery time and even delivery itself are not guaranteed.

The Internet is a best-effort network, meaning the message, like any other information crossing the network, must pass through several servers run by independent organizations that make no commitment to service availability or quality. Therefore, delivery time cannot be predicted, and the message may even get lost along the way.

However, as we will discuss later in more detail, all clients and servers involved in the delivery process follow a set of strict rules (protocols). This allows for the tracing of all relevant events and the recording of detailed information in a report appended to the message. Additionally, in case of delivery failure, the server may attempt delivery again, and the sender may request delivery reports and receipts to confirm that the message has been delivered and/or read by the recipient.

End-User Access to Email

End users can access the email system in several ways:

  1. Email Client: This method corresponds to the basic process discussed in the previous section, where the user runs a special application on their PC designed to interact with the email server. Email clients can be proprietary or open-source software, and a wide variety of them are available. Besides the basic functions of sending and retrieving messages from the email server, which are performed according to standard interaction protocols that ensure interoperability, they usually offer user-friendly interfaces and additional functions to classify and store messages, manage directories, and more. In this setup, messages are typically downloaded and stored on the user’s PC, which may not be convenient for users who need to access their mail from multiple devices.
  2. Webmail: This is the most common way users access email from their home PC, through a service offered by their ISPs or third-party organizations like Hotmail or Gmail. In this setup (see Figure 2), the client application running on the end user’s PC is an Internet browser (e.g., Explorer, Mozilla), which connects to a web server running a special webmail application. The web server acts as an intermediary and manages the connection with the email server. Additionally, messages are not downloaded to the user’s PC but are managed and stored directly on the web server. This provides a significant advantage for users who need to access their mail from multiple devices.
  3. Integrated Systems: This is the typical solution used by most corporations and large organizations. It integrates email access into a broader ‘collaborative’ environment that includes additional functions such as direct messaging, calendaring, contacts, and tasks, as well as support for mobile and web-based access to information. It also manages message storage on a central server. Popular products of this kind include Microsoft Exchange and IBM Lotus Domino. Users run proprietary client applications (e.g., Microsoft Outlook or Lotus Notes) on their PCs that connect to the corporate server, which in turn connects to the email server (see Figure 3). To assist mobile users, these systems often include an optional web interface, functionally equivalent to webmail, which allows access through a web browser. However, the primary interface is typically the proprietary one used on the organization’s intranet. Although this setup is specific and includes proprietary elements, it is essential to consider because it represents a significant portion of the market, especially for email archiving in corporations and large organizations.

Figure 2 – Webmail

 

Figure 3 – Corporate Mail with Integrated System

Interoperability of Email Systems

As discussed in previous sections, exchanging a message involves interaction among several agents (email clients and servers), which are generally heterogeneous systems based on different hardware and software platforms. Additionally, these systems are independently designed and implemented by different parties, potentially without any direct coordination.

One of the main challenges in the Internet email system is ensuring interoperability, i.e., correct and reliable communication among these heterogeneous systems. Interoperability is based on two main elements:

  1. Communication Protocols: These are sets of rules governing communication between agents, ensuring that agents can reliably and correctly interact using a common language and standard procedures.
  2. Message Format: This is a set of formal definitions specifying the structure of the message and how the message and its attachments are encoded, ensuring correct interpretation by different email clients and guaranteeing that the content of the message is correctly rendered to its recipient.

Another requirement is that interoperability must also be guaranteed over time. This means that when the definitions of protocols and message formats evolve, they should maintain backward compatibility, i.e., new rules should still be compatible with old ones. For example, a message formatted according to an older version of the message format standard should be presented correctly by an email client compliant with the new version. Unfortunately, this is not always the case, and it is a major concern in email archiving, where ensuring that archived messages remain readable over time, even as standards evolve, is crucial.

Internet Standards

The standardization process of the Internet is somewhat different from the usual ISO/IEC track, so it is worth explaining how these standards are developed and allowed to evolve.

Internet standards are developed and promoted by the Internet Engineering Task Force (IETF), which cooperates closely with major international standard bodies like ISO/IEC and the World Wide Web Consortium (W3C), the main international standards organization for the World Wide Web.

The standardization process, which dates back to the early days of the ARPAnet project, is highly cooperative and based on special documents called Request For Comments (RFC). RFCs are draft documents, mostly proposals for standards, published by the IETF and posted on the network as a ‘request for comments.’ Each RFC is assigned a unique number and is never rescinded or modified. If amendments are needed, a new RFC is issued with a different number, superseding the old one.

As stated in RFC 1796, which discusses the standardization process, “Not all RFCs are standards.” Some are just memoranda, remarks that people wish to share, research papers, or preliminary proposals on any matter concerning the Internet and Internet-based systems. The IETF assigns a status to each RFC.

‘Mature’ RFCs are rated Standard Track and are further divided into Proposed Standard, Draft Standard, and Internet Standard. Internet Standards (STD) each refer to an RFC (or a set of RFCs) and are given a unique number. Unlike the RFC number, when the standard evolves, the STD number does not change but simply refers to a new RFC that supersedes the original one.

Standardization of Email Transmission

Server-to-server and client-to-server interoperability are ensured by SMTP (Simple Mail Transfer Protocol), which is Internet Standard STD 10. SMTP dates back to August 1982 and is based on RFC 821. However, the protocol currently used by the majority of email applications is known as ESMTP (Extended SMTP) and is defined in RFC 2821, published in April 2001.

However, formally, the status of RFC 2821 is still a Proposed Standard, and the official standard is still the one defined by RFC 821. This situation of ‘going ahead of the official standard’ is typical of the Internet world, and it is of no use to argue whether it is right or wrong; we must simply cope with it.

SMTP specifies how the email client interacts with the email server to deliver the message and how email servers (often called SMTP servers) interact with each other to ensure the message passes through several agents and finally reaches its destination. The use of the SMTP protocol in the message delivery process is clearly shown in Figures 1 and 2.

Regarding the problem of email archiving, this standard is important because it defines the basic format of messages that can be handled by SMTP servers and go through the delivery process. This is a very basic format, supporting only simple text messages in plain ASCII (also called 7-bit ASCII or US-ASCII) characters, which are sufficient only for English and a few other languages. This limitation is overcome by defining a special way to encode richer content in plain ASCII characters, allowing the use of a more general set of characters in the message text, and including formatted text and multimedia content in email messages, as we will discuss in section 2.7.

Standardization of Client-Server Communication

Email clients can retrieve email from servers in several ways, supported by both standard and proprietary protocols. This is relevant to email archiving because the process of storing email messages must deal with how they are downloaded and handled by different client applications, which may affect the process and determine the format of archived messages.

POP3

POP3 (Post Office Protocol version 3) is the protocol most commonly used by email clients to retrieve messages from servers. The official Internet Standard is defined in STD 53 and is based on RFC 1939, published in May 1996. This protocol is limited in scope and allows for the download of messages only. It does not include the management of mail folders on the server side (e.g., Inbox, Sent, Drafts) or any other advanced features like server-based search or access to metadata. This is a severe limitation, especially when dealing with multiple clients, such as a PC and a smartphone, where folders should be synchronized.

IMAP4

IMAP4 (Internet Message Access Protocol version 4) is the most advanced and feature-rich protocol, officially defined by STD 55 and based on RFC 3501, published in March 2003. IMAP4 supports advanced folder management, server-based search, access to metadata, and offline operations. This makes it much better suited for use with multiple clients. However, it is more complex and demanding in terms of computing and network resources.

Webmail Protocols

The Webmail interface uses an Internet browser as the client application and a Web server (or a special Webmail server) as an intermediary that connects to the email server. The protocols used by the browser to communicate with the Web server are HTTP (Hypertext Transfer Protocol) and HTTPS (Secure HTTP). These protocols are not email-specific and are defined by Internet Standards STD 1 (RFC 2616, June 1999) and STD 66 (RFC 2818, May 2000), respectively.

The protocols used by the Web server to communicate with the email server are generally SMTP and IMAP, already discussed in previous sections.

This setup is highly relevant to email archiving, especially when it comes to ensuring that the archived message’s format includes all the information and content needed to faithfully reconstruct the message as seen by the user when accessing it via Webmail.

Standardization of Message Format

The Internet Standard format for email messages is defined by RFC 822 (August 1982), later superseded by RFC 2822 (April 2001), which specifies the format of the email header and body. The format defined by RFC 2822 is still the official standard, though it has been further refined by several other RFCs. The standard email format supports only plain text messages in US-ASCII encoding, which is a major limitation for modern email communication.

This limitation is overcome by the Multipurpose Internet Mail Extensions (MIME) standard, defined by STD 11, which is based on a set of five RFCs (RFC 2045 to RFC 2049, published in November 1996). MIME allows for the use of various character sets and multimedia content (e.g., images, sound, video) in email messages. It also supports the encoding of binary content in a 7-bit ASCII format, which is essential for the correct transmission of non-ASCII content in email messages. MIME is fundamental to modern email communication and is supported by almost all email clients and servers.

Conclusion

The email system is an essential part of modern communication, involving a complex and well-coordinated process of message exchange between various agents (clients and servers) over the Internet. The system is based on a set of standard protocols and message formats that ensure interoperability among heterogeneous systems and reliable communication across the network. These standards are defined and promoted by the IETF through a cooperative and evolving process of RFC publication and review.

The email system supports various access methods for end users, including traditional email clients, Webmail interfaces, and integrated corporate systems. Each method has its own strengths and weaknesses, but all rely on the same underlying protocols and message formats. The system’s success and widespread adoption are due to the interoperability guaranteed by these standards, which allow for seamless communication between different systems, platforms, and applications.

Understanding the standardization of email transmission, client-server communication, and message format is crucial for ensuring the long-term usability and accessibility of archived email messages. By following these standards, organizations can ensure that their archived email messages remain readable and accessible, even as technology evolves.