IT VL 6-7 Emails & Email Spam

E-Mails General Information

Two entities:

• Mail user agents (MUAs)

• Mail transfer agents / mail servers (MTAs)

Protocols

• simple mail transfer protocol (SMTP)

• Post Office Protocol (POP3)

• Internet Message Access Protocol (IMAP)

Mail Transfer Agent

MTA is a server that listens for incoming mail

MTA decides what to do with incoming mail:

• save locally: for pickup by useres (e.g. in var/spool/mail/…)

• forward to another MTA (via SMTP)

A forwarding MTA is known as a mail relay

An MTA does not deliver email to the MUA

• MTA is not an IMAP or POP3 server

Mail User Agent

Mail software used to read an write mail

Local acess to mail (/var/spool/mail)

Access to mail server via IMAP or POP

Well known MUAs include: Thunderbird, Outlook, Mutt

SMTP (Simple MAil transfer Protocol)

Text Protocol

• client sends commands (4 ASCII characters + parameters)

• server responds with reply code (3 decimal digits)

SMTP Envelope (network protocol level)

• e.g. SMTP commands: MAIL, RCPT, DATA

SMTP Content (message level)

• Message headers + message body

Hard to transfer binary data like images and video

Internet Message Format (RFC 5322)

• message Headers contain information about msg (e.g. To, From, Bcc, Date,…)

• message Headers processed by Mail User Agents → independent from SMTP envelope

• Message Body contains message itself and contains pure ASCII text

MIME (Multipurpose Internet Mail Extensions)

Purpose: messages with rich content

• International languages, multimedia messages, multipart messages

Approach: Convert everything to RFC 5322 ASCII message

• Only MUAs have to be modified, not the MTAs

replace non-ASCII characters with a special encoding

• example Quoted Printable Transfer Encoding → ä in UTF8 = 0xC3 0xA4 → convert to quoted printable =C3=A4 → Universität = Universit=C3=A4t

• Base64 transfer encoding e.g. Bild

MIME Content Types:

• media Types: e.g. Type text subtype plain or richtext ; Type video subtype mpeg; …

• Content Types: e.g. Type multipart subtype parallel; type message subtype external-body

Base 64 Encoding

Content is considered as binary data stream

Idea: Map each group of 24 bit (3 input bytes) to an output string of 4 ASCII characters

3x8 input bit rearranged to 4x6 output bit

Each of the 6 output bit is represented with one symbol

• We use printable ASCII characters as symbols

• Look up symbol from table of 2^6 = 64 characters

Base64 Padding

• Problem: Base64 output is arranged in lines with 76 characters each → What if input data is not a multiple of 3 bytes (24 bit)

• Solution fill data with zero bytes 0×00

• Indicate the numer of padding bytes by append “=” to avoid changing the content with the zeros

POP3 (Post Office Protocol 3)

Recipient wants to read mail on local machine but mail is usally handled by mail server

→ POP 3 transfers mail from server to MUA

POP3 ist outdated → use IMAP instead

Features limited to

• Retrieve a list of all mails

• Download mails from server to local machine

• Delete mails on server

provides different authentication mechanisms (Username/Password; Username/Digest(hashedPw); SASL;…)

Does not specify how to send mail

POP3 Problems:

• No extensive manipulation operations (limited to ∘ List, Download, Delete)

• Limmited Support for Concurrent access

IMAP (Internet Message Access Protocol)

Purpose: access mailbox remotely from Mail User Agent → Mailbox remains on a central mail server

Features:

• Concurrent access from more than one computer (MUA)

• Store messages in folders

• Partial fetch (download only required parts)

• Flagging of messages (e.g. as read or answered)

• Server-side search

No functionality to send messages (realized with SMTP)

IMAP Network Protocol

• ASCII Text Protocol

• Staful protocol with sessions (1. Non Authenticated, 2. Authenticated,3. Selected, 4. Logout,Closed) → für diagram Folie 49

• several commands are only valid in one stage

Client-Server Interaction

• Client command begins with alphanumeric string (tag) e.g. ‘A001’ which is created for every new command

• Server receives and works with the command

• Returns the same tag with a response code (OK/NO/BAD)

• Additional information begins with ‘*’ (untagged)

• Tagging enables pipelining and re-ordering of server responses

Flags Message Attributes

List of tokens associated with the message (sent by the server)

A flag can be permanent or session-only

Types:

• System flag: → Flag name that is pre-defined in the imap specification • All system flags begin with ‘\

• Keywords: → A keyword is defined by the server implementation • Keywords do not begin with ‘\’

User Authentification

SMTP after POP

• Authenticate via POP, save client IP address

• Allow SMTP afterwards if IP address matches

TMTP Authentification

• SMTP AUTH protocol extension → Requires Extended SMTP

• Plaintext login

  ∘ Base64-encoded (for compatibility, not security)

  ∘ Secure only if SMTP connection encapsulated by TLS

• Digest authentication (hashed password)

  ∘ Server sends challenge (arbitrary, unique string)

  ∘ Client „encrypts“ challenge with password, sends response

  ∘ Server checks response with known password

  ∘ Password hidden, but dictionary attacks on response possible

TLS Encryption

All email protocols support insecure cleartext transfers

Encapsulate connection with Transport Layer Security

• TLS ensures server authenticity, encrypts data

• Protects from passive surveillance and active man-in-the-middle attacks

Two approaches:

1. Connect to extra port for secure SMTP (deprecated)

2. Use STARTTLS command within ESMTP connection

EMail Spam

Spam: undesired mass message

Ham: non-spam message

Unsolicited Bulk Email (UBE): spam

Unsolicited Commercial Email (UCE): commercial spam

False positive: ham message classified as spam

False negative: spam messages classified as ham

Preventing Adress Harvesting

Adressing Munging

• Spammer crawls the web to retrieve email addresses

• Munge email address to dodge crawlers (Try not to annoy users)

• Publish email address as image

• Generate address with JavaScript

Disposable Adress

• creating a temporary throw away address

Preventing Spam from leaving your network

Spam Relays:

• Mail Transfer Agents (MTA) forward mails

• Open relays forward mails for anyone → easy for Spammer

• Today‘s mailservers are closed relays in default config → Require authentication

• Open proxies also prone to abuse (if misconfigured)

• Spam often originates from botnets

Port Blocking:

• MTA listens always on TCP/25

• Block outgoing connections from client computers → Force clients to use designated MTA for outgoing mails

• Block incoming connections to client computers → Force remote MTA to use domain MTA for incoming mail

Sender Helping With Identifying legitimate email

Hashcash:

• Idea: associate sending email with cost like a post stamp → Resource: computing power

• Find hash = SHA1(date, receiver address, random) where hash ≤ 0x00000FFFFFFFFFFFFFF → Modify random value until you find a matching hash value

• The result serves as proof-of-work → Finding 20 binary zeros requires on average 220 hash operations →Receiver can verify result with 1 hash operation

Sender Policy Framework

• Publish IP addresses of authorized MTAs As TXT record in Domain Name System → example.net IN TXT "v=spf1 ip4:192.0.2.0/24 -all"

• Receiver gets email from *@example.net →Looks up SPF/TXT record of example.net → IP address of sending MTA ≙ SPF definition?

• ∙ Unauthorized SMTP sender indicates spoofing

DomainKeys Identified Mail (DKIM)

• MTA signs outgoing mail → Puts signature in email header

• Public key published in Domain name System → Again TXT record, but under particular name

• Compared to SPF: Slighty more complex → survives SMTP relaying/forwarding

Slowing down Sender

Human Interaction Challange-Response

• MTA receives email from unknown sender

• MTA delays email and sends challenge (E.g. click URL or calculate 5+7)

• User solves challenge and sends response ∘ Requires manual interaction of sender → Spammer is expected to ignore challenge

• Mail is forwarded when response was correct

Greylisting

• SMTP was built for robustness

• MTA receives email from unknown sender

• Save sender in greylist and reply with temporary error

• If sender retries after ≥ 15 minutes, accept email → Spammer is not expected to retry later

Teergrube/ Tarpit

• Spammers send many mails in short time

• Idea: use your resources to slow down spammer

• Delay SMTP responses to keep TCP connection open ∘ Send choked multiline response to avoid timeout trigger (e.g. 451-Well... 451-Give me a moment 451-Just a second 451-Or a minute 451-Or two 451 Error, closing connection)

• Make sure you dont tarpit legitimate MTAs → wastes recources and annoys admins

Receiver-Side detection and filtering

Rule-based-Filtering:

• ∙ Parse incoming mail for known spam patterns → Define rules with regular expressions

• Key words and phrases → e.g. „fast viagra delivery overnight“

• Disguise attempts ∘ „V.A.L.I.U.M“ ∘ Unnecessary URL encoding

• Errors and patterns unusual for legitimate MUA ∘ Malformed headers ∘ HTML message without plaintext copy

Bayesian Filter:

• Abbildung

DNS Blacklists (DNSBL)

• Database of open relays and spammer IP addresses ∘ May include ranges of dialin hosts

• Various different DNSBL providers ∘ Trust required: false listing can cut off your email traffic

• DNS Whitelists also common for large email providers

Hash Value Databases

• Database of email spam hash values ∘ e.g. Distributed Checksum Clearinghouse (DCC)

• Calculate hash value over message body (not headers)

• Query database via UDP or pipelined TCP connection

• When a new email arrives, its hash value is compared against the hash values stored in the spam database

FIltering Actions

• Use combination of different measures ∘ Calculate probability/score per email

• Take actions, depending on score (e.g. Deny Email with error, move to spam folder, fall back to greylisting, challange/response or teergrube)

5 Types of ways to Prevent Spam

1. Prevent address harvesting

2. Prevent spam leaving your network

3. Sender helps identifying legitimate email

4. Slow down sender

5. Receiver-side only detection and filtering