Issues from Spam fight- hazard of SenderID and SPF

 

The mx mechanism

mx
mx/<prefix-length>
mx:<domain>
mx:<domain>/<prefix-length>

All the A records for all the MX records for domain are tested in order of MX priority. If the client IP is found among them, this mechanism matches.

If domain is not specified, the current-domain is used.

The A records have to match the client IP exactly, unless a prefix-length is provided, in which case each IP address returned by the A lookup will be expanded to its corresponding CIDR prefix, and the client IP will be sought within that subnet.

Example 1

"v=spf1 mx mx:deferrals.domain.com -all"

Check the mx record of the domain in the envelope sender.  Also check the mx-record of deferrals.domain.com, which looks like it is another set of servers whose job is to retry mail for deferring domains.  In any case, if the transmitting IP is found among those A-records, return a Pass.  Otherwise, return a Fail. 

Example 2

"v=spf1 mx/24 mx:offsite.domain.com/24 -all"

Check the mx-record of the current domain and expand it to its /24 CIDR range, and do the same thing with offsite.domain.com (expand it to its /24 CIDR range).  If the transmitting IP is found in the ranges, return a Pass.  Otherwise, return a Fail. 


The ptr mechanism
 
ptr
ptr:<domain>

The hostname or hostnames for the client IP are looked up using PTR queries. The hostnames are then validated: at least one of the A records for a PTR hostname must match the original client IP. Invalid hostnames are discarded. If a valid hostname ends in domain, this mechanism matches.

If domain is not specified, the current-domain is used.

If at all possible, this mechanism should be avoided because it will result in a larger number of expensive DNS lookups.

Example 1

"v=spf1 ptr -all"

If this SPF record is specied, we would typically see it in a domain which directly controls all its machines (unlike a dialup or broadband ISP) and allows all its servers to send mail. For example, hotmail.com or paypal.com might do this.
Suppose that the transmitting IP is 4.8.15.16 and the envelope sender is me@lost.com.  The process works as follows:

– The receving mail agent looks up the SPF record for lost.com and sees that it is the above.
– The hostnames for 4.8.15.16 are discovered to be lost.com (acquired by a reverse DNS lookup), island.lost.com and others.lost.com.
– The A-record for lost.com is 4.8.15.13, island.lost.com is 4.8.15.15, and others.lost.com is 4.8.15.16.  The last one matches the original IP.  If none had matched, then the result would have returned a Fail.
– Note that since the envelope domain is lost.com and the reverse DNS all included lost.com, this also constituted a match.

Example 2

"v=spf1 ptr:otherdomain.com -all"

Any server whose hostname ends in otherdomain.com is designated to send mail and returns a pass. Otherwise, return a fail.

The exists mechanism 
exists:<domain>

Perform an A query on the provided domain. If a result is found, this constitutes a match. It doesn’t matter what the lookup result is – it could be 127.0.0.2.

Example 1

In the following example, the client IP is 16.23.42.108 and the current-domain is lost.com.

"v=spf1 exists:example.net -all"

If example.net does not resolve, the result is Fail. If it does resolve, this mechanism results in a match.

The include mechanism 
include:<domain>

The specified domain is searched for a match. If the lookup does not return a match or an error, processing proceeds to the next directive. Warning: If the domain does not have a valid SPF record, the result is a permanent error. Some mail receivers will reject based on a PermError.

Example 1

In the following example, the transmitting IP is 4.8.15.16 and the current-domain is lost.com.

"v=spf1 include:lost.net -all"

If lost.net has no SPF record, the result is PermError.

Suppose lost.nets SPF record were "v=spf1 a -all". 
Look up the A record for lost.net. If it matches 4.8.15.16, return Pass.  If there is no match, the include as a whole fails to match, but the -all value is not used (from lost.net’s SPF record); the eventual result is still Fail from the outer directive set in this example, in this case lost.com.

Note that only the evaluated result of the referenced SPF record is used but its action is not.  The action in the outer directive is what is used.  In the above, if the SPFrecord for lost.com was the following "v=spf1 include:lost.net ~all", then lost.net would still return a Hard Fail, but the overall SPF result would be a Soft Fail.

Furthermore, care is needed to ensure that "include:" mechanisms do not place domains at risk for giving SPF Pass results to messages that result from cross user forgery. Unless technical mechanisms are in place at the specified other domain to prevent cross user forgery, "include:" mechanisms should give a Neutral rather than Pass result. This is done by adding "?" in front of "include:". The example above would be:

"v=spf1 ?include:lost.net -all"

This way, even if the lost.net SPF checks out, we still give it a Neutral because there could be some forging going on.  We neither confirm nor deny the result of the cross-domain checking.

Moving onwards to mechanisms, let’s take a look at them in a bit more detail.  Again, this information comes straight from the OpenSPF page, with extra commentary by me.

The all mechanism

all

This mechanism always matches. It usually goes at the end of the SPF record.

Example 1

v=spf1 mx -all

Allow the domain’s MXes to send mail for the domain, prohibit all others.  Reading the syntax from left to right, the version of SPF is 1.0, return a pass if the sending IP is in the MX records for the domain, return a Fail on everything else.  Note that the implied syntax is the following:

v=spf1 +mx -all


Example 2

v=spf1 -all

The domain sends no mail at all.  Read left to right, the version of SPF is 1.0, return a Fail on everything (ie, if any IP has this domain name in the envelope sender, return a Hard Fail).


Example 3

v=spf1 +all

A record like this defeats the purpose of SPF.  To interpret it, the version of SPF is 1.0, return a pass on everything.  If you are returning a pass on everything, it means that whatever IP is sending mail for your domain, you say that’s okay.  That means any IP can forge your domain.

The ip4 mechanism
ip4:<ip4-address>
ip4:<ip4-network>/<prefix-length>

The argument to the "ip4:" mechanism is an IPv4 network range. If no prefix-length is given, /32 is assumed (singling out an individual host address).  This is one of the easier records to interpret.

Example 1

"v=spf1 ip4:192.168.0.1/16 -all"

Allow any IP address between 192.168.0.1 and 192.168.255.255.  If the transmitting IP is not within this range, return a Hard Fail.

The ip6 mechanism
ip6:<ip6-address>
ip6:<ip6-network>/<prefix-length>

The argument to the "ip6:" mechanism is an IPv6 network range. If no prefix-length is given, /128 is assumed (singling out an individual host address).

Example 1

"v=spf1 ip6:1080::8:800:200C:417A/96 -all"

Allow any IPv6 address between 1080::8:800:0000:0000 and 1080::8:800:FFFF:FFFF.

Example 2

"v=spf1 ip6:1080::8:800:68.0.3.1/96 -all"

Allow any IPv6 address between 1080::8:800:0000:0000 and 1080::8:800:FFFF:FFFF.

The a mechanism
a
a/<prefix-length>
a:<domain>
a:<domain>/<prefix-length>

All the A records for the domain are tested. If the client IP is found among them, this mechanism matches.

If domain is not specified, the current-domain is used.

The A records have to match the client IP exactly, unless a prefix-length is provided, in which case each IP address returned by the A lookup will be expanded to its corresponding CIDR prefix, and the client IP will be sought within that subnet.

Example 1

v=spf1 a -all

Lookup the A-record of the current domain.  If it matches the transmitting IP, return a Pass. If not, return a Fail.

Example 2

v=spf1 a:example.com -all

Lookup the A-record of example.com.  If it matches the transmitting IP, return a Pass.  If not, return a Fail.

Example 3

v=spf1 a:mailers.example.com -all

Example.com has explicitly listed all of its outbound mailers in a special A-record under mailers.example.com.  Lookup the A-record for mailers.example.com, and if the transmitting IP is found amoung them, return a Pass.  If it is not, return a Fail. 

Example 4

v=spf1 a/24 a:offsite.example.com/24 -all

This SPF record lists two possible mailers, a/24 and a:offsite.example.com/24.  Lookup the A-record of teh current domain and assume that it resolves to 192.0.2.1; the entire class C of 192.0.2.0/24 would be searched for the client IP.  Similarly, assume that the A-record for offsite.example.com is 192.0.3.1.   It would be expanded to 192.0.3.0/24 and would be searched for transmitting IP.  If more than one A record were returned for the domain, each one would be expanded to a CIDR subnet.
If not match was no found, a Fail would be returned.

In my next post, we will get to the mx, ptr, exists and include mechanisms.  Then, we will take a look at some real-life SPF records.

This is essentially going to be a summary of the information that appears on the OpenSPF documentation web page.  Really, what else can I say that isn’t said there?  But, if you’re like me and rarely bother clicking on links inside of blog posts and would prefer to read it within the same web page you navigated to, then read on.  For brevity’s sake I will split this into a couple of posts.

In my previous posts, I’ve stated that SPF is a method of authentication where the receiving email host checks to see if the transmitting IP is allowed to send mail for the domain in the envelope sender.  It’s actually a bit more complicated than that.  In an SPF record, a domain defines zero or more mechanisms. Mechanisms can be used to describe the set of hosts which are designated outbound mailers for the domain in the envelope sender.

all | ip4 | ip6 | a | mx | ptr | exists | include

Domains may also define modifiers. Each modifier can appear only once.

I’ve had a document sitting on my shelf (ie, the window-sill 10 feet away from my desk) for about 6 months now just waiting to be read.  It’s entitled Sender Repuration in a Large Webmail Service.  It’s by Bradley Taylor, at Google, and is available to be read at the past documents from the Conference in Email and Antispam, 2006.

Anyways, I finally got around to reading it this week.  Like everyone else, Gmail uses a lot of Sender Authentication to do their filtering.  One of the ways they authenticate mail is with SPF, and for domains without SPF they use an algorithm they dub "Best-Guess SPF" which is meant to be a temporary measure until more domains come onboard and start publishing their SPF records.  They readily admit that the technique isn’t perfect, but it’s not bad, either.  Basically, it works in the following manner:

1. Check the domain of the envelope sender.  If it doesn’t publish SPF records, then check the MX-records and A-records of the sender’s domain.  If the sending domain comes from the same range of IPs as the MX-record or A-record, then the sender has been authenticated.

Example 1 (using fictitious numbers)

Transmitting IP = 4.8.15.16
Envelope sender = me@lost.com
A-record of lost.com = 4.8.15.11
MX-record of lost.com = 4.8.15.0/27 (4.8.15.0 – 4.8.15.31)

Since the transmitting IP is within the range of the MX-records (an abnormally large MX record, but hey, this example is fictitious), we have an authentication.

2. If that doesn’t work, get the reverse DNS of the sending IP.  If it matches the domain of the envelope sender, then the sender has been authenticated.

Example 2

Transmitting IP = 4.8.15.16
Envelope sender = me@lost.com
Reverse DNS of 4.8.15.16 = lost.com

The reverse DNS name matches the name of the domain in the envelope sender, so the sender is authenticated.

Example 3

Transmitting IP = 16.23.42.108
Envelope sender = me@others.com
Reverse DNS of 16.23.42.108 = island.com

The reverse DNS name does not match the envelope sender, therefore, no sender authentication.

3. If that doesn’t work, use a technique that is referred to as PTR zone.  If the sender is a subdomain of the DNS PTR’s zone, then it is authenticated as if the sender comes from the zone itself.  The example given in the document where I discovered this seems a bit backwards, so I’m going to clean it up a bit in order to conform to the description given.

Example 4

Transmitting IP = 16.23.42.108
Envelope sender = me@island.others.com
Reverse DNS of 16.23.42.108 = domain in PTR zone = airplane.others.com

This is close, but not an authentication.  The envelope sender (island.others.com) is not a subdomain of the domain in the PTR zone (airplane.others.com).

Example 5

Transmitting IP = 16.23.42.108
Envelope sender = me@island.others.com
Reverse DNS of 16.23.42.108 = domain in PTR zone = others.com

The domain of the sender (island.others.com) is a subdomain of others.com, and therefore we have an authentication.

Using this extra bit of authentication allows Gmail to authenticate almost twice as much mail as a standard SPF check.  That’s actually pretty good.  As to whether or not this is a good idea, OpenSPF has this to say about it:

Best-guess processing is a crude, non-standard attempt at guessing the IP address range of a domain’s outgoing mailservers.  "Non-standard" means it is not standardized and specific to the implementation.

Some find this remarkably good at detecting unforged messages from domains that have not yet published SPF records. Others consider it a security hole because it gives attackers a lot of additional potential targets (authorized hosts) to hack in order to abuse the domain.

From an anti-spam perspective, I think sender authentication is a good idea but it all depends on how it is used.  In my opinion, successful authentication is best used in conjunction with safe senders.  I first voiced my opinion a few weeks ago when I thought it was a security risk.  However, at the time I don’t think I was thinking ahead; I think the way to implement a safelist is to allow a sender to be on a safelist (ie, bypass spam filtering) if the sender can be authenticated. That way, spoofing the sender doesn’t work and if they do start spamming, there’s a much more reliable paper trail. 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s