There is a modern proverb that goes something like this:

Acknowledge the bad, but focus on the good.

In sum: do not be so naive as to think that bad things do not exist in this world, but rather choose to spend your energy and time focusing on the good things instead. Make a conscious decision that focusing on the good will not only get you further in life, but ultimately make you happier.

Before we get too carried away here – no, this is not a self-help, positive thinking piece - in the world of technology, IT professionals spend more than their fair share of time focusing on the bad, and it is often out of necessity. Every security decision comes with its own set of trade-offs, and IT professionals are often choosing the lesser of two evils.

But, the sentiment behind this saying poses an interesting question: can focusing on the good ever make you more secure?

However, the solutions we have in place to protect our email messaging systems typically do not adhere to the principle above. Rather, they hone in on the “bad”.

These methods range from the mainstream (content filters) to the paranoid (disposable email address services). They all offer some degree of control over abusive email, but none are perfect and many are time consuming.

So here we are - 32 years into the problem of spam, billions of dollars deep into the problem of solving it, but still haven’t been able to eliminate it.

Worse, in 2009, “spam” email comprised 81% of all messages sent. And while some general characteristics can be used to describe what the messages that comprise this 81% figure may have looked like (illegal html, all capital letters, invalid recipient field etc.), the truth of the matter is that the characteristics of spam are continually evolving.

In addition, every day nearly 150,000 new zombie computers are created and an average of 10,000 new malicious code signatures are added to software vendor Symantec’s threat database.

In sum: the ways in which the “bad guys” present themselves, and the methods they use to try to present themselves, are continually evolving.

So if the characteristics of 81% of email traffic purposefully vary on a continually basis, is there any constant in this equation?

Yes: the “good” guys.

The sending source, sending address, headers and general content you receive from email senders you communicate with on a regular basis has remained relatively constant - namely because there is little incentive for your colleague, mother, best friend, boss, personal trainer, roommate, or child to manipulate the properties of email.

They have a message they wish to send to you, and they do exactly that. The systems they use to send these messages to you all act alike and the email addresses they utilize are generally unvarying.

And it is these contacts and contact sources that comprise a successful whielist. For those interested in learning more about email whitelist best practices, check out the new Sendio white paper below How to Reclaim your Email Using Whitelists.

Download the White Paper

If ever there was a single image that shined a spotlight on the arms race that has been escalating over the past decade between spammers and the entities seeking to disarm them, it is Fast Company’s Infographic of the day, courtesy of New Scientist.  A copy of the full image can be found here. 

It chronicles the rise and fall of overall spam levels since 2004 – the rise of botnets in ’06, the fall of McColo in ’08, a new spoofing technique unleashed earlier in the year and several others.

But more interesting than the spam levels themselves is the breakdown of emails to responses in one particular “spam campaign.” The campaign was comprised of pharmaceutical spam sent out over a one month period in 2008 by just 1.5 percent of one botnet called “Storm.” The numbers go something like this:

-       35 million emails sent

-       8.2 million arrived at valid email servers/recipients

-       10,500 recipients clicked the link in the message

-       28 bought products

From a marketing perspective, these numbers are terrible and translate to a conversion rate of 0.000008%.

But when you extrapolate over the whole network of Storm botnets, the numbers tell a different story – and generated around $3.5 million in “sales” for the year.

Given the low cost of spamming, this is nearly all profit. If $3.5 million is not incentive to keep doing something, we’re not quite sure what is.

Which puts the rest of the numbers into perspective – spammers are highly incentivized to continue to find ways around content filters, blacklists and “isp shut-downs” – and as the numbers show, they will.

Increasingly, there is a real need here for change.  Spammers are not going away, yet we continue to do battle against them using the same old, tired techniques. It might just be time to take a new mainstream approach here.

Some background: Last week Amir Lev posted an attention-grabbing blog on Computerworld that posed some interesting questions that have surrounded challenge-response email protection systems for years, but didn’t quite grasp the whole story on how this technology really operates today.

The blog entry in its entirety can be found here.  We touched on this in an entry last week, but there is still some valuable ground to cover.

We won't make the argument here that challenge-response technology is the be-all end-all of email security (that goes against the philosophy of this blog, and the philosophy of this writer) – but it is far from the apocalyptic solution Lev describes and a huge step up from many of the “anti-spam” solutions on the market today.

When challenge-response technology is implemented properly it can be a valuable tool within a comprehensive email security solution – a mechanism to support a powerful dynamic whitelist, not a clunky auto-responder that ushers emails into an abyss.

Here we dive deeper into the shakiest parts of Lev's argument in a short series touching on some of the most common myths surrounding CR technologies…

Wow.  This is a serious business problem (abuse of email) reduced to the same type of partial truths, misdirection, and self-serving spin that radical liberals and reactionary conservatives use to discuss politics.  Any time you see sarcasm in a debate, check the facts.

CR is used to manage white lists.  If I know you, if I do business with you, if I ever send you a message, you are already on my white list.  So, everybody I already communicate with NEVER gets a challenge.  Instead, everything you send me gets delivered without risk of being mislabeled as "spam" by a content filter, with no administrative overhead.

If you are someone I don't know, and you are interested in establishing a relationship with me via email, when you send me your first message I run through an integrity check process for BOTH of our benefit.  Within seconds, I test your message with a variety a techniques (are you are a legitimate sending server; does the message come from a valid domain; check for spoofing; etc.) and if everything looks good, I send out a CR message.  It asks you to confirm, ONE TIME, that you are a real person and not simply software, so that I can automatically add you to my white list and thereby guarantee your important messages won't get lost.  This is good for both of use.  (Oh, and no CAPTCHA required.  That is old tech.  Just hit Reply)

If you are a "spammer," 97%+ of the time your message fails the basic integrity checks, so your abuse message is simply dropped.  These checks are totally deterministic (pass/fail) and not probabilistic (statistical guessing), so there are no "false positives" or "false negatives."  For three messages in 1,000, a challenge request is sent out, with two possible results.  Since spammers use botnets and other compromised systems as sending servers, their sender addresses are always forged.  If the sender address doesn't actually exist, the challenge goes nowhere.  If the sender address was stolen, the legitimate owner (what Amir call the "innocent user") receives the challenge message.  This tells them that their address has been compromised, which you would think they would be interested to know about .  Since there is no response to the challenge, the original abuse message gets dropped.  No downside here at all.

Now, Amir labels the 3 challenge messages that are sent out against the 1000 incoming messages as "backscatter" and "effectively spam" but uses an illogical argument.  The Internet email protocol SMTP (like all IP protocols) in not really like dropping a letter with a stamp in the postbox on the corner: a one-step one-way transaction.  With email, the sending server and receiving server go through a whole back-and-forth request-acknowledgment IP dialog setting up the communications process.  Actual challenge messages have no measurable impact on overall Internet message volume.

But there certainly is a problem, which Amir points out.  However, if people with his mindset (CR is inherently bad; the earth is the center of the universe) start with a faulty premise, they typically reach a faulty conclusion.  The issue: if you design spamtraps, content filters and reputation services with the presumption that challenges are bad, those systems are poisoning what is actually a highly effective process.  A challenge message is not "unsolicited commercial email" so categorizing it as "spam" is a fundamental design flaw in those systems.  "Branding" the users of CR systems as spammers and therefore mishandling the legitimate email is a mistake by the reputation systems, and sure sounds like true persecution of "innocent users."  All of the "lost mail" that Amir refers to is caused by other ill-conceived systems, not the CR process.

So, if someone I don't know sends me an email (asking for the privilege of getting into my inbox), and then "can't be bothered responding" to a challenge that actually helps both of us, how should I view that potential relationship? Or, if I'm in Sales and I worry that a challenge message might somehow make a new prospect feel uncomfortable or confused (because they read the emotional hatemail that the concept seems to create), maybe I should simply have the challenge feature turned off for my address while allowing everyone else in my organization to be free from email abuse.

In practice, businesses who use CR systems seem to love them because they are amazingly effective with virtually no administration.  It is interesting that the most vocal critics of the concept are people with a vested interest in potentially competitive products like content filters and reputation services.  Amir's closing "absurdum" comment pretty well sums up the substitution of emotional argument for actual logic: "if everyone used CR, email would become unusable" due to an infinite "loop."  Reality is: in a brand new email relationship, (1)  if you send me a message, (2) your CR system automatically adds my address to your white list, (3) so when my challenge request comes back you get it, (4) and when you hit reply you are now in my white list, and (6) we will now forevermore be able to send messages back and forth with no risk of loss.  No infinite loops, just reliable business communications without email abuse.

Yesterday CJ Fearnley of Open Source Magazine & Remote Responder posted an interesting blog on “best practices for SMTP blocking of email spam,” stressing to readers the dangers of blocking blacklisted Received headers at SMTP time, while endorsing the general practice of blocking blacklisted senders at SMTP time.

Here at Save the Mail we are big proponents of taking advantage of the ability to block invalid email at the time of SMTP transfer – when it’s possible to determine absolutely that a message is invalid.  Typically 90%+ of the email any individual or organization receives is invalid and it would be nonsensical to accept every one of these messages and then work to determine what is good and what is bad. 

Technologies that can weed out the most obvious spam before messages enter the network (e.g. directory synchronization checks and certain forms of greylisting) can be an immense help to organizations seeking to alleviate bandwidth requirements while protecting their email infrastructure.

However we can’t help but think that Blacklists of any kind are not one of these said technologies.  Blacklists have been in use for 10+ years but at their core, they tend to hurt legitimate companies who for whatever reason have angered the email gods, while not truly stopping the “dangerous” senders they are meant to.

From a spammers perspective: as soon as their IP is blocked, they get a new one and continue to spam. It is a numbers game for them and one blocked ip address means very little.  Meanwhile, law-abiding “emailers” are blacklisted daily and left to clean up the mess – without the advantage of being able to simply dump their ip and move onto a new one.

Its not much, but we did get a mention in the Boston Globe…

http://www.boston.com/business/technology/articles/2009/04/11/filters_getting_better_at_blocking_spam/?page=2

Other companies, like Sendio Inc. in Irvine, Calif., and Spam Arrest LLC of Seattle, use a “challenge-response” technique. Send an e-mail to a challenge-response user and you’ll get an automated reply, asking you to type in some words or numbers. This will prove your e-mail came from a human being and not a spam-spewing computer. If you send the correct reply, all your future messages are delivered immediately, but spam messages can’t get through.

For the record… Sendio’s sender address verification technology (SAV), also know generically as challenge response, DOES NOT require anyone to “…type in some words or numbers.” Our technology requires a simple “REPLY & SEND.” and ONLY in the case where the sender is completely unknown to the intended recipient. For example, anyone I send an e-mail to is automatically added to my personal accept-list, thus, is NEVER subjected to the address verification process.

Fear is used, universally, as a means to control people. Governments use it. Large businesses use it. So it should come as no surprise to anyone that “cyber bad guys” us it. Why do they use fear… Because it is is effective!

I often ask myself who comes up with terms like “scareware?” Talk about a self-fulfilling prophecy.

“Scareware” is, at its core, a Trojan horse. In most cases, the “malicious security software” that plagues computers around the world is willingly installed by the victims themselves. The purveyors of these threats, in many cases, get their victims to pay for the software under the guise that it is, itself, software designed to protect the user.

The easiest and best way for people to avoid falling victim to these types of attacks/threats is to use common sense.

• Don’t install software unless you can verify its  source is legitimate and reputable.
• Before installing any new software on your computer,  make sure your anti-virus software is enabled and its definitions are  up-to-date.
• Whatever you do, don’t disable your anti-virus  software. No legitimate software should ever require such an action.
• Finally, before installing any new software, make sure  your important files have been backed-up to a location off your  computer.

In the end, even people who follow all the best security practices sometimes still get hurt by malicious software. However, by following the 4 steps mentioned above, your risk of getting burned is greatly reduced, and even if you do get burned, at least your will not loose your data.

Spam in the Neighborhood
http://securitywatch.eweek.com/spam/spam_in_the_neighborhood.html

“Among others, experts at messaging security vendor Sendio have called out the recent trend toward local spam campaigns. In a recent research summary, the company’s CTO, Tal Golan, highlighted the use of methods including the spoofing of local news events, and regional news portal domains, to convince people to click on the (frequently malware-infected) URLs that spammers are trying to pawn off on them.”

http://www.eweek.com/c/a/Security/Google-Spammers-Rally-Back-From-McColo-Shutdown-639980/

“Location-based spam is the latest technique being used by ‘bad guys’ to increase the likelihood that an unsuspecting victim will not only read their message, but will actually click one of the links in the message,” explained Tal Golan, CTO of e-mail security firm Sendio. “This new methodology is the next salvo in the spam arms race, but is really just an extension of the ‘social engineering’ threat vector that has become so popular and effective in the last three years.”

Location based spam is the latest technique being used by “bad guys” to increase the likelihood that an unsuspecting victim will not only read their message, but will actually click one of the links in the message. This new methodology is the next salvo in the spam arms race, but is really just an extension of the “social engineering” threat vector that has become so popular and effective in the last 3 years.

Here is how this works…

Thanks to IP addressed based geolocation (see http://en.wikipedia.org/wiki/Geolocation_software), it is a trivial exercise for a bad guy to determine, with a surprisingly high degree of accuracy, the physical location where a company or organization’s email server is hosted. With this information in hand, the spammer has enough information to design a targeted attack.

For example:

Let’s assume you work for Google. Using a simple IP check, the spammer can determine that one of Google’s email servers has the IP address 74.125.67.100. Thanks to IP based geolocation (http://www.ip2location.com/free.asp), the location of this IP address can easily be determined to be in Mountain View, CA.

Using this data, the spammer will then query the website of a local newspaper, in this case the San Jose Mercury News, and will pick a local “hot topic” headline to be used as the subject for the message.

Finally, the spammer will extract actual content from the news and will insert it into the spam message and will include links that appear to provide the recipient with more information about the topic, but are actually links to dangerous, threat laden web sites. Unfortunately, social engineered attacks, specifically those using location, are proving to be highly effective at soliciting the all important “click” from the unsuspecting victim.

At Sendio we have seen all types of social engineering based attacks increasing steadily. While it is difficult to determine exact figures, our best estimates place social engineered location-based attacks between 10% – 30% of all unsolicited email.

What effect did the November 2008 “McColo” shutdown have on spam (http://www.securityfocus.com/brief/855).

The McColo shut down had a measurable impact, but Sendio’s customers, the vast majority of whom are small, medium and large enterprises, did not see anywhere near as dramatic a change as the major free email providers (Gmail, Yahoo, AOL, MSN, etc.) The levels of spam/uce have, based on our estimates, moved beyond the level seen immediately prior to the McColo shutdown.

As we have seen over the course of the last 6+ years, the bad guys are extremely well organized, motivated, and appear to be well funded. Unfortunately, thanks to the reactive nature of the current status quo spam countermeasures, the arms race continues in favor of the bad guys.