Cool Technology of the Week

I've written about the challenges of Spam filtering - false positives and false negatives.

Recently we've experienced a doubling of the volume of incoming Spam. It's essentially a "Spam Denial of Service attack" that is overwhelming our Spam filters. The filters have a failsafe behavior that automatically lets Spam through if the servers get overwhelmed. Leaks of Spam and the increasing challenge of providing reliable, secure, 99% spam free email has caused us to revisit our email configuration and spam filtering products.

Our Spam filtering company, Symantec, provided onsite engineers to examine our configuration and hardware design. They suggested reconfiguration, enhancement of our CPU capacity and an upgrade to the latest software version that "learns" about common email patterns within the organization and whitelists selected traffic, relieving the burden on the spam filtering servers. Symantec also suggested replacing our software-based product with their 8300 series appliance. The appliance is better equipped to process large volumes of mail.

As a class of technologies, Spam filters include pattern recognition, Bayesian probabilistic decisionmaking, and neural network techniques among others. The best comparison of Spam flters, I've found is a recent Infoworld article.

The article illustrates the difficulty in improving our situation. The Symantec product comes out best in class, but only stops 96.4% of Spam. There were products that did better, but most had offsetting problems with false positives. Only Sendio and Proofpoint had better Spam blocking rates and no "critical" false positives. They both had much higher "bulk email" false positives than Symantec which accounted for the "best in class" rating for Symantec. The Infoworld evaluation was based on the Symantec appliance. The appliance has the same anti-spam engine as their software, but can perform additional functions e.g. better reporting, smtp-based throttling based on locally observed reputation, and others. We are testing the appliance now.

Our challenge is that as a healthcare provider, we cannot have false positives. A critical patient email, lab notification, or followup from a medical colleague must be delivered. We will accept a bit more Spam in order to have few false positives.

Thus, for now, we've concluded that an appropriately configured, hardware optimized Symantec configuration is our best bet. The war against Spam is a continuous battle, but for now, 96.4% filtering with very few false positives, wins the race. Hence, Symantec Anti-Spam (formerly known as Brightmail) and their 8300 series appliance is my candidate for the Cool Technology of the Week. Spam is an elusive target, so we'll continue to watch the efficacy of all the available products.