Anti-Spam

[andrewducker]

Having seen numerous articles on Bayesian Spam Filtering, I decided to take a leap and see what it could do for me.

I snagged a copy of POPfile, followed the incredibly simple install instructions and downloaded my mail. during installation I'd given it the name of 4 "buckets" I wanted mail put into - spam, friends, livejournal and general.

I also told it that I didn't want it putting the name of the bucket at the start of the subject line (so all spam mails would have [spam] in the subject line), but to instead put the bucket into the email headers (where it wouldn't be seen). My email reader The Bat happily filters based on header information and I didn't see why other people should have to put up with subject lines being mangled in replies.

Of course, the first time I ran the software it had no idea where the emails should go, as I hadn't trained it at all. It therefore filed them all in bucket numer one, which happened to be "spam". I then opened the User Interface (web based and very easy to use) and went through all of the non-spam emails, telling it what group they should belong in. I also set up filters in The Bat to put emails from each bucket into their own folders.

Once I'd initially trained it on about 20 emails, it's managed to spot livejournal posts perfectly (they obviously have enough in common to be simply spottable), spam near perfectly (no false negatives, only the occasional false positive) and friends fairly well (Joe and Mike are now both recognised instantly as friends, and it'll learn the rest of you as I get emails from you).

It's not perfect (yet), but it's doing a darn good job and I'm confident that as I feed it more email it'll get better and better. Apparently I should only touch it when it categorises something incorrectly, so I happily predict that by this time next week I'll pretty much have forgotten it's there.

Oh, and for all you wierdos that don't use Windows - it's Perl based and installs on Macs and Linux...

Highly recommended

Comments

[protempore.livejournal.com]

Yes sir, that POPfile is some good stuff.
I upgraded to a newer version on June 11, and reset my stats then . . . so, my current stats as of June 11 are:

Messages classified: 3,334
Classification errors: 40
Accuracy: 98.8%

Not bad, not bad at all.

[kpollock.livejournal.com]

Not much good to me who reads most stuff at work and so relies mainly upon web-based mail clients.

I'd have to read my mail at home to filer spam.

[allorin.livejournal.com]

Doesn't Outlook Express do most of that?

[cleodhna.livejournal.com]

I find that Mail - the client that comes integrated with Mac OS X - works a treat. It includes a Bayesian spam filter that's pretty accurate and has pretty clearly definable rules for filtering stuff.