Someone please point out the error of my ways

[andrewducker]

I've been thinking about email, and the ways that it's fucked, and possible solutions.  And sadly, I'm not a deep technical expert in the area, although I pay a fair bit of attention and I'm not completely ignorant, so I was hoping that someone would be able to point out the error of my ways...

Anyway, the major problem with email is that idiots can use it to send spam and viruses.  And you can't easily trace it to its source, because the protocols it uses to send email aren't even remotely secure.  You can, in fact, pretend to be President Bush, or God in the headers, and the receiving server will have no idea you're not telling the truth.

Now, one proposed solution is SPF, which checks a list for the domain of the supposed sender that specifies which domains are allowed to send email for it.  So, for instance, I could set SPF up so that ducker.org.uk mail is allowed to be sent from both *.ducker.org.uk and smtp.orange.net (my phone provider - who won't allow me to send email through my standard mail provider, and insist on taking care of it themselves).  The main problem with this, as far as I'm concerned, is that it means that hundreds of millions of not-terribly-technical people will have to set up quite arcane settings that may preclude them from sending email when they get them wrong.  Hardly ideal in anyone's books.

Now, the simple solution is to say that when it comes to sending email, mail from andrew@ducker.org.uk should only come from a mail server that's *something*.ducker.org.uk.  Anyone up to the take of setting up their own servers is perfectly capable of also setting up the names so that the mail server is in the same heirarchy as the hosts that email is sent for, after all.  And this is almost certainly fine for absolutely everyone who uses their server to send email.

And it provides total security, providing you use authenticated to make sure that only people who are allowed to use mail.ducker.org.uk can use it.  Which is, in fact the case - if you don't have a password, you can't use that server to send email.

The problem is for ordinary people, who _can't_ use their server to send email.  For people who are stuck sending email through their ISPs mail server, or through their mobile phone provider's email server, this is just going to leave them unable to send email at all.  They can still read email (no ISP, as far as I know, stops people _reading_ email from other servers), but they can't send it.

At which point it hit me - there's a perfectly good mail reading protocol called IMAP, which allows you to store your email on your server in a series of folders and read them remotely without permanently moving them from the server to your local PC.  It's a vast improvement over POP for general email reading, and having switched to it, I can't think of a single reason why anyone who ever reads their email from more than one place would use anything else. 

The important bit in that last paragraph is "series of folders"...  One of those folders tends to be "Sent Items" (or equivalent) and it occurred to me - why not have one of them be "Outbox"?  Add a 'special' folder, that would be monitored for emails being placed in it, and they would then be passed onto the correct outgoing mail server (and also to Sent Items, according to your preferences).  This addition would mean that the current method of sending email - SMTP - would be used _only_ for server to server communication, and IMAP+Outbox would be used for sending email from client to server.

Is there a massive flaw in my suggestion - other than it'll take a fair bit of time for any such thing to become standard?  Has it been suggested many times before?  Does anyone have any suggestions who would be a good person to tell me why it's a stupid idea?

Comments

[sbisson.livejournal.com]

Well, Outbox is a standard folder on many systems used by IMAP clients to hold local mail before delivering it to a server.

Most direct IMAP/Server combos already work the way you say.

The problem with running IMAP as an ISP is that you're suddenly taking over responsibility for managing your users' email. It's all very well being a temporary post box, it's a whole 'nother kettle of fish managing the mail. IMAP isn't the world's fastest protocol, and hooking it up to millions of mailboxes isn't particularly easy - especially when you're also having to back up all that mail on a daily basis.

The added infrastructure costs will be prohibitive.

Not an easy task at all!

[andrewducker]

You mean they don't already provide IMAP? When I was looking for a domain host I don't remember any of them not providing IMAP.

Of course, ISPs may not be as happy doing so - but I don't know what the overhead for 'idle' IMAP connections is - a quick google found this tho:

IMAP connections are generally very efficient compared to POP connections. Each reconnection during a POP session requires re-authentication of the user, whereas an IMAP connection requires only a single authentication because the connection remains open. POP connections, therefore, involve much greater performance overhead than IMAP connections. Netscape Messaging Server, in particular, has been designed to require very low overhead by open but idle IMAP connections.

[sbisson.livejournal.com]

I'm focusing on the consumer ISP side of things here.

The problem is really rather complex. There's going to be a massive increase in storage needed - unles you run mail quotas and that rather invalidates the purpioses of IMAP. Don't forget that each directory a user creates will chew even more inodes. And each IMAP session will need to hold that tree in memory (so we'll need hefty processing boxes).

With POP3 it's easy to run a structured tree for all your mail files, as its a relatively fast hierarchical structure. But that plus IMAP? I'd be worried about the added complexity.

You could load balance IMAP servers, but that still leaves us with the file system problem. Terabytes of reliable storage remain expensive - and pooled virtual stores even more so.

IMAP works well for up to several hundred users - I'd be concerned at its operaton for millions. AOL's "IMAP" implementation is a hack on its databases - and requires them breaking several RFCs to do it. And AOL runs very strict messaging quotas, with only a small percentage of users opting to use Communicator for IMAP access.

I'm not going to get into the anti-spam and liability issues. Suffice to say there's going to need to be a mass rewriting of Ts and Cs, and a need for a lot of CPU.

[drplokta]

Your post advocates a

(X) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
(X) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
(X) It will stop spam for two weeks and then we'll be stuck with it
(X) Users of email will not put up with it
(X) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
(X) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
(X) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
(X) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
(X) Huge existing software investment in SMTP
(X) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
(X) Armies of worm riddled broadband-connected Windows boxes
( ) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
(X) Outlook

and the following philosophical objections may also apply:

(X) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
(X) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

(X) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!

[andrewducker]

Mailing lists and other legitimate email uses would be affected

Short of signed email, no solution to spam is going to allow bouncing of email, or sending of email in another person's name, which means that mailing lists will have problems. I see this as a tradeoff, and although I'm on several mailing lists (and have been for years) I's rather have less spam and slightly harder to use mailing lists.

It will stop spam for two weeks and then we'll be stuck with it

It will stop spam faking its origins, which will help in tracking it down to its source and using legislative/technical methods to stop it.

Users of email will not put up with it

Shouldn't be a hassle for 99% of users. People who use their ISPs for addresses, people using webmail and people who already use their domains for sending email will be completely unaffected after all.

Lack of centrally controlling authority for email

Only an issue if you're expecting instant switchover. A gradual changeover would be far more likely with any solution, and offering this as a possibility would lead fairly swiftly to the major email clients using it, and then it'd be transparent.

Huge existing software investment in SMTP

Hence sticking to SMTP for server to server communications

Armies of worm riddled broadband-connected Windows boxes

Said boxes, if they didn't have your password, wouldn't be able to connect to anything at all, at least not with forged headers.

Sending email should be free

Agreed. Why is this an issue?

Ideas similar to yours are easy to come up with, yet none have ever been shown practical

Possibly because nobody has added the functionality to any of the popular IMAP servers and an email client. Oh, and because people don't actually discuss the issues, they just respond with a checkbox form that was kinda funny twice, 5 years ago.

[drplokta]

Short of signed email, no solution to spam is going to allow bouncing of email, or sending of email in another person's name, which means that mailing lists will have problems. I see this as a tradeoff, and although I'm on several mailing lists (and have been for years) I's rather have less spam and slightly harder to use mailing lists.
Lots of spam solutions allow both of these things. Filtering allows both, and SPF allows mail forwarding.

It will stop spam faking its origins, which will help in tracking it down to its source and using legislative/technical methods to stop it.
It's already pretty well impossible for spam to fake its origins. You can track any SMTP email down to the IP address it actually came from by studying the Received From: headers and finding the end of the chain or the first break in the chain.

Shouldn't be a hassle for 99% of users. People who use their ISPs for addresses, people using webmail and people who already use their domains for sending email will be completely unaffected after all.
Most users don't have IMAP accounts. It's a big hassle for them. They won't do anything about it until they have to. They won;t have to until everyone else has done it. End of story.

Only an issue if you're expecting instant switchover. A gradual changeover would be far more likely with any solution, and offering this as a possibility would lead fairly swiftly to the major email clients using it, and then it'd be transparent.
See above. Without an authority who can say "As of such-and-such a date, email servers may no longer accept SMTP connections from clients", nothing can happen.

Said boxes, if they didn't have your password, wouldn't be able to connect to anything at all, at least not with forged headers.
Said boxes will have your password the first time you type it after you've been infected, assuming you haven't just saved it in order to stop having to type it every time, which everyone will. This is the biggest single killer of anti-spam measures -- how do you distinguish between a legitimate email sent by a home user and a spam email sent from the same PC using the same email address, account and server, from a worm.

Agreed. Why is this an issue?
Because millions of people will have to pay for IMAP accounts if they want to be able to send email.

Possibly because nobody has added the functionality to any of the popular IMAP servers and an email client. Oh, and because people don't actually discuss the issues, they just respond with a checkbox form that was kinda funny twice, 5 years ago.
The checkbox list is a discussion of the issues. It's the single most useful document I know of to check any anti-spam solution against.

[sbisson.livejournal.com]

Possibly because nobody has added the functionality to any of the popular IMAP servers and an email client. Oh, and because people don't actually discuss the issues, they just respond with a checkbox form that was kinda funny twice, 5 years ago.

Actually Exchange works just the way you describe.

[drplokta]

Also, I don't think that this proposal has any good points that aren't shared by the alternative proposal of only allowing people to send email via their ISP's mail gateway using SMTP authentication to log in, and it has a number of additional bad points.

[sbisson.livejournal.com]

Of course there's a simpler option - for ISPs to default smarthost all mail, drop mail that's not for registered domains, and then forward onwards. You can do this by setting up default routes for all port 25 traffic inside your network, forcing it through an appropriately configured SMTP relay.

Legitimate mailing lists carry on working, and fake headers never leave the network. Load balancing works well, and you also minimise mail traffic in your network, as SMTP envelopes don't get expanded too early...

If an ISP is using SPF it can then apply the appropriate X-headers for both its relay and any registered mail servers.

It's a known method that works. You don't see spam from AOL (at least only from faked AOL addresses!).

[drplokta]

Doesn't work for people travelling with laptops. My Powerbook is my outgoing SMTP server, because it's easier than trying to change the SMTP server for each network I connect it to, and cheaper than subscribing to a mail forwarding service that's not tied to an ISP. At the very least, it should be a requirement that any network provider blocking outgoing connections on port 25 advertises its mail gateway via DHCP. And all the email clients need a rewrite to use the DHCP-supplied value if there is one, and the locally configured value if there isn't.

Also, it doesn't stop zombies from sending spam from the infected user's real email address.

And the other problem with schemes that rely on ISPs changing the way their servers work is identifying which SMTP servers aren't playing the game and refusing to accept emal from them.

[sbisson.livejournal.com]

This isn't blocking SMTP, it's enforcing smarthosts. If you're roaming then you should be using authenticated SMTP or a VPN to communicate with your ISP or your business' mailserver.

Smarthosting makes sense in many different ways. For one, it keeps bandwidth usage to a minimum as SMTP envelopes don't expand until they reach the destination server.

DHCP advertising of smarthosts isn't a workable solution as many broadband connections are NATed.

What's good about this technique (which is used by BT, AOL and Earthlink to name three ISPs), is that you don't need to change how a SMTP server works. Sure, you change user behaviour (a good thing here, as it mitigates risk), but you also can use it to detect zombie behaviour, and use simple SMTP back-off defences to prevent spaming and worm proagation.

A simple SMTP 5xx error message can inform users of why they're in violation of the network TOCs, and what they can do to avoid receiving the message in future - and as mail clients render these as mail there's no need for changes here, either.

It's quarantine rather than blocking.

A win-win.

[azalemeth.livejournal.com]

I haven't read most of that (It's early! I have college! Sorry!) - but why not simply enable SSL for SMTP and have a nice, fat, secure username and password? Or restrict smtp relay from &your_ip_address, and relay from smtp.orange.com. Incoming email shouldn't be affected in the slightest as you *should* have either pop3 or Imap boxes that actually store the stuff, and get it from everyone not in your big fat RTBL (real-time black list). IMAP is better in terms of features, but puts a much higher load on the server.