Friday, November 30, 2007

Standardizing on Gmail IMAP

Web based access to email allows access to email anywhere where you have an Internet connection. Many web-based email services now offer a lot of space to archive your mail. With Google’s addition of IMAP support in their Gmail service, we can now leverage the advantages of web-based and “local” mail handling.

If you access your email on different machines or web sites, trying to keep all your email activities can be quite a chore. Another difficulty is keeping email organization in sync across computers (i.e., not wasting time rereading and resorting email on each different machine). At the same time, it is often preferable to use your favorite email reader to view and/or compose email, particularly if you need to access your mail while offline.

This article interlaces howto steps for using Gmail as your primary email server while maintaining a separate email address and allowing use other email readers along with a bit of history. If you have any comments or experience for other email clients’ access to Gmail, feel free to comment to this entry and I will try to verify and incorporate your experience here.

Old School e-Mail Access — POP3

Most people (whether they know it or not) have been accessing their mail via POP3 (Post Office Protocol). This simple protocol describes how mail is transferred from a mail server to the e-mail client—the e-mail reader. Generally, this assumes that the mail will sit on a mail server until the e-mail moves the mail from that server. Such mail servers are not generally intended to save mail long periods of time. Most e-mail programs, by default, will delete the mail from the mail server. When this happens, the mail cannot be accessed from anywhere else—any other e-mail program or web site—because the mail is no longer on the server to be retrieved.

It is possible to configure most e-mail programs not to delete e-mail from the POP3 server. This allows access to the same e-mail from several readers. But most POP3 servers are not really designed to save e-mail long term—it is not very efficient for them to store such mail, among other things. Also, it leads to a lot of wasted time, having to re-organize e-mail at each location where you access your mail. I won’t go into much detail about this because I want to focus using Gmail’s IMAP.

Any modern operating system comes with its own e-mail program Outlook Express for Windows (before Vista), Windows Mail for Vista, Apple Mail for OS X, and a whole variety of open source options that run on Linux and other operating systems. Many companies have made a living on email programs; Eurdora, Lotus Notes, Outlook, etc. See Wikipedia Comparison of e-mail clients for a longer list of options.

Untethered Freedom—Web Access to Mail

Web access to e-mail has provided a big step in freeing one’s self from the machine where their email program and, more importantly, their email is. There are two approaches to web access:

  1. E-mail viewer
  2. E-mail reader client
  3. Dedicated e-mail systems

An e-mail viewer is a very simple client that simply views whatever e-mails are on the server. It does not store the e-mails anywhere so if another client has deleted the e-mail, the viewer will not be able to see it. Due to this limited functionality, such viewers are rare, but can sometimes be useful http://www.mail2web.com/ is one of these.

There are several e-mail reader sites that act very much like the stand-alone e-mail program that you have on your computer. Such reader sites will download and store e-mail from the mail server into its own, managed area before viewing. Since these readers are web-based, you can access the mail from anywhere you have access to the web. With such web services, you can usually configure them to retrieve mail from your own separate mail server (e.g., sbcglobal.net, your own domain). With such web sites, you can usually get an e-mail address specifically for that site and use it as your primary e-mail address. There are many... one particularly well done example is http://fastmail.fm/ although you need to pay to get enough storage space to be useful, long term. These services usually also have the ability for you to access their messages via another reader program, if you prefer the interface of another program, via POP3 and/or IMAP4.

Then, there are full-fledged e-mail web services such as Hotmail, Yahoo!, or Google Mail (Gmail), as well as many others. Such web services manage all the email themselves and are geared primarily to support their email address as the primary email address. They may not allow access via an external e-mail client (or charge extra to allow that).

Modern Mail Handling with IMAP4 — Internet Message Access Protocol

We all organize our mail into folders, don’t we? If we do, then, traditionally this organization only occurs on the client reader. This means that if you have access to your email from different locations, you have to organize the mail for each client. This is because, using POP3, the mail is copied to the mail reader and organized locally, independent of the other readers.

The IMAP4 protocol was designed to allow storing and managing of e-mail on the mail server; this includes allowing folders to be created to hold mail. This is a real boon to those who like to be able to access the same view of their organized mail from any reader or location. The downside is that it is a complex protocol, so different servers and readers support the it differently, making consistent configuration sometimes tricky. Also, relying on IMAP is most useful if there is enough storage space to hold all the mail you would want to access from all those locations.

Local Client Access to Mail

With web access to mail, that should be enough, right? For most people they might be, but for others, they may need access to email while offline. Offline mail readers also provide more powerful or more efficient email authoring features. In these cases, access and caching of email in a local client is essential. Access via POP3 has its downsides, but IMAP access allows the common view to e-mail, as just described.

Google Mail with IMAP support

Google has recently added IMAP access support to their Gmail service. This means that you do not have to rely on its own web interface or POP3 to access your mail. With its support for IMAP access, built-in spam filtering, its large storage size and its powerful online text capabilities, this makes a great mail service to base your mail access on.

First, follow the instructions in Supported IMAP Client List; this will give you the basic setup to get IMAP access working. I will detail information needed to make your Gmail IMAP integration more seamless. For those of you who know what you’re doing or have an e-mail reader that is not listed in on their instructions, here are the essential facts (for the rest of you, use the instructions). Note, that not all mail programs support IMAP mail access; if that is the case, you will have to drop back to using POP3 access, with its reduced features.

Protocol:  IMAP4
IMAP Server: imap.gmail.com, SSL Security=true, Port=993
SMTP Server: smtp.gmail.com, SSL Security=true, Port=465
For both the IMAP and SMTP servers the userid userid@gmail.com and the password is the corresponding Gmail password.

Using an Alternate E-mail Address

Though not an IMAP issue, if you want to use a different primary e-mail address than the userid@gmail.com address. There are two things you will want to do to make this all work:

  1. Set the email address used in sending email
  2. Forward all the email from the provider of the primary email address to your Gmail address so that all the mail will be available there.

If you do not want to expose your Gmail address as your primary address and you will want your mail recipients who use the “reply” button to direct responses to an alternate address, you will want to set the desired email address in the client. With most e-mail applications you can specify the email address you want in the configuration. This is distinct from the Gmail login information specified for the IMAP (or POP3) and SMTP servers and it is distinct from the “Reply To” address that you can optionally specify. Since there is not any validation of this address, you can set this to any address you wish.

Note that this does not completely hide your Gmail address; it is still listed in each, normally hidden, email header.

The second part of tying your primary email address to GMail is to forward all your email from the ISP to your Gmail address. The method for doing this is dependent on the your email provider. And, in order to avoid overflowing your storage capacity at your email service provider and generating error messages to people sending mail to you, be sure that the email at the ISP will no longer be saved there. You might want to try this all out first before you cancel mail delivery to your existing email account.

Special IMAP and Gmail Folders

Most e-mail readers that support IMAP create and manage some special folders:

  • Inbox — Primary incoming mailbox (created and managed by the server)
  • Drafts — The folder where the client application saves email in progress
  • Sent Mail — The folder where the client (or server) can save email that has been sent
  • Trash — The folder where deleted mail goes.
In addition, Gmail maintains some additional special “folders”
  • Starred — Flagged e-mail items are replicated in this folder.
  • Spam — The folder that temporarily holds mail treated as spam.
  • All Mail — A view of all the mail that is not in the Trash or Spam folders.

The folder names shown are the ones used by Gmail. Other client readers usually use different names for folders of equivalent functionality, thereby creating duplicate folder types. For example, Gmail uses “[Gmail]/Trash” to temporarily hold deleted mail while Outlook saves its mail in progress to “Deleted Messages.” See the following for more details on each of these special folders.

The trick is knowing how to sync these folders with the activities that occur in the mail reader. The general behavior of how Gmail treats these folders is described in the Google Help. I will describe some of the differences between how Gmail treats these folders and compared with email readers followed by some configuration details for various email readers.

Inbox Folder

The Inbox is like the old POP view of mail—a flat list of incoming messages. A big reason to switch to Gmail as the primary email server is that you can benefit from its filtering of spam from your incoming mail. The inbox also benefits from Gmail filters that “archive” mail which remove them from the Inbox view of the mail (filters are often used to tag such messages so that they would appear in different folders).

Most email readers use the same name for this folder as Gmail, so the view of this folder across email readers is consistent (whew!).

Drafts Folder

Email programs often have their own way to store email in progress. The result is that they get saved into folders of different names. If you are concerned about the status of incomplete messages, be sure to check “draft” messages in their different folder incarnations—Gmail’s name for this folder is “Drafts.”

I need to experiment with this a bit more... it might be that you’ll need to move the drafts from non-Gmail drafts folders to the Gmail one in order for you to have Gmail treat them as drafts (and vice versa).

IMAP support on the iPhone seamlessly supports these folders... see below.

Trash Folder

Similar to Drafts messages, deleted items are handled differently between Gmail and by various mail clients. This is made more complex because Gmail does not actually delete an email that you delete via a different mail reader unless that you use the mail reader to actually move the message to the “[Gmail]/Trash” folder. If the mail reader moves deleted messages to a folder of a different name, then you will have to occasionally move them manually from that folder to the “[Gmail]/Trash” folder. Failing to do this leaves the mail on the Gmail server, viewable via the “[Gmail]/All Mail” folder.

Starred Mail Folder

If the email reader has the ability to flag email, those messages will automatically be added to the “[Gmail]/Starred” folder. Likewise, Starred mail items will appear “flagged” in your email reader and appear in the “[Gmail]/Starred” folder. Moving an email to the “[Gmail]/Starred” folder will flag/star it as well, should the email reader not have that feature.

Sent Mail Folder

In general, you can configure your email program to use any accessible SMTP server to send mail. If you use the Gmail server, however, it will automatically add that mail to the Sent Mail folder for the account that is associated with SMTP server. (It can do this since the Gmail account information is necessary to access its SMTP server).

Most email clients have the ability to save a local copy of sent mail. If you use the Gmail server as the outgoing server, then there is no need to do this; it will save mail sent to the “[Gmail]/Sent Mail” folder. Because of this, it is also not necessary to set the mail program to send a copy of sent mail to yourself. Configuring your mail client to do either of these simply results in duplicate sent mail on the server.

Email Reader Program Configuration

The following describes how to set up various e-mail client programs. Should you have any experience to add, please let me know.

Netscape 7.2

Gmail Netscape
[Gmail]/Drafts [Imap]/Drafts
[Gmail]/Trash [Imap]/Trash
[Gmail]/Sent Mail [Imap]/Sent*
* Not necessary, see “Sent Mail Folder,” above.

To be completed.

Microsoft Outlook XP (aka 2002) and 2003

Gmail Outlook
[Gmail]/Drafts Drafts
[Gmail]/Trash

Deleted Items

[Gmail]/Sent Mail Sent Items*
* Not necessary, see “Sent Mail Folder,” above.

To be completed.

Microsoft Outlook Express

Gmail Outlook Express
[Gmail]/Drafts
[Gmail]/Trash

[Gmail]/Sent Mail *
* Not necessary, see “Sent Mail Folder,” above.

To be completed.

iPhone Mail Support

Gmail iPhone
[Gmail]/Drafts Draft Messages†
[Gmail]/Trash Deleted Messages†
[Gmail]/Sent Mail Sent Messages†
† Not necessary, see description.

Since Gmail IMAP support was not available when the iPhone was released, rather than using the most visible way to set up Gmail, you need to set up IMAP manually as described. The nice feature about iPhone IMAP support is that you can define how the device’s special folders map to the IMAP folders. This allows you to automatically maintain the same view of your Sent Mail, Deleted Messages, and Drafts as you’d expect to see in Gmail itself.

To be completed.

Caveats

Okay, with all that there are some road bumps that I would like to see “fixed”:

  1. If there were an option to tread a “deletion” like most email clients do, then users would not have to think so hard about whether they actually deleted a message or if it were just hidden in the “All Mail” folder. They might also support IMAP’s mark-for-deletion feature as well.
  2. I haven't tracked down the problem, but most IMAP readers have the ability to specify which folders should be automatically synchronized (subscribed). This information must be saved on the server because each time I try to exclude a folder from synchronization, it reappears as a “subscribed” folder. Because the “All Mail” folder contains a view all the mail (except those in the Spam and Trash folders), this can double the amount of disk space used locally, if you are retrieving the full content of all mail messages.