Adventures Using the Email/Text Gateway Fun and Not-So-Fun Discoveries

21 Mar 2017

What if I told you that your web app has an interface to any cellphone on the planet, smart or not, without any downloads? People don’t seem to realize it, but this totally exists. For free. Every major cell phone carrier in the U.S. provides an e-mail to text message gateway, which allows you to deliver an e-mail to anyone on their network as a text message. The opposite direction also works – users can send a text to your e-mail address.

I think this gateway is vastly underutilized. Yes, lots of applications need a shiny UI or realtime connections, but there are plenty of tools you can build with text messages as your only user input. I recently built an app that compiles a Spotify playlist based on all the songs titles it receives over text message. No fancy UI required! If you want to hear a song, just send text me the title. There are plenty of other apps that could work like this – imagine running an auction this way or taking a vote amongst your friends.

If you decide to build an app that uses the gateway, here are some tips that I wish I’d known going in.

The first and biggest hurdle is that cell carriers provide next to zero documentation. As far as I can tell, carriers provide at most one page instructing you to add something like “@vzwpix.com” to the number in order to send e-mail to it. This lack of documentation made finding these other pitfalls was extra “fun” for yours truly, but on the bright side it means these next few tidbits might save you a lot of headache.

The biggest head-scratcher was that some carriers didn’t accept e-mails from new top-level domains. It’s hard to say why when your message just disappears into a black hole, but after switching from a .plus domain (new TLD) to a .io domain (older TLD) things worked fine. My guess is that these new cheaper domains are often used by spammers as throw-away domains, and that they’ve preemptively blocked them.

Another undocumented difficultly is that people won’t be able to reply across the gateway if you try to use a third-party e-mail service. Normally, e-mail responses are sent to the address specified in the “Reply-To” header of the SMTP message, but some major cell carriers like Verizon don’t follow this rule and instead use the immediate sender’s IP address. So, if you use a service like SendGrid or Mailgun they will receive all your replies rather than whatever server you’ve specified. I tried fiddling with the e-mail headers for quite a while to get around this but without success. I would love to be wrong on this, but I really think to get replies to work you’ll need to host your own SMTP server. For what it’s worth, setting up a Postfix server isn’t that bad if you follow step-by-step instructions so don’t let this discourage you.

The third hurdle is that different carriers will convert text messages differently, i.e. the e-mail you receive can be in a variety of formats. Depending on the carrier, the message you receive could have the reply in the body of the e-mail, inside a giant HTML blob, or often in an attached text file. I won’t lie to you – figuring out how to parse each kind of response is a drag. It seems like other Python users are using these e-mail gateways, but I haven’t found any PyPI modules for automatically parsing each carrier’s format. If this is something you’re really interested in, e-mail me and perhaps I’ll try turning my code into a module.

Next, I’d recommend using a service to convert your inbound e-mails to HTTP POSTs. You could, instead, leave a long-running process to scan your e-mail inbox but then you need to think about scheduling, file locks, and passing data to your main web app process. Much easier to outsource this problem. Mailgun will convert your inbound e-mails into HTTP POSTs for free, forever. Now your task is reduced to setting up a route to handing those incoming POST requests – very standard web stuff. The contents of any attached files will also come across in the POST. It’s worth noting that (1) you don’t need a paid account with Mailgun and (2) there doesn’t seem to be any limit on how many e-mails they will convert for you. They just seem to do this service out of the goodness of their hearts. Initially I used a paid Heroku plugin to do this conversion for me, and it stings a bit knowing that I basically lit that money on fire.

These are, as far as I can recall, all the major potholes I fell into while developing my own app. These little nuggets collectively represent at least two solid weeks of frustration so I hope they help someone out there!