As a business with a need to communicate important time-sensitive information to clients and customers, it is important to use a transactional email platform that meets the main business needs.
In an effort to select the best platform for our business, we went through an evaluation process. Specifically, we were looking for a platform to deliver one-to-one emails from our applications, including:
- Sign-up Confirmations
- Password Resets
- Notification Emails
Transactional vs. Marketing Email
What is the difference between transactional and marketing emails?
- Transactional Emails are sent one-to-one, meaning there is a single sender (our application) and a single recipient (the user) for a single unique email generated by the app.
- Marketing Emails are sent one-to-many, meaning there are a large number of recipients that receive the same email. This is your classic bulk or spam blast email.
In our case, we focused on transactional email platforms.
Narrowing Down the Field
There were quite a few providers to look at, including:
If you’re looking for a more comprehensive breakdown of the pricing/features for each of these options, take a look at this great article from VentureHarbour.
Based on the feature set and pricing, we narrowed our options to Mandrill and Sendgrid.
Measuring Delivery Speed
As we utilized both platforms for a few weeks, we noticed a significant number of Mandrill emails that were not immediately showing up in recipients’ inboxes. We were seeing delivery time delays of over an hour or more in many cases.
A review of our Mandrill outbound activity feed showed that emails were being delivered to recipient servers almost immediately. However, they were not showing up in inboxes for a long period of time! Unfortunately, it’s difficult to rely on the activity feed alone to determine delivery time as transactional email providers have no control over how the recipient servers handle the emails after delivery.
Upon reviewing SMTP responses and executing further tests, it was evident that delayed emails were being throttled by recipient servers. You can read a bit more about how throttling works here and here.
The issue with the delay of emails through Mandrill was rarely seen with Sendgrid. In order to draw further conclusions and determine actual delivery speed performance, we conducted controlled tests for each platform. Our test conditions were:
- We sent one-to-one transactional emails to a variety of recipient email providers in North America.
- To give a complete picture, we wanted to compare performance when sending to major email providers (Gmail, Microsoft/Hotmail, Yahoo), as well as a major ISP and another minor provider.
- We used shared IP pools, and did not have a dedicated IP address.
- Tests were conducted over several random days.
- Obviously, it would be nice to have some long-term data, but we had to limit the time and resources we put into our testing.
Sendgrid has Faster Delivery Times than Mandrill
Testing results are displayed below.
The left column shows the recipient email provider (actual email addresses have been removed for privacy reasons). We used the same 8 recipient email addresses for both Sendgrid and Mandrill. The columns under “Receieved Rate at Given Time Intervals” show the percentage of emails that actually showed up in the recipient’s inbox after a specific period of time.
As indicated with the figures above, Sendgrid significantly outperformed Mandrill in our tests. This confirmed what we had been seeing anecdotally over the past several weeks.
The conclusion: Mandrill was throttled by email providers at a much higher rate then Sendgrid.
We believe that critical, time sensitive transactional emails (like sign-up confirmations or password resets) should be delivered quickly (within one minute). Given the discovered results, Mandrill was proven to be the lesser option while Sendgrid was far superior.
Footnote: Mandrill’s Bait and Switch
In February of this year, Mandrill significantly changed their pricing and eliminated their free plan. There’s been a pretty big outcry following the announcement, which you can read more about if you’re interested.