Some time ago, a user, in response to an email we sent out to everyone outlining some new app updates, said that he did not feel comfortable with us using Mailchimp to send out newsletters. Privacy is first and foremost on our list of priorities, and this user had a great point. But, if not Mailchimp, how else could we manage to send emails on a large-scale basis? There aren’t really any privacy-focused email services, nor am I even sure what that would look like. The only solution was building our own.

Building our own was something we were reluctant to undertake. Forums on the web speak of the unthinkable dangers involved in managing your own email service. There’s just too much infrastructure that needs to be set in place before one can deploy a functioning mass email system, including handling bounces and complaints. Not to mention the scary aspect of risking a poor deliverability rate if these components are not handled properly.

Challenges and impossibility aside, we had no choice in the matter. Privacy is important, and the less dependencies we have on fluctuating third-party privacy policies, the sounder we can sleep at night.

As of June 10, 2018, Standard Notes sends emails completely independent of Mailchimp, including mass emails. We use a custom built architecture on top of SQS and SES, which we share below, that fulfills our simple requirements in ways Mailchimp couldn't.

There are a few components worth mentioning:

Unsubscribe Mechanism

One of the most limiting aspects of Mailchimp, and many other newsletter service providers, is the default unsubscribe mechanism. Out of the box, you only get a “Unsubscribe from all email” option. What if you want to give users the option to unsubscribe from only a certain subset of emails but still receive other important email? You may be able to pull it off in Mailchimp using lists, but it’s unwieldy and difficult to customize.

One of the first things we designed in our system is a new kind of email subscription system that’s easier for both users and company alike to manage.

Each user in our system receives an EmailSubscription object. Each subscription carries a level field, which indicates the level of email this user has indicated they wish to receive. For us, this range is from level 0, which is completely unsubscribed, to level 3, which is the “hear everything we have to say” option.

Each email we send will have two options in the footer:

• Decrease email level
• Unsubscribe from all email

When a user clicks either option, they are taken to this page:

Pressing unsubscribe sets their email level to 0, and pressing decrease will decrement their existing level. This model makes thinking through subscriptions easier, and makes adjustments feel more natural. From a user perspective, Unsubscribe is traditionally a very permanent action, without an easy way to recover from.

Using email levels makes changing preferences non-permanent, and a user can quickly go up or down on the levels, depending on what feels right to them. We’ve personally always been a "Level 2" sort of company, sending an email on the order of once every month or two, but a large part of that was probably due to using an inflexible email system.

Technical Overview

Job Queues

We use Simple Email Service from AWS as our email provider. The tricky part was working with SES’s maximum send rate, which is the number of emails you can send per second. Ours is not too large, so we had to make sure that our queuing architecture didn’t dequeue faster than our limit.

We use Shoryuken to integrate our Rails application with AWS’s SQS. Shoryuken is a well designed open source library that makes integrating with SQS extremely simple.

Our email sending limit L was 28 per second, so we had to make sure that no more than 28 jobs ran per second. To do this, we configured the Shoryuken concurrency value to a little less than 28. On average, each email sending transaction takes 0.5 seconds, so we measure the time difference between start and finish, and if it’s less than 1s, we sleep for the difference. This ensures that no more than L jobs are handled per second. There are likely better ways to handle this, but for our size, this solution works well.

SMTP vs HTTP

Rails comes with easy SMTP integration out of the box. However, managing the lifecycle of ActiveMailer jobs is not particularly straightforward. It was important we know when an email delivery began and ended, especially with regards to our queueing limitations. With ActiveMailer, pulling this off on a per job basis was tricky and tacky. SES provides an HTTP based API with an easy aws-sdk-ses gem. This allowed us to track requests on a per job basis using familiar begin/rescueblocks.

Sending an email using the HTTP API is straightforward:

subject = campaign.subject
htmlbody, textbody = campaign.get_html_and_plain(transaction)
encoding = "UTF-8"
ses = Aws::SES::Client.new
params = {
destination: {
to_addresses: [
recipient,
],
},
message: {
body: {
html: {
charset: encoding,
data: htmlbody,
},
text: {
charset: encoding,
data: textbody,
},
},
subject: {
charset: encoding,
data: subject,
},
},
source: sender
}
resp = ses.send_email(params)

Email Templating

Rendering and styling emails from a template file was another tricky part of our implementation. If you’re using ActiveMailer, this is automatically handled and made tremendously easy. For our implementation, we needed to dynamically read a layout file (contains shared HTML, like headers and footers) and a template file (specific per email), combine the two, replace any ERB tokens (<%= user.unsubsribe_link %>) with proper values, and finally, and probably most importantly, apply CSS styles to elements inline. Finally, we needed to do all this in a performant manner.

The solution that worked best for us was to precompile whatever parts of the template we could as part of the build process, and dynamically handle as little as possible per email sent.

We used Premailer, which in my experience has been a must in making emails look good. Premailer will apply CSS styles to HTML elements inline, ensuring proper compatibility across all email clients. However, Premailer can be slow, and we found that when rendering templates and styles dynamically per email sent, each transaction would take 5 seconds to complete. That’s no good. What we needed was a way to precompile templates with styles before run time. This was tricky, but here’s how it works:

Each email campaign is an object with a precompile method, which is run during build time:

def precompile
layout_path = "#{TEMPLATE_ROOT}/layout.html.erb"
layout_raw = File.open(layout_path).read

template_path = "#{TEMPLATE_ROOT}/#{self.template}"
template_html = File.open(template_path).read
result = layout_raw.gsub("<%= yield %>", template_html)
premailer = Premailer.new(
tokenized_text,
:with_html_string => true,
:css => [
"public/assets/mailers/style.css",
]
)
premailed_text = premailer.to_inline_css
path = "#{TEMPLATE_ROOT}/generated/#{self.template}"
File.open(path, "w+") do |f|
f.write(premailed_text)
end
end

Then, during runtime, and for every email sent, we render the precompiled template with proper user values:

def get_html_and_plain(user)
template_path = "#{TEMPLATE_ROOT}/generated/#{self.template}"
template_html = File.open(template_path).read

rendered_template_html = ERB.new(template_html).result(binding)
HtmlToPlainText is part of Premailer
include HtmlToPlainText
plain = convert_to_text(rendered_template_html)
rich = rendered_template_html
return rich, plain
end

The result: Each email transaction takes less than half a second to render and send. Success.

What’s Next

We're ecstatic to be closing down our Mailchimp account, saving quite a bit on monthly costs, and more importantly, taking stricter measures to protect user privacy by removing dependencies on capricious third-party privacy policies. Email is an important part of any web company, so it’s worth putting the time in to build a well-fitted solution.

If you want to build your own simple email campaign system for your Rails app, you can check out our recipe, which includes the classes, controllers, and jobs we used in our implementation.

What is Standard Notes?

We build an open source, encrypted notes app that respects user privacy and productivity. Standard Notes features a suite of simple cross-platform applications with seamless sync, and an extensions system that offers a wide range of editors (including Markdown, Tasks, Vim, and Code), themes, automated cloud backup options, and other useful features.

You can learn more at standardnotes.com.