VERP on Rails19 Jan 2014
We sent newsletter campaigns from our Rails app. One of the main requirement of such campaign is how many emails bounced?. We need to track all bounced emails and map them again to a specific campaign because there will be multiple campaigns going on all the time.
To track bounced emails efficiently, we need to set unique return path
for every recipient. This technique is called
Variable Envelope Return
Path or VERP
Postfix supports VERP with
-V switch. For eg.
sendmail.postfix -V -f bounced
will generate return path for
If you want more control and more information like from which campaign this email was sent, we need to send more information in the return-path.
So it is best to generate our own return path pattern and parse it
once it is received. We generated a pattern where return-path will
generated from all the required information for tracking.
This pattern was given to
mail( from: 'firstname.lastname@example.org', to: 'email@example.com', return_path: generate_verp_pattern )
So now the first problem was solved. We were sending unique return-path for every email that was going out from the system.
Next part is to track it once it bounces and update database.
Postfix allows piping incoming email to a particular address to a
script. So if we pipe all incoming emails to
our script then we can parse the incoming address and update database.
For this, we have to edit
/etc/postfix/aliases)file as follows:
This means all incoming emails to
firstname.lastname@example.org will be
piped to our script. This actually means that the whole email with
body, headers, attachments etc is forwarded to our script.
/etc/aliases file is a text file that is used by postfix as a
table to redirect mail for local recipients. To rebuild this table
after a change, we need to run
This will rebuild the table for postfix. Now we have completed the second part of the process.
The shell script will get the bounced email content now. There can be multiple bounced emails generated at the same time. So we can’t directly pass them to rails runner scripts or rake tasks. Because that will kill our server by launching multiple rails instances. Instead we need to use some background tasks mechanism.
We were already using
resque, so decided to use it for bounced emails
also. So a resque worker will actually update the database. Our script
just has to enqueue the job for resque.
We broke this enqueuing process into two parts.
First - A shell script which will use correct RVM Ruby version and call the ruby script.
Second - A Ruby script which will enqueue the job to resque.
So the shell script looked like -
#!/bin/bash rvm use ruby_version@ruby_gemset ruby /path/to/ruby/script
And Ruby script looked like
require 'rubygems' require 'resque' # Adds the incoming bounced_email to background job class BouncedEmail def initialize(content) Resque.enqueue_to(:bounced_email_receiver, 'BouncedEmailReceiver', content) end BouncedEmail.new($stdin.read)
We had to go in 2 steps here because we had multiple apps using multiple rubies on same server. If you have only one ruby then you can make a executable ruby script instead of shell script which decides which ruby to use.
Now its upto resque worker to parse the content and update database.
For that, we used bounced_email gem which
detects lot of things such as
type of failure
etc. As it is integrated with
which was unique pattern generated by us only) and were able
to parse it to update the database. With
bounced_email we got some
more relevant information for free :)