---
created_at: '2013-08-12T13:30:14.000Z'
title: LavaBit's Architecture (2009)
url: http://possibility.com/LavabitArchitecture.html
author: yankcrime
points: 130
story_text: ''
comment_text: 
num_comments: 57
story_id: 
story_title: 
story_url: 
parent_id: 
created_at_i: 1376314214
_tags:
- story
- author_yankcrime
- story_6199050
objectID: '6199050'

---
## Getting to Know You

### \* What is the name of your system and where can we find out more about it?

Lavabit  
[http://lavabit.com](http://lavabit.com/)  
  
<http://lavabit.com/network.html>  
<http://lavabit.com/about.html>

### \* What is your system for?

Lavabit is a mid-sized email service provider. We currently have about
140,000 registered users with more than 260,000 email addresses. While
most of our accounts belong to individual users, we also provide
corporate email services to approximately 70 companies.

### \* Why did you decide to build this system?

We built the system to compete against the other large free email
providers, with an emphasis on serving the privacy conscious and
technically savvy user. Lavabit was one of the first free email
companies to provide free access via POP and later via IMAP. To this day
over 90 percent of our users access the system using POP or IMAP.

### \* How is your project financed?

The project was initially financed by the founders, but now lives off
money collected via advertising and paid users. Ongoing development
efforts are subsidized by our consulting business; quite simply we work
on the code base for Lavabit when we have slowdowns in our consulting
business.

### \* What is your revenue model?

Offer a superior product and hope that its increasing use leads to
advertising revenues, and paid account upgrades.

### \* How do you market your product?

We rely on word of mouth to grow the service. Since most of what we
provide is free, we can't justify the cost of advertising (at least
right now).

### \* How long have you been working on it?

The service has been running since the summer of 2004. Originally we
called the service Nerdshack, but changed the name to Lavabit at the
request of our users in December of
2005.

### \* How big is your system? Try to give a feel for how much work your system does.

Every day the system handles approximately 200,000 email messages, while
rejecting another 400,000 messages as spam. Lavabit currently averages
about 12,000 daily logins, of which 80 percent are via POP, 10 percent
are via IMAP and 10 percent are via the webmail system. The website
itself sees about 2,500 unique visitors per day, resulting in
approximately 170,000 page and file requests.

### \* Number of unique visitors?

Approximately 12,000 unique visitors per day and approximately 28,000
unique visitors per month.

### \* Number of monthly page views?

3,728,686 for Jan 2009  
3,929,292 for Feb 2009  
These numbers only consider HTTP requests.  

### \* What is your in/out bandwidth usage?

We currently send about 70 gigabytes per day through our upstream
Internet connection. See this page for a graph:  
  
<http://lavabit.com/grahps.html>

### \* How many documents, do you serve? How many images? How much data?

Our system currently handles approximately 180,000 inbound emails, and
another 20,000 outbound emails per day. This translates into about 70
gigabytes of traffic.

### \* How fast are you growing?

We see about 150 new user registrations per day.

### \* What is your ratio of free to paying users?

We currently have approximately 1,500 actively paying customers.

### \* What is your user churn?

Our daily login average has recently been growing by about 250 per
month. We hope to grow a lot faster when our new website and webmail
system launch later this year.

### \* How many accounts have been active in the past month?

34,247 between 2/10/2009 and
3/10/2009.

## How is your system architected?

### \* What is the architecture of your system? Talk about how your system works in as much detail as you feel comfortable with.

**For SMTP, POP and IMAP Connections**  
We use a 2-tier architecture. There is an application tier that runs our
custom mail daemon and a support tier made up of NFS and MySQL servers.
A hardware based load balancer (Alteon AD4) is used to split incoming
SMTP, POP and IMAP connections across the 8 application servers (Dell
1650's with 4gb of RAM). The application servers also run memcached
instances.

The application servers are used to handle the bulk of the processing
load. The daemon itself is a single process, multi-threaded application
written in C. Currently each daemon is configured to pre-spawn 512
threads for handling incoming connections. Another 8 threads are used to
asynchronously pull ads from our advertising partner's HTTP API, and
perform maintenance functions. Maintenance functions involve updating
in-memory tables, expiring stale sessions, log file rotation and keeping
the ClamAV signatures up to date.

From an architecture standpoint, each incoming connection gets its own
thread. This allows us to use blocking IO. We currently rely on the
Linux kernel to evenly split the processor among the connections.

We currently call our mail daemon "lavad" and it fluently speaks SMTP,
POP and IMAP. The daemon is also responsible for applying all of our
business logic and interfacing with the different open source libraries
we use.

When accepting messages from the outside world via SMTP, the daemon will
perform the following checks:

  - If the recipient is valid
  - Whether the incoming IP is listed on an RBL
  - If the return path can be validated using SPF (libspf2)
  - Against any size or rate limits for the account
  - Against the user’s gray list
  - For viruses (libclamav)
  - For a valid domain key signature (libdomainkeys)
  - Whether the message looks like spam based statistical token data
    (libdspam)
  - And finally against any filters used to sort or delete messages
    matching a regular expressions

Whether a specific check is used depends on the user’s preferences, and
the account plan they have. For example, the spam filter is limited to
paid users (because of the load it places on the database).

Depending on the outcome of the different checks, the user can choose to
label the message, reject it, or in some cases delete it silently. If
the message needs to be bounced, a bounce message is only sent if the
return path can be verified using a) SPF or b) if the sender is verified
using domain keys, and the sender matches the return path.

As a final step, the message is encrypted using ECC (if applicable),
compressed using LZO and then stored on the NFS server.

For POP connections, the process is relatively simple. The user
authenticates and requests a message. The daemon loads the message,
checks the hash to make sure the date hasn’t been corrupted,
decompresses the data, and then decrypts it (if applicable) before
sending it along to the client.

Because we need the plain text password to decrypt a user’s private key,
we don’t support secure password authentication. We decided to support
SSL instead (which encrypts everything; not just the password). We
handle the SSL encryption at the application tier rather than on the
load balancer because we feel the application tier is easier to scale.

On a side note are failure to support secure password authentication
hasn’t stopped people from clicking the Secure Password Authentication
checkbox in Outlook, and creating a support nightmare for us. Outlook
doesn’t enable SMTP authentication by default either, so that creates
another support nightmare for us. If any mail client developers read
this; please start making port 587 the default instead of port 25, and
auto detect SMTP authentication.

When retrieving messages for users that have the statistical spam filter
enabled or users who have selected a plan with advertising, the daemon
will also insert small text signature. The signature will have a link
for training the server side spam filter and/or a small text
advertisement.

For IMAP connections, the daemon also presents messages in folders and
allows server side searches of messages. Currently searches are handled
by reading in all of the message data from disk; which results in a
large performance hit if the folder is large. If the user connects
multiple times using the same credentials, the connections will share a
centralized copy of their mailbox state, which also creates lock
contention issues. Search is certainly one area where we need to
improve.

For outbound messages, the daemon will authenticate the user's
credentials against the database, apply any sending limits for the
account (to prevent abuse), check whether the From address matches an
email address associated with the credentials provided (to prevent
spoofing), and finally the daemon checks whether the message contains a
virus. Assuming all of the checks are passed, the message is cleaned up,
signed using domain keys, and then relayed via our internal network to a
Postfix server that handles relaying it to the final destination.

The daemon uses pools for sharing anything it can, including MySQL
connections, ClamAV instances, cURL instances (for pulling ads),
Memcached instances, libspf2 instances, etc. To keep deployments simple,
we compile all of the open source libraries we rely upon into a single
archive that is then dynamically loaded at runtime. We don’t compile the
libraries directly into our application because doing so would require
us to release the daemon under the GPL, and we don’t rely on dynamic
linking since we don’t want any of the key libraries to be automatically
updated by the operating system without us knowing.

**For HTTP Connections**  
Like inbound mail connections, HTTP connections are split between two
servers using the load balancer. Apache is currently used to handle the
requests. While most of the website is static XHTML files, our
registration engine is written in C (with libgd for the CAPTCHA images,
and libcurl for processing credit cards using the PayFlow Pro HTTP API).
All of our C applications rely on the Apache CGI interface.

The preferences portal is currently written in Perl and the webmail
system is based on a popular open source client and is written in PHP.
We modified the webmail system to fit more smoothly into our site. The
webmail client currently connects to the mail system using IMAP, with
each web server getting a dedicated IMAP
server.

### \* What particular design/architecture/implementation challenges does your system have?

**The Big Problem**  
While it is very easy to setup a mail system that reliably handles email
for a few thousand users, it is incredibly difficult to scale that same
system beyond a single server. This is because most email servers were
originally designed for use on a single server. If you grow these same
systems beyond a single server you typically need to use a database
and/or a NFS server to keep everything synchronized between the
different nodes. And while it is possible to build large database and
NFS instances, it is also very expensive, and depending on the setup it
can be very inefficient.

If you want to avoid the single database or NFS server problem, you can
do so by adding a lot of complexity. For example, if you wanted to
implement a very large Cyrus system the typical solution is to use LDAP
for authentication, and then use an IMAP/POP reverse proxy to intercept
incoming connections and forward them to the specific Cyrus server for
that user. The problem with managing a system like this is the
relatively high number of critical pieces that can fail. The following
image visualizes what a system like this might look like:

![Embedded
Image](data:image/png;base64,[encoded data, 113004 characters])  

For a full write up on this design, see
<http://www.linuxjournal.com/article/9804>

The problem with systems designed this way is the large number of
critical services. If a Cyrus server goes down, then all of the users
hosted on that system are offline. You can mitigate this risk with a
failover system that is periodically rsync’ed with the master, or by
using a SAN, but these options are either inefficient or expensive.
(There is at least one medium sized free/paid email company that uses a
system like this, and presumably this is why the limits on their free
plan are so low.)

At one point Yahoo Mail relied on a very large NetApp device to
centrally store mail. And while the NetApp devices scale well, they are
also very expensive. It was this high cost that kept Yahoo Mail from
matching Gmail’s 1GB quota for almost a year. When they finally
completed their move to a distributed architecture in 2007, they began
offering unlimited storage.

It is precisely because of how easy it is to implement small mail
systems, and so difficult to implement large mail systems that the
Internet sees literally hundreds if not thousands of free email
companies start and then fail each year. If the system becomes popular
they have no easy way to scale it, so they are forced to either stop
accepting free accounts, or shut down the system. Only a small number of
companies have the financial and technical resources to build systems
from the ground up to support a large user base. It is also why Lavabit
followed the example of the large providers, and implemented a
completely custom platform; it was the only way we could support 100k+
users with a cost basis low enough to make the business profitable.

The key to keeping any large system manageable and cost effective is to
keep it simple. The fewer critical failure points you have in the
system, the better things will run.

**Current Problems with Our Implementation**  
We have locking issues when multiple IMAP connections try to access the
same mailbox; only one thread is currently allowed access to the mailbox
at a time, which can present a problem if the user makes a request that
takes a long time to process (ie search or bulk fetches via IMAP).

Reading in the full message when only the header is needed has caused a
performance problem. We also need to implement code for indexing
messages and processing searches using an index instead of reading in
all of the data for each search request. Unfortunately we don’t have
dedicated search gurus on hand to help with this like our competitors.

The statistical spam filter currently stores all of its token and
signature data in the database. Checking the tokens in a message against
a database using SQL is very inefficient. And because of how difficult
it is to scale MySQL databases, doing this for 200,000+ messages a day
would be very expensive. At some point we will change the way token data
stored so that the load is more evenly distributed across the cluster
and then we will be able to offer the filter to everyone.

The webmail system does not keep IMAP connections open. As a result data
is freed and then reloaded frequently. Also because the whole message is
loaded when the webmail system is only requesting the header, a lot of
unnecessary data is often pulled from the NFS servers.

Naturally were working to fix all of these issues.

### \* What did you do to meet these challenges?

We implemented a custom mail system, which was designed from the ground
up to efficiently handle a large number of users. A custom platform has
also allowed us to implement a lot of custom business logic that would
have been difficult, if not impossible using an off-the-shelf system.

### \* How did your system evolve to meet new scaling challenges?

We started with a single server that used Postfix and Qpopper. We used
amavisd for virus and spam filtering and a custom policy daemon to make
sure users didn’t send to much mail, or spoof someone else’s address.
This system worked well for about 4 months; but with a few thousand
users the system started to choke. We had to turn off new user
registrations until we could transition onto a custom multi-server
platform.

Originally there was an application for handling SMTP connections and a
separate one for POP connections. Each application would spawn multiple
processes with multiple threads so that if one died from a segmentation
fault another could take its place. (This is how Apache and Postfix are
designed for both reliability and security.) Over time we combined the
protocols into a single process which spawns a larger number of threads.
We could do this because over time we worked out the memory bugs. In the
last year we’ve only had 4 nodes die from segmentation faults, all of
which were triggered by bugs in the libraries we use. (But we also
haven’t released any major changes into production in the last year.)
This single daemon design has made our use of things like database
connections much more efficient.

### \* Do you use any particularly cool technologies or algorithms?

The way we encrypt messages before storing them is relatively unique. We
only know of one commercial service, and one commercial product that
will secure user data using asymmetric encryption before writing it to
disk. Basically we generate public and private keys for the user and
then encrypt the private key using a derivative of the plain text
password. We then encrypt user messages using their public key before
writing them to disk. (Alas, right now this is only available to paid
users.)

We also think the way our system is architected, with an emphasis on
being used in a cluster is rather unique. We would like to someday
release our code as free software. We haven’t yet because a) we don’t
want anyone else building a competing system using our code, b) while
we’ve moved more settings and logic into a configuration file over the
last couple of years, there is still a lot of logic hard coded, and c)
we’ve created the code specifically for Cent OS, and don’t have the
resources to test and support it on other operating systems right now.
We’ve spent some time looking for a company to sponsor open sourcing the
code, but haven’t found one
yet.

### \* What did you do that is unique and different that people could best learn from?

One of the ways to gain an advantage over your competition is to invest
the time and money needed to build systems that are better than what is
easily available to your competition. It is the custom platform we
developed that has allowed us to thrive while many other free email
companies either stopped offering their service for free, or shut down
altogether.

That said, you should always start by improving the components that will
make the most difference to your users, and move on from there. For
Lavabit that meant starting out with a custom mail platform, but
continuing to use Postfix for outbound mail, MySQL for synchronization,
and NFS for file storage.

This may be a good place to note that in 2004 the only major database
system with production ready cluster support was Oracle and it remains a
very expensive option (way beyond the budget of a free service like
ours). Since then SQL Server and MySQL have both improved/added support
for clustering (replication and failover is \_not\_ not the same as
cluster support). And while the MySQL cluster implementation still needs
work before developers can stop worrying about the scalability of the
database, the world is getting closer to that point. Distributed caching
(memcached) and distributed file systems (Lustre/GFS) have also matured
since we started in 2004. Throw in cloud services like S3 into the
equation and it is almost easy to implement a highly scalable website or
service.

### \* What lessons have you learned?

The only way to guarantee success is through hard work.

### \* Why have you succeeded?

We are committed to providing a superior service and offering it on
terms we think all users should be demanding. We are also committed to
continually improving the service we offer.

### \* What do you wish you would have done differently?

There are a number of areas in our platform we wish were implemented
differently. In most cases we made the decisions we did because
implementing them the "right" way would have taken longer.

A good example is the IO model were using. The asynchronous IO model
used by lighttpd and memcached is more efficient than our current model,
but we felt doing things this way would have taken longer while giving
us little initial benefit. See this quite famous web page for a full
write up on the issue:  
<http://www.kegel.com/c10k.html>

We also wish we had finished the IMAP server earlier than October of
2007, and finished our custom webmail system by now.

### \* What wouldn't you change?

We are happy with the decision to enter the email service business.
Overcoming the challenges involved in building a reliable and scalable
mail platform has been rewarding.

Personally I also enjoy knowing that the system I helped create is being
used by 12,000 each day. Sometimes I find myself thinking "there are
1,000 people connected to this system right now." I like those numbers.

### \* How much up front design should you do?

Collectively, the engineering team has spent thousands of hours doing
research to help make our mail system better. This knowledge has been
invaluable not only in improving the mail system, but also in helping
our professional services clients.

The bottom line is that it is easier to make changes to a design
document than it is to the code. What this means is that if you don’t
clearly understand how something should be implemented, it pays to write
design documents first. The hours you save in the end will far outweigh
the hours you spend writing the documentation.

### \* How are you thinking of changing your architecture in the future?

We’ve had a major update to our website and our application tier in the
works for almost a year already. The details involved in this update
(outside of what has already been mentioned here) are still a secret. I
will definitely need to update this write up when we are ready to push
the development tree into production.

## What infrastructure do you use?

### \* Which programming languages does your system use?

The main application daemon is written in C. Web pages generally use
XHTML, CSS and Javascript.

We still have a number of legacy web applications and maintenance
scripts written in Perl, and the webmail system is currently in PHP, but
these are all slated for conversion to C as time allows.

Our consulting projects typically involve development in C\#; and we
spent a lot of time thinking about whether to implement the system in
C\# back in 2004. In the end, we felt that Windows and .NET would not be
a good choice for a scalable mail platform. We felt that the increased
performance, the lack of licensing costs, and the availability of so
many open source libraries for handling mail meant that the best choice
for us was to go with Linux and C.

If we had to make a similar decision today, there is a chance we would
not have chosen to go with C. Given the stability and efficiency of
Windows 2008, the growing amount of open source C\# code on the Internet
and the availability of Mono as an alternative to .NET, we may have
opted for C\# instead.

In our experience, the decision on what platform to choose for a project
can often be broken down into simple math. Building applications in C\#
is typically faster than C. For us that typically means 3 to 4 times
faster than using C, and about 1.5 faster than using PHP. If you can
figure out how much additional development time it will take to use one
platform over another, it becomes easy to calculate whether the
performance and license savings of one platform will offset the
increased cost of development. In general hardware and software is cheap
compared to development time, so the number of applications which can
justify being built in C or C++ is getting very small.

On a side note, we think the productivity gap between IIS/.NET/SQL
Server/Visual Studio and Apache/PHP/MySQL/Eclipse is largely a result of
how well the Microsoft tools have been integrated with each other.

### \* How many servers do you have?

We have 14 servers dedicated to the mail platform. We have 1 server
dedicated to monitoring, and another 11 servers used for website
hosting. Most of the websites we host were developed by the Lavabit
team, so we don’t consider it part of our core business.

### \* How is functionality allocated to the servers?

We move services from one server to another as necessary. Our long term
goal is to create a global pool of servers than can handle everything,
so fewer clusters are needed, and the load is distributed more evenly.

### \* How are the servers provisioned?

We typically buy servers on eBay, and then install and configure them
ourselves.

### \* What operating systems do you use?

The mail system is currently using Cent OS 4. The application servers
use the 32 bit version, and the database and storage servers use the 64
bit version.

We also use Windows 2003 for hosting other things not related to the
mail service.

### \* Which web server do you use?

Apache 2.0

### \* Which database do you use?

MySQL 4.1

### \* Do you use a reverse proxy?

Our load balancer will route connections from the same IP to the same
node, which tends to make caching easier, but we do not use a "true"
reverse proxy to route connections based on user credentials.

### \* Do you collocate, use a grid service, use a hosting service, etc?

We’ve created our own platform. We lease space inside a colo facility to
host all of our equipment.

### \* What is your storage strategy?

The storage servers have 3ware RAID cards, and use SATA drives. We are
currently using the ext3 file system, and share files via NFS. The
database servers have PERC 4 cards and use SCSI.

### \* How much capacity do you have?

Overall, we are only using about 10 percent of our total processing
power, and about 25 percent of our currently subscibed bandwidth. The
two areas where we currently have issues are the IMAP servers dedicated
to handling webmail connections, and disk throughput on the NFS servers.
Throughput on the NFS servers is limited by the IO controller cards, and
the file system in use (ext3). Ultimately we feel there is a lot of room
for growth available to us by making our code more efficient.

The servers in the "earth" cluster are only averaging about 8 percent
utilization. This 8 node cluster is used to handle inbound SMTP, POP and
IMAP connections, and run memcached instances. A typical CPU graph for a
server in that cluster looks like:

![Embedded
Image](data:image/png;base64,[encoded data, 105444 characters])

As you can see the servers allocated to the "earth" cluster still have
plenty of room for growth. In contrast, if you compare this to a CPU
graph from one of the servers in the "mars" cluster you will notice one
of our current capacity problems. The "mars" cluster is made up of the
two nodes dedicated to handling IMAP connections from the webmail
system, and the CPU graphs for both nodes looks like:

![Embedded
Image](data:image/png;base64,[encoded data, 149492 characters])

As you can see from the above graph, this server is having issues. Both
servers dedicated to handling webmail traffic will get memory upgrades
sometime next week which will help, we also have plans to push a new
build with some minor improvements that we hope will help with the
problem. This utilization issue has only shown up in the last month or
so.

Some of you may be wondering why we don’t use servers from our "earth"
cluster to handle more of the webmail load. The source of this
utilization issue is the way our code is written. Adding more hardware
to the application layer would actually make the problem worse, as it
would further stress our NFS servers, and hurt the performance seen by
users not connected via the webmail system. The root problem is that
when a request comes in via IMAP for a message header our code loads the
entire message off the NFS server. As a result instead of having to make
a 4kb read per email we are pulling what in some cases might be a 128
megabyte message off the NFS server. The result is about 100 times more
data is transferred than is actually needed to satisfy the IMAP request.
We’ve tried using memcached to cache messages, but unfortunately too
many of the messages are larger than 1 megabyte and can’t be stored in
memcached.

Why didn’t this problem show up sooner? Most IMAP clients store a copy
of the message header locally, so they only need to download new message
headers when they connect. However because the webmail client is
stateless, it needs to download the message headers every time a user
logs in. Because our current webmail system has only been available
publicly for about a year, it took some time for webmail traffic to grow
large enough to make this problem critical.

From a database standpoint we still have plenty of capacity. Here is a
CPU graph of our master database:

![Embedded
Image](data:image/png;base64,[encoded data, 111664 characters])

We’ve spent a lot of time optimizing our database usage because of how
difficult it is to scale out a database.

Unlike our database servers, the NFS servers are currently disk bound.
While I don’t have a graph of disk utilization (it averages 95 percent),
I do have a graph of the network utilization:

![Embedded
Image](data:image/png;base64,[encoded data, 163360 characters])

As you can see, these servers are being seriously taxed. I should note
that 85 percent of this traffic is for the two IMAP servers handling
webmail traffic. We are hoping to push a new build that will fix this
problem soon.

And finally, our two HTTP servers running Apache are quite happy as you
can see from this utilization graph:

![Embedded
Image](data:image/png;base64,[encoded data, 116644 characters])

The spike is from the nightly slocate and rsync cron jobs.

### \* How do you grow capacity?

We added 8 application servers about 1 year ago, and we still have
plenty of capacity. We ordered more memory for the nodes in the "mars"
cluster, and will install it next week. We also have plans to add more
memory to our database server and hard drives to our NFS server this
summer.

If our growth rate remains close to what it has been historically, we
will to keep using our current hardware until the summer of 2010. At
that point we are hoping to make the jump from Cent OS 4, to Cent OS 6,
and move the application tier from 32 bit hardware to 64 bit hardware.

### \* How do you handle session management?

We use sticky sessions. The load balancer routes incoming connections to
nodes based on the incoming IP. Session state is then maintained
internally by our application using custom code. In a handful of
specific scenarios we also serialize a portion of the session state data
and then store it in memcached so it is available to the other nodes.

### \* How is your database/datatier architected?

We have a master database which replicates critical tables to a slave
server for backup purposes. All of our application traffic goes to the
master database, since the slave server uses significantly older
hardware. We pull the offline backups from the slave database server
using mysqldump. We can afford to have the slave tables locked for
extended periods because no production traffic goes to it.

Our overall strategy is to use highly optimized SQL with no views,
stored procedures or triggers. We only use joins when absolutely
necessary. As a result, we have gotten very good throughput from our
database server. We also place an emphasis on minimizing database use by
caching anything we can either at the application level or with
memcached. We try to save CPU cycles on the database server by
performing as much work as we can on the application servers (sort
operations are good example of that). Our theory is that even if it
takes ten times the amount of work to sort a result set at the
application layer, it is a hundred times easier to scale out the
application cluster than it is to scale out the database servers.

Stats from our primary database server (as of March 11th, 2009):

Uptime: 495 days 2 hours 37 min 17 sec

Questions: 4896924841

Queries per second avg: 114.474

And some typical performance numbers:

% user % nice % system % iowait % idle

1.35 0.00 0.60 0.48 97.57

The hardware for the database server was acquired in 2005 and is a dual
Opteron, with 8 GB of RAM, and 6 SCSI drives in a RAID 5 configuration.

### \* Which web framework/AJAX Library do you use?

Jquery on the client side. We have our own C framework on the server
side.

### \* How do you handle ad serving?

We download the ads from our current ad network via their HTTP interface
and then insert them into messages as they leave the server via POP or
IMAP.

While there are a lot of ad networks on the Internet today, we have only
found a couple that provide an API which allow us to download ads onto
the server and then insert them into emails (or web pages). It seems the
more widely used strategy is to use Javascript on the client, which in
our opinion is an inferior delivery method.

### \* What is your object and content caching strategy?

We currently use memcached for storing objects, and the libmemcached
library for interfacing with memcached servers. We’ve also implemented
custom C code for handling serialization.

One major drawback we’ve run into is that memcached is not designed for
caching objects larger than 1 megabyte. This is a problem for us, as we
like to store the compressed (and in some cases encrypted) version of
emails in memcached to reduce the load on our NFS servers. Since a
significant percentage of emails are larger than 1 megabyte, we can only
partially implement this strategy.

We have looked into increasing the size limit for memcached but found
that the current memcached codebase doesn’t handle small objects as
efficiently when the slab size is increased. In the future we may start
breaking large messages up into chunks and spreading them across
servers.

### \* Which third party services did you use to help build your system?

We’ve used contract programmers in the past to develop tools for
managing the platform; but these days we prefer to keep things in house.
We currently don’t rely on any 3rd parties for anything critical (other
than our colo provider for space, power, cooling and bandwidth). We
don’t like the idea of making excuses if another company makes a
mistake. We also believe that given the nature of the service were
providing keeping things in house results in competitive advantages.

### \* How do you health check your server and networks?

We have Nagios setup with just over 500 different checks (across 24
total hosts). A number of the checks are custom Perl scripts that can
check specific things inside our database, or at the application layer.
We also have a third party service checking for sanity on our critical
load balanced ports just in case something breaks between our network
and the outside world.

### \* How you do graph network and server statistics and trends?

We use Cacti. And while Cacti is not the easiest tool to use, it is both
powerful and free. One of our engineers spent a week or so and learned
enough about Cacti to create the 120 or so different graphs we currently
have setup.

### \* How do you test your system?

We’ve written several command line programs that emulate the various
protocols for testing during development. When were ready to deploy we
will also test the build with several of the more popular email clients.

When were ready, we will deploy a new build onto a single node and let
it run for a week or more to make sure it is stable. During that time we
monitor support emails and log files for any hidden problems.

### \* How do you analyze performance?

We pull statistics into Cacti via SNMP and custom scripts, and then
monitor the resulting graphs. We then focus our development efforts on
the areas we think will make the most difference. During development we
will also use gprof to profile the code and optimize the functions we
spend the most time executing.

### \* How do you handle security?

We keep all of our servers on a private subnet, and use the load
balancer to provide connectivity to the ports we care about (DNS, HTTP,
SMTP, POP, IMAP and the SSL alternatives). We use OpenVPN to get access
to other ports (SSH, MySQL, NFS, SNMP from outside the facility).

We have an Intrusion Detection System in place along with some custom
scripts to make sure things stay secure (the details of which are
secret). We also use a least privilege model (chroot and suid) to
minimize the amount of damage someone could do if they did compromise a
service.

### \* How do you handle customer support?

Currently the engineering team takes turns handling support requests. We
feel this keeps the engineers aware of what could be improved. Support
requests are primarily received via our contact form and email.

### \* How do you decide what features to add/keep?

We listen to our users and try to focus on what they ask for. We also
use the service ourselves, so we have a good feel for what improvements
would make the biggest impact. Our list of desired features has grown
very long...

### \* Do you implement web analytics?

Yes, we use Awstats for simple log file analytics.

### \* Do you do A/B testing?

Not specifically, but we have written a number of small programs that
test specific functionality, and use them to verify builds. We’ve also
written a number of unit tests that are executed when the server starts
to make sure the environment is sane and everything is working
correctly.

### \* How many data centers do you run in?

So far we’ve kept the system inside one 42U rack. At least in the short
term, we plan to grow the system through hardware upgrades (if the need
arises).

### \* How do you handle fail over and load balancing?

We use a hardware load balancer (Alteon AD4). The load balancer presents
a single virtual IP to the internet and then splits incoming connection
requests among the different application servers. The load balancer
modifies the IP packets in realtime to maintain the illusion. The load
balancer also monitors the different application server nodes and
automatically removes any that fail.

### \* Which DNS service do you use?

Good old fashioned BIND for the domains we care about. Requests are
split among the different DNS servers using the load balancer. We let
our registrar handle DNS for the domains that we don’t consider
critical.

### \* Which routers do you use?

Our colo provider has Cisco 6509 routers.

### \* Which switches do you use?

Linksys gigabit switches. Since all of our app servers have two gigabit
interfaces we take advantage of this with a physical/logical network for
public traffic, and a separate physical/logical network for database,
NFS, and memcached traffic.

Technically the Alteon AD4 is considered a Layer 7 switch.

### \* Which email system do you use?

We use our own custom platform, although Postfix is still used to relay
outbound mail.

### \* How do you handle spam?

We rely on 3rd party RBL’s, along with several of our own additions.
Paid users also have access to a statistical filter based on DSPAM.

### \* How do you handle virus checking of email and uploads?

ClamAV is used to scan inbound and outbound messages. In 2004 we looked
at using the Sophos library, but it was too expensive for us to license.
Since we don’t want to involve another process, our options remain
limited to products that have library APIs.

### \* How do you backup and restore your system?

All critical servers use RAID 5. The MySQL database is replicated to a
slave server, which is then used to create offline backups.

As for the mail data, we feel it changes too frequently to make an
offline backup solution viable (not to mention the security implications
of keeping messages after a user deletes them). For a handful of our
corporate users, mail data is stored on at least two NFS servers, and
then deleted by the client app from both servers when the time comes.

### \* How are software and hardware upgrades rolled out?

Individual nodes are taken offline, the software upgraded, and then
brought back online. Typically we test new versions on one of the nodes
for a week or more before rolling it out across the entire cluster. As a
result we spend a lot of time making sure we have version N and version
N – 1 compatibility.

### \* How do you handle major changes in database schemas on upgrades?

We test, we pray, and we monitor.

We’ve only done two non-compatible rollouts since coming online in 2004.
The first was when we switched to our custom platform in February/March
of 2005 (we had to fail back to the old system twice before our daemon
was stable enough to use full time). The second upgrade was in October
of 2007 (I think) when we changed the file format used to store messages
and had to convert all of the existing data over. This cutover went
smoothly, because we minimized the code that was changed between the two
versions to just that used for decrypting and compressing messages.
During the upgrade, unconverted messages were hidden from users via the
database.

We may need to make one more leap sometime in 2009/2010 as we have
several major database and file system changes planned that won’t be
backwards compatible. Stay tuned...

### \* What is your fault tolerance and business continuity plan?

We keep offsite backups of application code, configuration information,
and key database tables.

### \* Do you have a separate operations team managing your website?

Yes. Only two people have access to the production equipment and
databases.

### \* Do you use a content delivery network? If so, which one and what for?

The mail system data changes far too frequently for a CDN to make sense.

In contrast the consulting side of our company has deployed three
websites that used Akamai or Internap to deliver video. We recommend
their use if your hosting static content that is particularly sensitive
to latency (audio and video primarily) or you have a web application
that must load quickly all over the globe.

### \* How much do you pay monthly for your setup?

We feel our current pricing for space, power and bandwidth with
Colo4Dallas is a very competitive 4 figure amount; but we’ll keep the
exact amount to ourselves for now.

## Miscellaneous

### \* Who do you admire?

Google, Amazon, Live Journal and Facebook on a technical level. I feel
these companies succeeded because their founders built better websites
than the competition and then coupled their initial success with good
technical and business decisions as their sites grew. The jury is still
out on Google on an ethical level.

I also admire Fog Creek software for how it (supposedly) treats both
customers and employees.

In the same vein, I still wonder how MySpace survived after making so
many poor technical decisions early on. Although I will admit they have
improved tremendously since their early days.

### \* Have you patterned your company/approach on someone else?

Our corporate philosophy is a combination of many different approaches.
The biggest influences have probably been the book [Founders at Work by
Jessica Livingston](http://www.amazon.com/gp/product/1590597141) and the
Joel Spolsky books [Joel on
Software](http://www.amazon.com/gp/product/1590593898) and [The Best of
Software Writing](http://www.amazon.com/gp/product/1590595009).

### \* Are there any questions you would add/remove/change in this list?

You might want to consider creating a short form of this questionnaire.
It took awhile to write all of this. :)
