Cost of Storing All the Email in the World

Here’s an interesting theory from a Slashdot comment.

Google doesn’t even need the limit. (Score:5, Interesting)
by image (13487) on Wednesday May 19, @09:21AM (#9194576)
(http://www.breakdownindustries.com/)

A few people have made the comment that Google can do this because 99% of the people will only use a few MBs of storage anyway. Reasonable theory, but here’s another idea — it doesn’t matter if everyone uses a massive amount of storage.

First, figure out how many people there are in the world that might potentially use Gmail. Then figure out what is the potential maximum amount of unique data each of those people could generate on a daily basis. Then determine the size of the redundant information that could pass through the Gmail servers.

Note that a huge percentage of emails and attachments are sent to multiple recipients. For each piece of email or attachment compute and store a unique hash. Each account consists of only a list of hashes and some header metadata. This redundant information will significantly reduce the total storage space.

A quick seach finds this Berkeley study [berkeley.edu] that suggests that there were about 400 PB of email (unique) generated last year. Assuming that you can save 1 GB of data for the fully-loaded cost of $1 (US) [pricewatch.com], storing all of the internet’s annual email traffic costs $500M annually in the worst case.

The best case is significantly better than that, as you can:

a) compress text by up to 80%
b) store every mail only once
c) store every large binary only once
d) add storage as needed, not up-front
e) reduce the cost of storage over time [littletechshoppe.com]

This is off-the-cuff, but Google is looking at maybe a $50M annual investment in storage to store all the email on the internet, even if everyone uses it. They don’t even need a storage limit. Period.