The Storage Explosion

I am a big believer in the “scarcity and abundance” theory of IT development. The theory basically postulates that if you want to understand the near term future of information technology development the most important thing to consider is the scarcity and/or abundance of the “big 3” foundations of computing power: processing power, storage and bandwidth. Understanding the absolute and relative levels of these three technologies is the closest thing possible to having an IT industry crystal ball as they have a huge influence over everything from system architectures to capital investment plans to end user demand.

A 15 Year Rocket Ship Ride
It is with this in mind that I found a recent article on Tom’s Hardware to be fascinating. The article details the increases in storage capacity and performance over the last 15 years. Some of the numbers involved are mind boggling.  In the last 15 years, the storage capacity of top-end hard drives has increased 5,907X from 130MB in 1991 to 750GB in 2006. To put this in perspective, during the same time period here are some increases in the other core technologies of a “bleeding edge” PC:

                                       

End User Technology	1991	2006	X Increase
LAN Bandwidth	10 mbit/s (10BaseT)	1 gbit/s (1000BaseT)	102X
WAN Bandwidth	14.4 kbit/s (v.32bis)	3 mbit/s (Cable)	213X
CPU Performance	54 MIPS (486DX)	27079 MIPS (X6800)	501X
Hard Disk	130MB	750GB	5907X

The price difference is even more dramatic. In 1991 a megabyte of storage cost about $7.00, now it costs $0.000527. That’s a 13,274X price improvement or a 99.9925% price drop in 15 years. Not too shabby.

It’s interesting to note though that disk I/O performance improvement has lagged dramatically with only a 121X improvement from 0.7 MB/s to 85 MB/s. This clearly makes disk I/O one of the biggest, if not the biggest bottleneck in a modern PC. Put another way, it would theoretically take 3.1 minutes to write the entire contents of a 130MB  disk drive in 1991 while it takes about 2.5 hours to write the contents of a 750GB drive in 2006 (as anyone who has tried to back-up a monster like this well knows).

Forecast Calls For: More, Cheaper Storage
The near term prospects for additional gains in storage capacity remain promising. For traditional magnetic storage hard drives, the introduction of perpendicular recording should continue to drive platter densities higher, but much like CPUs, the limits of physics are starting to catch up with magnetic hard drives which suggests that they will not be able to continue their capacity gains indefinitely. Not to worry though, two new storage technologies are finally coming to market this year including Flash memory hard drives and holographic storage. Flash memory hard drives use NAND chips instead of magnetic platters to store data. They have a number of important advantages over traditional magnetic media including dramatically faster file access times and the ability to process far more I/O operations per second.  For example, SanDisk just announced today a 32GB flash drive that accesses files in 0.12 milliseconds vs. 8 milliseconds for the best magnetic drive, a 67X improvement (although its read rate is only 64 MB/s).  Flash drives also draw 50-70% less power than magnetic drives, weight about 50% less, do not produce noise or vibrations, give off very little heat, and are less fragile than magnetic drives. But all these improvements do not come cheaply as the initial 32GB flash drives will cost more than $20/GB or about 35-40X more than a magnetic drive.  Still you should expect to these as an option on high end ultra-portable laptops in a few months.  I also expect that many PC enthusiasts may buy a flash drive for their "C:" drive to dramatically improve Windows Vista boot times.

2007 will also see the first commercially available holographic storage drives, although these drives are likely to only compete with archival media, such as tape drives and Blu-Ray media, for the foreseeable future. Holographic areal storage densities have doubled in just the past year and at 515Gb/inch are already more than 2X those of the most advanced magnetic drives.  To put this in perspective, Sony’s Blu-Ray disks which have just started shipping after many delays have a 50GB capacity while InPhase’s initial holographic disk (which had it’s 1^st shipments just a few days ago) have a 300GB capacity and should go to 500GB in the next year or so. Put another way, you could store 106 DVDs on a single (slightly larger) holographic disk (although you can only write to these disks at about 23 MB/s, so be prepared to spend over 6 hours putting you DVD collection on a single holographic disk). Because it uses light rather than magnetic heads to access data, holographic storage should theoretically delivery better access times and faster read throughput than magnetic storage mediums. Holographic storage is expected to seriously challenge tape backup systems in the short term, but could also displace consumer storage mediums such as DVD and Blu-Ray as prices fall. Holographic disk drives are theoretically possible, but significant technical challenges still have to be overcome especially in terms of being able to rewrite the media and improve write speeds.

What Happens In A World Awash With Storage?
You may be asking yourself, this is all very interesting but why do you, someone who generally is focused on software and the internet, care so much about storage? I care because storage capacity is clearly the most abundant and faster growing component of the “big three”(processing power, bandwidth, and storage) today and it looks as though the relative disparity between the three may even increase further in the short term.  This situation should have some very interesting implications for both software and internet related businesses, some of which I will outline in my next post on the topic.