Once upon a time.....
A close friend of mine came into my dorm room inquiring about my pre-order for a 50 Megabyte hard drive expansion for my Amiga 500+ he'd just seen on the special orders sheet at RAF Alconbury's Base Exchange.
The best thing about being in the military, and single, and no girl friend, and no bills of any kind, is you can spend stupid amounts of money on real stupid shit!
Ahhh....those where the days!
When the sales lady at the BX handed me my brand new 50Mb hard drive months later, one thought raced through my mind!
"What am I gonna fill it with!?"
Fast forward 25-ish years...
Amazon Prime recently delivered another batch of Toshiba P300 3TB HDD's for my server, and the same question popped into my mind.
But lets back up to June of 2020. A very good friend of mine whom will remain nameless (hey Steve, how's it going! 😘) invited me to a London data center for the day. He handed me a brand new storage model prototype with 10 Seagate Exos 16TB drives in it. 160 Terabytes. It was heavy!
To put that into context:
In 1991 a 50 Mb drive gave me 50,000,000 bytes of storage
In 2020 a 16Tb drive gives you 16,000,000,000,000 bytes of storage
Back in the 90's, file sizes where generally measured in KILO bytes. Today, your average web page has images in the MEGA bytes (1000 times larger).
When I'm playing with virtual machines, most running Windows 10, the VM (initial start) file size is around 90 to 100 GIGABYTES.
In 2002, when I worked for Charrington's Solid Fuels, we had just started getting into online shopping. At a meeting with a potential partner I told him I was expecting our image storage to exceed 1TB of storage by 2005 for all the products we planned on displaying. So here's 2020 and my wife has 2TB of holiday photo's! Just vacations! Not including family gatherings, holiday gatherings, etc.
In fact my RAID 5 (redundant) drives on my home server for pictures is now at 10TB an climbing!
But back to 1992, and my buddy telling me I was insane for spending $500 on something I would never fill.
Well I did. Took me 4 years (1996), but I did. Funny enough it was photo's that did it!
Now back to 2020.....
I now have a home server with 36 TB of storage and my main concern is not "if" they might get filled, but when. AND.....is that going to happen within the drives lifespan!
Modern drives have a very finite lifespan. 3 to 5 years, depending upon how its used. For me that is in two ways.
Very heavy usage.
or
Not used at all.
Let me explain.
My favourite thing is going on driving holidays with wifey! We see some amazing shit and always enjoy ourselves. When we come back there is usually 100 to 200 MEGABYTES of images to upload. Granted most of them are delete-able, out of focus or not staged right, etc. But we keep them all. They get dumped onto the server. People might see them, but generally not. They are there for us, and we look at them rarely.
Then you have my job. I'm a programmer and I love my job. I write software for the hell of it. I have Windows Services running that interrogate my Elite:Dangerous local profile, uploading data to various online trackers, and retrieving data I generally ignore. Because I can. I can write web services to do that. Nobody will see it, but I know it's there, running, silently, because I will it to happen. All of this takes processing power. Most of the time it's a virtual machine doing the processing. My virtual machines average 150 to 200 GIGABYTES (dev configured) in size.
And I make them all the time. Delete them when I don't need it. Copy another, several times, to try something I'm not sure will work. If it fails, delete it, copy it again, start over.
Due to this extreme range of usage, my server drives get the full range of life cycle. 3 to 5 years. Some last more, maybe 6, but none have passed this mark. Some implode earlier, at 2 years. Two weeks ago (Aug 2020) one of the RAID drives for my images died requiring replacement after 4 years of use. Amazon had a new drive to me in 24 hours, swapped it out and pressed on. No drama. No data loss.
That brings me to the purpose of this article.
What in the actual fuck am I going to do with 36 TERABYTES of storage?
My home server has 12 Toshiba 3TB P300 drives in it. I settled on this drive due to cost, ease of replacement, and the ID sticker is on the head of the drive. Easy to see when I remove the front panel of the server. When a drive fails (or starts generating errors) Windows tells me the serial number of the drive that's failing. Reading this number easily is important!
Currently I am at 60% capacity. This goes up by 3 to 4% each year.
At some point, I'll have to change the size to 6 or even 10TB for each drive. But not yet.
When I was a kid, I remember my Mom pulling out a huge box filled to the top of photos spanning 30 years. I remember being amazed at the black and white images. Super cool. Well, we are now fully digital in my house. My box is now measured in Terabytes. I can only imagine my kids, at my age, might be using Petabytes.
While a hard copy photo, well cared for, might last centuries. A hard drive does not. 3 to 5 years skippy! That's all you get!
Most people would use an online storage like Facebook, Microsoft Onedrive, Google Drive, etc to store this. My images are mine, no-one else's. I'll manage storage myself thank you!
My home server notifies me the instant a drive error is detected. I then log in and decide is it a "real error" or an anomaly. For example, the wife transferred several hundred from one folder to another and our power went out. The transfer didn't complete. Even though the server has a UPS (Uninterruptible Power Supply) unit, her machine does not. So an error occurred, the transfer had to be restarted. Those errors I ignore. But in August 2020, drive 11 started generating sector faults very badly. Within two days you could even hear the read head "clicking" as it tried to access the drive. Time to replace it, which I did.
This is a task I accept willingly to have the benefit of a large storage server.
Most people would never take this on, opting for public storage. I am not a public person!
So my 36TB will have to do.
For now.
What will I fill them with!?