Storage Expenses Are Presumed To Be: Fill & Download for Free

Wine is not an ingredient per se, but since it is not normal to order only wine at a restaurant, I will treat is as an ingredient of the meal.Patrons who plan to splurge on a great meal, will overlook the absurd pricing of the available wines.300% markup is not uncommon, and the cheaper the wholesale price, is the higher the margin added. A $10 wine at wholesale, can go as high as $30 in the restaurant, while a $50 bottle will probably see less than double that, maybe going for around $85.When you order the cheapest bottle available, your restaurateur will smile all the way to the cash register. Wine has no preparation, other than storage and serving. So compared to even the most expensive dish that needs an expensive chef to prepare, the wine is a money maker.

There are fairly inexpensive ways to store 100 TB of data. The things that increase the expense:What will you do with the data once it’s stored?Is it purely “legal archive” data that will be put in a vault somewhere and only accessed upon receiving a legal demand for the data? (This is fairly cheap and easy)Or will it be highly active data that resides on a website accessed by thousands or more per day, queried, displayed, downloaded, etc? (This is way harder, and 100TB of “payload” data may well need another 100TB - or more - of search structure data to make it easy to query)Or will it be somewhere between this, such as research datasets that researchers need to be able to access and organize, but maybe not needing heavy indexing or organizing?Also, what is the 100TB of data measuring? Is it just the raw or “payload” data, or is it data + search structures? As I mentioned a bit above, 100TB of “raw” data may need another 100TB or more storage space to hold search structures so users can make sense of the data. Even if the 100TB of data is in files, the file-system metadata associated with numerous files will take up nontrivial space.Another question is will you want this data redundantly backed up? Presumably you don’t want to lose any of this data, so you may need multiple “copies” of the data - and the search structures - so you don’t lose this data if one of your disks/whatever goes bad.Nowadays, spinning disks are cheap enough that you could build a rig that could hold 100 TB - without backup - for under $10,000 US. If you want it to be fairly redundant, push it to about $20K. A quick check online shows you can buy 10TB hard drives for about $400 each, so if you get 20 of them for 200TB, you’ve got some redundancy.Storing this much data in a cloud may get expensive quickly, even for “cheap” cloud storage like Amazon Glacier, which isn’t immediately accessible. But you can go and do the math there and see if it works for you. The current price for Glacier is $0.004/month/gigabyte, so storing 100 terabytes at the list price would be $0.004 * 100 * 1024 = $400/month, although access is “archival” and definitely not live. And you’d need metadata in glacier that you can get at somewhat easily so you could make sense of your glacier data. S3 is more accessible, but more expensive at $0.23 per GB or $2350/month for 100TB.(Updated: I was confusing S3 with Glacier)This is where “build versus buy” gets to be an interesting question, as not only do you have to factor in stuff like how much the hardware costs - you also have to have staff (or personal knowledge) in how to deal with it, and hosting and power costs are nontrivial for something with a bunch of (power hungry) spinning disks.As for the cloud, it’s a lot easier - and you don’t need as much system administration knowledge to deal with it - but may end up being expensive if you have lots of people frequently accessing the data, as this means more expensive storage.

Cloud service providers offer many different cloud pricing points depending on your compute, storage, database, analytics, application and deployment requirements. Prices can be subject to the type of hosting option you choose. Although not always the case, AWS is presumed to be the least expensive option available and remains the leader in the cloud computing market. But, Microsoft Azure and Google (GCP) are not far behind. Here’s a pricing breakdown of the most popular storage options offered by each of the three major cloud providers - if you want a more detailed description of the storage options you can find that on this blog post.