Originally posted by sonhouse
NASA could use that level of storage, they are getting gigabytes per day of data, they must be single handedly supporting the HD industry!

You've got that wrong somewhere.

lets say by "gigabytes per day" you meant 10 Gigabytes / day.
Thats 3,650 Gigabytes per year or just under 4 terabyte hard disks. (or just 2 of 2 terabyte disks)
Thats hardly 'single handedly supporting the HD industry'. Even I have a terabyte hard disk.

Even if you meant 'terabytes per day' that would still end up being less than 1000 drives.

Compare this to Google who 2 years ago was using 850TB for search, 220TB for analytics and 70TB for Google Earth.
http://googlesystem.blogspot.com/2006/09/how-much-data-does-google-store.html

I bet Youtube has even more than that.

Originally posted by twhitehead
Yes I can see that the new laser is the way to go. Finally I will be able to back up my whole hard disk. But then again, by the time I actually have one of these new lasers, my hard disk will be 100 terabytes.

Aye. If the optical media R&D guys can just catch up a little - ie. produce a consumer product that can back up a whole home network using less than 5 disks, a lot more money would stream in for yet more R&D.

Originally posted by twhitehead
You've got that wrong somewhere.

lets say by "gigabytes per day" you meant 10 Gigabytes / day.
Thats 3,650 Gigabytes per year or just under 4 terabyte hard disks. (or just 2 of 2 terabyte disks)
Thats hardly 'single handedly supporting the HD industry'. Even I have a terabyte hard disk.

Even if you meant 'terabytes per day' that would still end up ...[text shortened]... om/2006/09/how-much-data-does-google-store.html

I bet Youtube has even more than that.

Here is a link to the latest stuff at NASA:

http://www.nasa.gov/centers/ames/news/releases/2010/10-33AR.html

There is a supercomputer there with a 1.4 petabyte storage unit. That is just one computer. I saw a report that in 2003 the storage requirements just for space missions was 40 terabits in that year, 7 years ago. I would bet it would be ten times that now. Plus all the data storage for non-space missions. I couldn't find any numbers closer than this on google.

Here is another reference to storage at that facility:

Data Storage
Our HEC users often require staggering
amounts of data storage. With 25 petabytes
(PB) of tertiary storage and 2 PB of disk
storage on the floor, NAS’ mass storage
system allows users to archive and retrieve
important results quickly, reliably, and
securely. We also provide customized
training and support to help users efficiently
manage large amounts of data. For
example, scientists working on ECCO2

I might point out, that is just one facility.

I found an information email line and I asked that question of NASA, maybe I'll get a reply sometime in the next year🙂

Originally posted by sonhouse
I saw a report that in 2003 the storage requirements just for space missions was 40 terabits in that year, 7 years ago. I would bet it would be ten times that now.

1 terabyte = 8 terabits
40 terabits= 5 terabytes
5 terabytes X 10 = 50 terabytes
Allow for redundancy through raid, add 1 for every 5 disks = 60 terabytes.
Allow 30% free disk space = 80 terabyts

Total requirement = 80 hard disks.

Still not quite 'single handedly supporting the HD industry'.

Originally posted by sonhouse
There is a supercomputer there with a 1.4 petabyte storage unit.

Thats better, but still no more than 2000 drives could serve that up. A company with 2000 employees has that many drives (admittedly not all terabyte drives but they will be in a year or so.)

Originally posted by twhitehead
Thats better, but still no more than 2000 drives could serve that up. A company with 2000 employees has that many drives (admittedly not all terabyte drives but they will be in a year or so.)

Here is the answer to the email I sent, surprised I got back an answer this soon!

Dear Don Jennings,



Thank you for contacting the NASA STI Help Desk. This is a very interesting question! The NASA Office of the Chief Information Officer (NASA OCIO) would be the most likely source for this information as they are the organization responsible for consolidating all of NASA’s data centers. For more information please see: http://i3p-acq.ksc.nasa.gov/i3p/default.cfm. Contact information is available at: http://i3p-acq.ksc.nasa.gov/Contracts/Contract6.cfm



The NASA employee directory is also available online at: https://people.nasa.gov.



I am unsure whether this information has been aggregated and would be publicly available.



Thank you for your interest in NASA scientific and technical information. Please do not hesitate to contact us should you need further assistance.



Sincerely,





Kim Lyall

Outreach Specialist

NASA STI Help Desk

(P) 443-757-5802

(F) 443-757-5803

help@sti.nasa.gov



Follow CASI on Twitter

Originally posted by twhitehead
Thats better, but still no more than 2000 drives could serve that up. A company with 2000 employees has that many drives (admittedly not all terabyte drives but they will be in a year or so.)

Well here is the most data density I ever heard about: The LHC in Cern, the gigantic collider, will generate 15,000 Terabytes per year!:

http://esciencenews.com/articles/2008/11/03/what.do.with.15.million.gigabytes.data

That has to make every other data center on Earth look like a commodore 64🙂

Originally posted by sonhouse
Well here is the most data density I ever heard about: The LHC in Cern, the gigantic collider, will generate 15,000 Terabytes per year!:

http://esciencenews.com/articles/2008/11/03/what.do.with.15.million.gigabytes.data

That has to make every other data center on Earth look like a commodore 64🙂

I bet Google stores more than that in live databases, whereas that will probably be archived as it gets processed and never be all on hard disks at one time.

Originally posted by twhitehead
I bet Google stores more than that in live databases, whereas that will probably be archived as it gets processed and never be all on hard disks at one time.

See what you can dig up!

Originally posted by sonhouse
See what you can dig up!

I have not found a good source but I did come across:
Yahoo: 2 Petabyte database.
Ebay: 2Petabyte database and a 6.5 Petabyte database.
Walmart: 2.5 Petabytes
Bank of America: 1.5 Petabytes
Dell: 1.5 Petabytes

I believe Amazon, Sprint and a few others have large databases.

I would be interested in Googles:
1. Street view, Google earth / maps.
2. Search and analytics.
3. Youtube.
4. Gmail

There is also a video sites in China that claim to have more data than Youtube, but I couldn't find it.

Also some of the major sites that deal with shareware downloads.

My refs for the above figures (dated a year or two ago, please note! )
http://www.dbms2.com/2009/04/30/ebays-two-enormous-data-warehouses/
http://www.computerworld.com/s/article/9087918/Size_matters_Yahoo_claims_2_petabyte_database_is_world_s_biggest_busiest
http://www.dbms2.com/2008/10/15/teradatas-petabyte-power-players/

Originally posted by sonhouse
That has to make every other data center on Earth look like a commodore 64🙂

I must also point out that what really makes a data centre special is not the shear drives space (thats easy, 1 petabyte of disk space could be found on any respectable hard disk dealers shelves) but activity and throughput.
Data going into and out of companies like Google would definitely be many times greater than what LHC has to handle.

Firefox 3 claims to have been downloaded 8 million times in 24 hours. Thats 7.7MB X 8 million = 61 TB in 24 hours for one tiny program.
What do you think windows update does? (think about how ie has a larger market share and is a much larger download then add on the rest of windows ! )

And we haven't even started on Bittorrent.

There is a claim that nearly a million people downloaded the last two episodes of Lost in less than a day.

http://torrentfreak.com/bittorrent-download-record-shuttered-by-lost-series-finale-100525/

Originally posted by twhitehead
I must also point out that what really makes a data centre special is not the shear drives space (thats easy, 1 petabyte of disk space could be found on any respectable hard disk dealers shelves) but activity and throughput.
Data going into and out of companies like Google would definitely be many times greater than what LHC has to handle.

Firefox 3 c ...[text shortened]... .

http://torrentfreak.com/bittorrent-download-record-shuttered-by-lost-series-finale-100525/

I don't see how a download system can count for data storage, that's apples and taco's🙂

Sure SOMEONE gets that 61 TB, but distributed over 8 million comps. In the case of NASA, all that data gets piled up in one place. As well as whatever Google needs. That's a different thing than bittorrent doing TB/day or whatever. That would be the internet bandwidth load not total storage.

Originally posted by sonhouse
I don't see how a download system can count for data storage, that's apples and taco's🙂

I know, but my point was that it is the data transfer or manipulation that makes a datacenter special, not the total storage space. Some of the figures I quoted earlier were for single databases ie not only do they have that much data, but you can query the whole lot at once. I don't believe NASA puts all that data (quoted per space mission) in one database, but rather that is the total data collected in many different forms in many different databases.
Similarly the LHC may simply put all the data onto tape and then process it linearly. They may be able to do all their work with a 1 Gb hard disk. Of course with the prices of hard disks these days, they might well stick it all onto hard disk.

But apart from the fact that the data is all related, the datacentre they use will be far from unique. Most large datacentres of web hosting companies have hundreds of servers and could probably store that data with ease.