CCL: Cloud backup in computational chemistry

One way Would be to reduce the file size such that you don’t need
 high
 amount of space and also reduces cost . Python blosc is a high performance
 compressor . Blosc works well for compressing numerical arrays that
 contains data with relatively low entropy, like sparse data, time series,
 grids with regular-spaced values, etc.
 Another one is feather format in Apache arrow . Hope you get some ideas .
 Abhij
 On Tue, Mar 1, 2022 at 3:47 PM Detlev Conrad Mielczarek detlevcm#,#
 googlemail.com <owner-chemistry^ccl.net> wrote:
 >
 > Sent to CCL by: Detlev Conrad Mielczarek [detlevcm]![googlemail.com]
 > Please note, RAID is NOT a backup.
 >
 > The topic is fairly complex if done rigorously. Look up "bit rot"
 - data
 > life is finite too.
 > Look up the risk of data corruption on a RAID system - or loss with say
 > malware or error.
 >
 > The topic is not simple - unfortunately.
 > And then there is data lifetime on disks which is not necessarily a
 > resolved question.
 > Tape is considered the best for long term storage at current - maybe
 > archive quality DVDs.
 >
 > Detlev
 >
 >
 >
 > Sent from my BlackBerry
 >
 >
 >
 >           Original Message
 >
 >
 > > From: owner-chemistry[*]ccl.net
 > Sent: 1 March 2022 21:19
 > To: detlevcm[*]googlemail.com
 > Reply to: chemistry[*]ccl.net
 > Subject: CCL: Cloud backup in computational chemistry
 >
 >
 > Rather than cloud, may I suggest a 2 TB Raid system? They can be had for
 > less than $200, and RAID 1 would provide adequate long-term stability.
 >
 >
 > Dan Strahs
 >
 >
 >
 >
 > > From: owner-chemistry+dstrahs==pace.edu##ccl.net
 > <http://pace.edu#%23ccl.net>;
 <owner-chemistry+dstrahs==pace.edu##ccl.net
 > <http://pace.edu#%23ccl.net>>; on behalf of Rassolov,
 Vitaly RASSOLOV~!~
 > mailbox.sc.edu<owner-chemistry##ccl.net>
 > Sent: Tuesday, March 1, 2022 9:35 AM
 > To: Strahs, Dr. Daniel Bernard
 > Subject: CCL: Cloud backup in computational chemistry
 >
 > Open Science Framework https://www.cos.io stores data at $4000/TB,
 > guaranteed for 50 years.  I don't have any personal experience with it, and
 > it seems pricey for an individual.  If your institution is affiliated with
 > the OSF, perhaps the price is different.
 >
 > Center for Open Science
 > At the Center for Open Science, our mission is to increase openness,
 > integrity, and reproducibility of scholarly research. Promoting these
 > practices within the research funding and publishing communities
 > accelerates scientific progress
 > www.cos.io
 >
 >
 >
 >
 > Vitaly Rassolov
 > Department of Chemistry and Biochemistry
 > University of South Carolina
 > 631 Sumter St, Columbia SC 29208
 >
 > https://sc.edu/study/colleges_schools/chemistry_and_biochemistry/our_people/directory/rassolov_vitaly.php
 >
 > > From: owner-chemistry+rassolov==mail.chem.sc.edu=ccl.net
 > <owner-chemistry+rassolov==mail.chem.sc.edu=ccl.net> on behalf of
 Andrew
 > DeYoung andrewdaviddeyoung#%#gmail.com<owner-chemistry=ccl.net>
 > Sent: Monday, February 28, 2022 4:48 PM
 > To: Rassolov, Vitaly <RASSOLOV=mailbox.sc.edu>
 > Subject: CCL: Cloud backup in computational chemistry
 >
 > Hi,
 >
 >
 > I have ~1.5 TB of analysis data on my home computer related to my PhD
 > work, which I completed last year.  I would like to put this data in
 > long-term storage in the cloud.  I will retain a backup on an external hard
 > drive, which I will keep at home.  But I'm looking for an additional backup
 > in another location (the other location being, ideally, the cloud).  These
 > files are not themselves MD trajectories from my PhD, but rather are output
 > and analyses related to the trajectories.  (The trajectories themselves are
 > tens of TB and are being retained, for now, by my university.)
 >
 >
 > Does anyone have experience with archiving scientific data on cloud
 > storage services such as AWS S3, AWS S3 Glacier, Microsoft Azure, or
 > iDrive?
 >
 >
 > I realize this is the Computational Chemistry List, not the Information
 > Technology List, but I'd be grateful for any insight you may have from the
 > perspective of a computational chemist.
 >
 >
 > I will be paying for this myself, so the service needs to be reasonably
 > priced for a single user for ~5-10 TB or less.  It's not immediately clear
 > to me whether all of these services are really available to individuals.
 >
 >
 > I don't need fancy features like automatic backup, delta copying, and so
 > forth; I just need a place to safely store some files.  I use Linux for
 > running computation, but I'm not an IT expert by any means; I am willing to
 > learn, though, especially when good documentation is available!  And,
 > clearly, transferring ~1.5 TB to the cloud is itself a nontrivial task, but
 > I'm willing to see what the options are.
 >
 >
 > Here are some links to some seemingly popular services in my country (US),
 > but perhaps there are others more particularly suited for scientific data:
 >
 >
 > AWS Amazon S3: https://aws.amazon.com/s3/
 > AWS Amazon S3 Glacier: https://aws.amazon.com/s3/storage-classes/glacier/
 > Microsoft Azure: https://azure.microsoft.com/en-us/services/#storage
 >
 > iDrive: https://www.idrive.com/
 >
 >
 > Thank you for your time!
 >
 >
 > Best,
 > Andrew
 >
 > Andrew DeYoung, PhD
 > Department of Chemistry
 > Carnegie Mellon University
 >
 >
 >
 > -= This is automatically added to each message by the mailing script =->
 >
 > --
 Cheers,
 Abhik Seal  Ph.D. (Cheminformatics)