From owner-chemistry;at;ccl.net Tue Mar 1 23:33:01 2022 From: "Abhik Seal abhik1368*o*gmail.com" To: CCL Subject: CCL: Cloud backup in computational chemistry Message-Id: <-54616-220301180642-18620-wZCSe3q/bcSLCb0MMsUDxQ(~)server.ccl.net> X-Original-From: Abhik Seal Content-Type: multipart/alternative; boundary="0000000000007b8c0e05d9303767" Date: Tue, 1 Mar 2022 17:06:24 -0600 MIME-Version: 1.0 Sent to CCL by: Abhik Seal [abhik1368*gmail.com] --0000000000007b8c0e05d9303767 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable One way Would be to reduce the file size such that you don=E2=80=99t need h= igh amount of space and also reduces cost . Python blosc is a high performance compressor . Blosc works well for compressing numerical arrays that contains data with relatively low entropy, like sparse data, time series, grids with regular-spaced values, etc. Another one is feather format in Apache arrow . Hope you get some ideas . Abhij On Tue, Mar 1, 2022 at 3:47 PM Detlev Conrad Mielczarek detlevcm#,# googlemail.com wrote: > > Sent to CCL by: Detlev Conrad Mielczarek [detlevcm]![googlemail.com] > Please note, RAID is NOT a backup. > > The topic is fairly complex if done rigorously. Look up "bit rot" - data > life is finite too. > Look up the risk of data corruption on a RAID system - or loss with say > malware or error. > > The topic is not simple - unfortunately. > And then there is data lifetime on disks which is not necessarily a > resolved question. > Tape is considered the best for long term storage at current - maybe > archive quality DVDs. > > Detlev > > > > Sent from my BlackBerry > > > > Original Message > > > > From: owner-chemistry[*]ccl.net > Sent: 1 March 2022 21:19 > To: detlevcm[*]googlemail.com > Reply to: chemistry[*]ccl.net > Subject: CCL: Cloud backup in computational chemistry > > > Rather than cloud, may I suggest a 2 TB Raid system? They can be had for > less than $200, and RAID 1 would provide adequate long-term stability. > > > Dan Strahs > > > > > > From: owner-chemistry+dstrahs=3D=3Dpace.edu##ccl.net > > on behalf of Rassolov, Vitaly RASSOLOV~!~ > mailbox.sc.edu > Sent: Tuesday, March 1, 2022 9:35 AM > To: Strahs, Dr. Daniel Bernard > Subject: CCL: Cloud backup in computational chemistry > > Open Science Framework https://www.cos.io stores data at $4000/TB, > guaranteed for 50 years. I don't have any personal experience with it, a= nd > it seems pricey for an individual. If your institution is affiliated wit= h > the OSF, perhaps the price is different. > > Center for Open Science > At the Center for Open Science, our mission is to increase openness, > integrity, and reproducibility of scholarly research. Promoting these > practices within the research funding and publishing communities > accelerates scientific progress > www.cos.io > > > > > Vitaly Rassolov > Department of Chemistry and Biochemistry > University of South Carolina > 631 Sumter St, Columbia SC 29208 > > https://sc.edu/study/colleges_schools/chemistry_and_biochemistry/our_peop= le/directory/rassolov_vitaly.php > > > From: owner-chemistry+rassolov=3D=3Dmail.chem.sc.edu=3Dccl.net > on behalf of A= ndrew > DeYoung andrewdaviddeyoung#%#gmail.com > Sent: Monday, February 28, 2022 4:48 PM > To: Rassolov, Vitaly > Subject: CCL: Cloud backup in computational chemistry > > Hi, > > > I have ~1.5 TB of analysis data on my home computer related to my PhD > work, which I completed last year. I would like to put this data in > long-term storage in the cloud. I will retain a backup on an external ha= rd > drive, which I will keep at home. But I'm looking for an additional back= up > in another location (the other location being, ideally, the cloud). Thes= e > files are not themselves MD trajectories from my PhD, but rather are outp= ut > and analyses related to the trajectories. (The trajectories themselves a= re > tens of TB and are being retained, for now, by my university.) > > > Does anyone have experience with archiving scientific data on cloud > storage services such as AWS S3, AWS S3 Glacier, Microsoft Azure, or > iDrive? > > > I realize this is the Computational Chemistry List, not the Information > Technology List, but I'd be grateful for any insight you may have from th= e > perspective of a computational chemist. > > > I will be paying for this myself, so the service needs to be reasonably > priced for a single user for ~5-10 TB or less. It's not immediately clea= r > to me whether all of these services are really available to individuals. > > > I don't need fancy features like automatic backup, delta copying, and so > forth; I just need a place to safely store some files. I use Linux for > running computation, but I'm not an IT expert by any means; I am willing = to > learn, though, especially when good documentation is available! And, > clearly, transferring ~1.5 TB to the cloud is itself a nontrivial task, b= ut > I'm willing to see what the options are. > > > Here are some links to some seemingly popular services in my country (US)= , > but perhaps there are others more particularly suited for scientific data= : > > > AWS Amazon S3: https://aws.amazon.com/s3/ > AWS Amazon S3 Glacier: https://aws.amazon.com/s3/storage-classes/glacier/ > Microsoft Azure: https://azure.microsoft.com/en-us/services/#storage > > iDrive: https://www.idrive.com/ > > > Thank you for your time! > > > Best, > Andrew > > Andrew DeYoung, PhD > Department of Chemistry > Carnegie Mellon University > > > > -=3D This is automatically added to each message by the mailing script = =3D-> > > -- Cheers, Abhik Seal Ph.D. (Cheminformatics) --0000000000007b8c0e05d9303767 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
One way Would be to reduce the file size such that you do= n=E2=80=99t need high amount of space and also reduces cost . Python blosc = is a high performance compressor .=C2=A0Blosc works well for compressing numeric= al arrays that contains data with relatively low entropy, like sparse data,= time series, grids with regular-spaced values, etc.

Another one is feather format = in Apache arrow . Hope you get some ideas .

Abhij



On Tue, Mar 1, 2022 at 3:47 PM Detlev Conrad Mielczarek detlevcm#,#googlemail.com <owner-chemistry^ccl.net> wrote:

Sent to CCL by: Detlev Conrad Mielczarek [detlevcm]![googlemail.com]
Please note, RAID is NOT a backup.

The topic is fairly complex if done rigorously. Look up "bit rot"= - data life is finite too.
Look up the risk of data corruption on a RAID system - or loss with say mal= ware or error.

The topic is not simple - unfortunately.
And then there is data lifetime on disks which is not necessarily a resolve= d question.
Tape is considered the best for long term storage at current - maybe archiv= e quality DVDs.

Detlev



Sent from my BlackBerry



=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Original Message =C2=A0=C2=A0 =C2=A0

> From: owner-chemistry[*]ccl.net
Sent: 1 March 2022 21:19
To: detlevcm[*]googlemail.com
Reply to: chemistry[*]ccl.net
Subject: CCL: Cloud backup in computational chemistry


Rather than cloud, may I suggest a 2 TB Raid system? They can be had for le= ss than $200, and RAID 1 would provide adequate long-term stability.


Dan Strahs




> From: owner-chemistry+dstrahs=3D=3Dpace.edu##ccl.net <owner-ch= emistry+dstrahs=3D=3Dpace.edu##ccl.net> on behalf of Rassolov, Vita= ly RASSOLOV~!~mailbox.sc.edu<owner-chemistry##ccl.net>
Sent: Tuesday, March 1, 2022 9:35 AM
To: Strahs, Dr. Daniel Bernard
Subject: CCL: Cloud backup in computational chemistry
=C2=A0
Open Science Framework https://www.cos.io stores data at $4000/TB, guaranteed = for 50 years.=C2=A0 I don't have any personal experience with it, and i= t seems pricey for an individual.=C2=A0 If your institution is affiliated w= ith the OSF, perhaps the price is different.

Center for Open Science
At the Center for Open Science, our mission is to increase openness, integr= ity, and reproducibility of scholarly research. Promoting these practices w= ithin the research funding and publishing communities accelerates scientifi= c progress
www.cos.= io




Vitaly Rassolov
Department of Chemistry and Biochemistry
University of South Carolina
631 Sumter St, Columbia SC 29208
https://sc.edu/study/colleges_schools/chemistry_and_biochemistry/our_p= eople/directory/rassolov_vitaly.php

> From: owner-chemistry+rassolov=3D=3Dmail.chem.sc.edu=3Dccl.net <owner-ch= emistry+rassolov=3D=3Dmail.chem.sc.edu=3Dccl.net> on behalf of Andrew DeYou= ng andrewdaviddeyoung#%#gmail.com<owner-chemistry=3Dccl.net>
Sent: Monday, February 28, 2022 4:48 PM
To: Rassolov, Vitaly <RASSOLOV=3Dmailbox.sc.edu>
Subject: CCL: Cloud backup in computational chemistry
=C2=A0
Hi,=C2=A0


I have ~1.5 TB of analysis data on my home=C2=A0computer related to my PhD = work, which I completed last year.=C2=A0 I would like to put this data in l= ong-term storage in the cloud.=C2=A0 I will retain a backup on an external = hard drive, which I will keep at home.=C2=A0 But I'm looking for an add= itional backup in another location (the other location being, ideally, the = cloud).=C2=A0 These files are not themselves MD trajectories from my PhD, b= ut rather are output and analyses related to the trajectories.=C2=A0 (The t= rajectories themselves are tens of TB and are being retained, for now, by m= y university.)


Does anyone have experience with archiving scientific=C2=A0data on cloud st= orage services such as AWS S3, AWS S3 Glacier, Microsoft Azure, or iDrive?= =C2=A0=C2=A0


I realize this is the Computational Chemistry List, not the Information Tec= hnology List, but I'd be grateful for any insight you=C2=A0may have fro= m the perspective of a computational chemist.


I will be paying for this myself, so the service needs to be reasonably pri= ced for a single user for=C2=A0~5-10 TB or less.=C2=A0 It's not immedia= tely clear to me=C2=A0whether=C2=A0all of=C2=A0these services are really av= ailable to individuals.


I don't need fancy features like automatic backup, delta copying, and s= o forth; I just need a place to safely store some files.=C2=A0 I use Linux = for running computation, but I'm not an IT=C2=A0expert by any means; I = am willing to learn, though, especially when good documentation is availabl= e!=C2=A0 And, clearly, transferring ~1.5 TB to the cloud is itself a nontri= vial=C2=A0task, but I'm willing to see what the options are.


Here are some links to some seemingly popular services in my country (US), = but perhaps there are others more particularly suited for scientific data:<= br>

AWS Amazon S3:=C2=A0https://aws.amazon.com/s3/
AWS Amazon S3 Glacier:=C2=A0https://aws.amazon.com= /s3/storage-classes/glacier/
Microsoft Azure:=C2=A0https://azure.microsoft.com= /en-us/services/#storage

iDrive:=C2=A0https://www.idrive.com/


Thank you for your time!


Best,
Andrew

Andrew DeYoung, PhD
Department of Chemistry
Carnegie Mellon University



-=3D This is automatically added to each message by the mailing script =3D-=
E-mail to subscribers: CHEMISTRY^ccl.net or use:
=C2=A0 =C2=A0 =C2=A0 http://www.ccl.net/cgi-bin/ccl/s= end_ccl_message

E-mail to administrators: CHEMISTRY-REQUEST^ccl.net or use
=C2=A0 =C2=A0 =C2=A0 http://www.ccl.net/cgi-bin/ccl/s= end_ccl_message
=C2=A0 =C2=A0 =C2=A0 http://www.ccl.net/chemistry/sub_un= sub.shtml

Before posting, check wait time at: http://www.ccl.net

Job: http://www.ccl.net/jobs
Conferences: http://server.ccl.net/chemist= ry/announcements/conferences/

Search Messages: http://www.ccl.net/chemistry/sear= chccl/index.shtml
=C2=A0 =C2=A0 =C2=A0 http://www.ccl.net/spammers.txt

RTFI: http://www.ccl.net/chemistry/aboutccl/ins= tructions/


--
<= div dir=3D"ltr">
=C2=A0
Cheers,
Abhik Seal =C2=A0Ph.D. (Cheminformatics)

--0000000000007b8c0e05d9303767--