Hi,
I’m not sure if this is the right community for my question, but as my daily driver is Linux, it feels somewhat relevant.
I have a lot of data on my backup drives, and recently added 50GB to my already 300GB of storage (I can already hear the comments about how low/high/boring that is). It’s mostly family pictures, videos, and documents since 2004, much of which has already been compressed using self-made bash scripts (so it’s Linux-related ^^).
I have a lot of data that I don’t need regular access to and won’t be changing anymore. I’m looking for a way to archive it securely, separate from my backup but still safe.
My initial thought was to burn it onto DVDs, but that’s quite outdated and DVDs don’t hold much data. Blu-ray discs can store more, but I’m unsure about their longevity. Is there a better option? I’m looking for something immutable, safe, easy to use, and that will stand the test of time.
I read about data crystals, but they seem to be still in the research phase and not available for consumers. What about using old hard drives? Don’t they need to be powered on every few months/years to maintain the magnetic charges?
What do you think? How do you archive data that won’t change and doesn’t need to be very accessible?
Cheers
This is my day job, so I’d like to weigh in.
First of all, there’s a whole community of GLAM institutions involved in what is called Digital Preservation (try googling that specifically). Here in Germany, a lot of them have founded the Nestor Group (www.langzeitarchivierung.de) to further the case and share knowledge. Recently, Nestor had a discussion group on Personal Digital Archiving, addressing just your use case. They have set up a website at https://meindigitalesarchiv.de/ with the results. Nestor publishes mostly in German, but online translators are a thing, so I think you will be fine.
Some things that I want to address from your original post:
- Keep in mind that file formats, just like hardware and software, become obsolete over time. Think about a migration strategy for your files to a more recent format of your current format falls out of style and isn’t as widely supported anymore. I assume your photos are JPGs, which are widely not considered safe for preservation, as they decay with subsequent encoding runs and use lossy compression. A suitable replacement might be PNG, though I wouldn’t go ahead and convert my JPGs right away. For born digital photo material, uncompressed TIFF is the preferred format.
- Compression in general is considered a risk, because a damaged bit will potentially impact a larger block of compressed data. Saving a few bytes on your storage isn’t worth listing your precious memories.
- Storage media have different retention times. It’s true that magnetic tape storage has the best chances for survival, and it’s what we use for long term cold storage, but it’s prohibitively expensive for home use. Also, it’s VERY slow on random access, because tape has to be rewound to the specific location of your file before reading. If you insist on using it, format your tapes using LTFS to eliminate the need for a storage management system like IBM Spectrum Protect. The next best choice of storage media are NAS grade HDDs, which will last you upwards of five years. Using redundancy and a self correcting file system like ZFS (compression & dedup OFF!) will increase your chances of survival. Keep you hands off optical storage media; they tend to decay after a year already according top studies on the subject. Flash storage isn’t much greater either, avoid thumb drives at all cost. Quality SSD storage might last you a little longer. If you use ZFS or a comparable file system that provides snapshots, you can use that to implement immutability.
- Kudos for using Linux standard tooling; it will help other people understand your stack of anything happens to you. Digital Preservation is all about removing dependencies on specific formats, technologies and (importantly) people.
- Backup is not Digital Preservation, though I will admit that these two tend get mixed into one another in personal contexts. Backups save the state of a system at a specific point in time, DigiPres tries to preserve only data that isn’t specific to a system and tends to change very little. Also, and that is important, DigiPres tries to save context along with the actual payload, so you might want to at least save some metadata along with your photos and store them all in a structure that is made for preservation. I recommend BagIt; there’s a lot of existing tooling for creating it, it’s self-contained, secured by strong checksums and it’s an RFC.
- Keep complexity as low as possible!
- Last of all, good on you for doing SOMETHING. You don’t have to be perfect to improve your posture, and you’re on the right track, asking the right questions. Keep on going, you’re doing great.
Come back at me if you have any further questions.
Wow, that was a rabbit hole of information, links and ideas. Thanks a lot. I reached a point of what I would call “satisfaction” https://cdn.nationalarchives.gov.uk/documents/selecting-storage-media.pdf was back linked by Nestor and it seems to give me an idea of what I’ll do next. Thanks again 👍
Good to hear! When you go with the National Archives UK, you can’t fail. They have some very, VERY competent people in staff over there, who are also quite active in the DigiPres community. They are also the inventors of DROID and the maintainers of the widely used PRONOM database of file formats. https://www.nationalarchives.gov.uk/PRONOM/Default.aspx Absolute heroes of Digital Preservation.
And have multiple copies in at least two locations of anything truly important to guard against disaster (such as a fire or regionally appropriate natural disaster). I got a spare drive to copy all the music that I’ve made and sent it to my father in a different part of the country. I could lose everything and be pretty bummed, but not that (without severe depression). I also endorse use of a safe deposit box at a bank if you don’t have someone who can hold data in a different city.
Yeah, you can always go crazy with (off site) copies. There’s a DigiPres software system literally called LOCKSS (Lots Of Copies Keep Stuff Safe).
The German Federal Office for Information Security recommends a distance of at least 200km between (professional) sites that keep georedundant copies of the same data/service, so depending on your upload capacity and your familiarity with encryption (ALWAYS backup your keys!), some cloud storage provider might even be a viable option to create a second site.
Spare drives do absolutely work as well, but remember that, depending on the distance, data there will get more or less outdated and you might not remember to refresh the hardware in a timely manner.
A safe deposit box is something that I hadn’t considered for my personal preservation needs yet, but sounds like a good idea as well.
Whatever you use, also remember to read back data from all copies regularly and recalculate checksums for fixity checks to make sure your data doesn’t get corrupted over time. Physical objects (like books) decay slowly over time, digital objects break more spontaneously and often catastrophically.
This is actually a real problem… A lot of digital documents from the 90’s and early 2000’s are lost forever. Hard drives die over time, and nobody out there has come up with a good way to permanently archive all that stuff.
I am a crazy person, so I have RAID, Ceph, and JBOD in various and sundry forms. Still, drives die.
nobody out there has come up with a good way to permanently archive all that stuff
Personally I can’t wait for these glass hard drives being researched to come at the consumer or even corporate level. Yes they’re only writable one time and read only after that, but I absolutely love the concept of being able to write my entire Plex server to a glass harddrive, plug it in and never have to sorry about it again.
This is interesting, haven’t heard of it. I think the problem with the disc format is you aren’t getting 28 TB of content on there unless you span multiple discs which is a pain in the ass
unless you span multiple boxes of discs which is a pain in the ass
FTFY
That was my thought as well. I’m also thinking it might be a challenge to get something to read them in 100 years.
It’s crazy that there isn’t a company out there making viable cold storage for the average consumer. I feel like we’re getting even further away from viability now that we use QLC by default in SSDs. The rot will be so fast.
What about magnetic tape? Isn’t it like super durable?
yeah, but for OP’s amount that’s an overkill, the drives are very expensive
Wherever you choose to store it, you should still consider following the 3-2-1 backup rule.
Don’t use DVDs or other disc media
Ugh, sounds icky. Thanks for the advice:)
Assume anything you can buy has a shelf life and set a yearly reminder on your calendar to copy forward stuff more than five or so years old, if those files are of significant value to you. Or for the documents, print them out—paper has better longevity than any consumer-available electronic storage.
That being said, quality optical discs are probably the best option in terms of price to longevity ratio for the average person right now. Just keep in mind that they are not guaranteed to last forever and do need to be recopied from time to time.
(I have yet to have a DVD fail on me, but I keep them in hard plastic jewel cases in climate-controlled conditions, and I’ve probably just been lucky.)
I use LTO magnetic tape for archiving data, but unfortunately the tape drives are VERY expensive. The tape itself is relatively cheap though (this is a 5-pack at 12TB uncompressed, 30TB compressed per cardridge, totaling at 60TB uncompressed, 150TB compressed. This is a lot cheaper than hard drives, and lasts for much longer), has large storage capacity and 30+ years of shelf life. Yes, I know, LTO 9 has come out, but I won’t be upgrading, because LTO 8 works just fine for me, and is much cheaper. The drives are backwards compatible by one generation though, e.g. you can use LTO 8 tape in an LTO 9 drive.
5 k€? No wonder no one uses tape for home usage. You can come up with a lot of cheaper alternatives for that price.
I got a used unit for much cheaper from the place I previously worked at. If I had to spend 5K on one, I probably wouldn’t use it either.
Might be a dumb idea but hear me out. How about sealing a reputable enterprise or consumer SSD in one of those anti static bags with a desiccant and then sealing that inside a pvc pipe also with desiccant and then burying it below the frost line? You’ll just have to dig it up and refresh everything every couple of years, think 3 years at most iirc for consumer ones. Obviously this isn’t a replacement for a backup solution just archival so no interaction with it. It’ll protect it from the elements, house fires, flooding, temperature fluctuations pretty much everything and its cost effective. Hell you can even surround the hard drive bag in foam then stuff in the pvc pipe for added shock absorption. Make a map afterwards like a damn pirate (its night time so my bad if I sound deranged)
edit I took a nap: in hindsight I should’ve clarified. I went with an ssd in this idea since its more durable than a mechanical, better price for storage capacity compared to m-disc, and most likely to be compatible with other computers in the future in case you need it for whatever reason. Of course you can use another storage media, like m disc, just know of the drawbacks. Like needing a m-disc burner (~100$), several discs depending on how big of a capacity you need (price varies), pray that there’s still a drive that can read m-disc in the future and know that’s its gonna be slow when getting your data back regardless. All you would have to do to modify the idea would be getting a disc case that kinda suspends the disc so nothing is touching it’s surfaces. Then the same idea: antistatic bag with desiccant, foam or even bubble wrap around it, stuffed in a pipe with desiccant buried below your frost line. People usually skip the “in optimal conditions” part when talking about m-disc but this way we get close to those optimal conditions
went with an ssd in this idea since its more durable than a mechanical, better price for storage capacity
how? sorry but that does not add up to me. for the price of a 2 TB SSD you could by a much larger HDD
and most likely to be compatible with other computers in the future in case you need it for whatever reason.
both of these use SATA plugs, it should be the same
oh my bad i was referring to the storage capacity difference between ssds and m-disc. I’ll edit it in
oh, I see now, sorry! from mechanical I instantly thought you mean an HDD
This is a very, very bad idea.
SSDs are permanent flash storage, yes, but that doesn’t mean you can leave them unpowered for extended periods of time.
Without a refresh, electrons can and do leak out of the charge traps that store the ones and zeroes. Depending on the exact NAND used, the data could start going corrupt within a year or so.
HDDs suffer the same problem, though less so. They can go several years, possibly a decade, but you’d still be risking the data on the drive but letting it sit unpowered for an extended time.
For the “cold storage” approach you should really be using something that’s designed to retain data in such conditions, like optical media, or tape drives.
Yeah that’s why I said it needs to be refreshed and also edited in an option for m-disc in case they want to go the optical route
Of course you can use another storage media, like m disc,
You’ll just have to dig it up and refresh everything every couple of years, think 3 years at most iirc for consumer ones
You might be interested in git-annex (see the Bob use case).
It has file tracking so you can - for example - “ask” a repository at drive A where some file is, and git-annex can tell you it’s on drives C and D.
git-annex can also enforce rules like: “always have at least 3 copies of file X, and any drive will do”; “have one copy of every file at the drives in my house, and have another at the drives in my parents’ house”; or “if a file is really big, don’t store it on certain drives”.
Don’t use DVDs. They suffer bitrot, as do “metal” hard drives.
The local-plus-remote strategy is fine for any real-world scenario. Make sure that at least one of the replicas is a one-way backup (i.e., no possibility of mirroring a deletion). That way you can increment it with zero risk.
And now for some philosophy. Your files are important, sure, but ask yourself how many times you have actually looked at them in the last year or decade. There’s a good chance it’s zero. Everything in the world will disappear and be forgotten, including your files and indeed you. If the worst happens and you lose it all, you will likely get over it just fine and move on. Personally, this rather obvious realization has helped me to stress less about backup strategy.
So you would suggest to get bigger and bigger storages?
I really like and can embrace the philosophical part. I do delete rigorously data. At the same time, i once had a data lost, because I was young and stupid and tried to install Suse without an backup. I still am sad to not to be able to look at the images of me and my family from this time. I do look at those pictures/videos/recordings from time to time. It gives me a nice feeling of nostalgia. Also grounds me and shows me how much have changed.
Fair enough!
So you would suggest to get bigger and bigger storages?
Personally I would suggest never recording video. We did fine without it for aeons and photos are plenty good enough. If you can still to this rule you will never have a single problem of bandwidth or storage ever again. Of course I understand that this is an outrageous and unthinkable idea for many people these days, but that is my suggestion.
Never recording videos… That is outrageous ;) Interesting train of thought, though. Video is the main data hog on my drives. It’s easy to mess up the compression. At the same time is combines audio, image and time in one easy to consume file. Personally, i would miss it.
3-2-1 rule with restic. Check it out.
Checked it out, thanks. I have to figure out, how it compares to my rsync Script
Waaaaay better.
Restic allows you to make dedupe snapshots of your data. Everything is there and it’s damn hard to loose anything. I use backblaze b2 as my long term end point / offsite… some will use AWS glacier. But you don’t have to use any cloud services. You can just have a restic repository on some external drives. That’s what I use for my second copy of things. I also will do an annual backup to a hard disk that I leave with a friend for a second offsite copy.
I’ve been backing up all of my stuff like this for years now. I used to use BORG which is another great tool. But restic is more flexible with allowing multiple systems to use a single repository and has native support for things like B2 that BORG doesn’t.
We also use restic to backup control nodes for some of supercomputing clusters I manage. It’s that rock solid imho.
Someone else has mentioned M-Disc and I want to second that. The benefit of using a storage format like this is that the actual storage media is designed to last a long time, and it is separate from the drive mechanism. This is a very important feature - the data is safe from mechanical, electrical and electronic failure because the storage is independent of the drive. If your drive dies, you can replace it with no risk to the data. Every serious form of archival data storage is the same - the storage media is separate from the reading device.
An M-Disc drive is required to write data, but any DVD or BD drive can read the data. It should be possible to acquire a replacement DVD drive to recover the data from secondary markets (eBay) for a very long time if necessary, even after they’re no longer manufactured.
M-disk, never heard of that. I got a quick research done and it seems to be exactly what I was looking for. Thank you!
Don’t over complicate it. 3 copies: backup, main, and offsite; 2 different media: hdd and data center; 1 offsite. I like blackblaze but anything from google to Amazon will work.
Good advice. My off-site is my brother’s place.
I am using https://duplicati.com/ and https://www.backblaze.com/ ( use their b2 cloud storage its variable and 6$ a month for 1TB or less depending on how much you use) run a schedule beckup every night for my photos. It’s compressed and encrypted. I save a config file to my google so say if my house and server burn down. I just pull my config from google then redownload duplicati and boom pull my back up down. The whole set up backs up incremental so once you do the first back up its only changes that are uploaded. I love the whole set up.
Edit: You can also just pull files you need not the whole backup.
Use a raid atrray, and replace drives as they fail. Ideally they wouldnt fail behind your back, like an optical disk would.
That is an always ON approach? For example with an NAS? While that is a very save approach, it does not fit the idea of having something “on the shelf”. Thank you for the advice though :)
You could turn it off and turn it back on every X period of time, but that doesn’t guarantee something doesn’t go wrong in between. It sounds like you don’t have alot of data relatively speaking. Is there a reason not to keep it on your present machine and do the above? Cost? IIRC you can get a 1 tb m.2 for under $150.
