*Edit: I have figured out how to use BTRFS and enable what it calls “transparent file compression”, and I’m going to use that on most of my old storage devices. The only problem I’m having is that I want to use F2FS on my oldest storage device, as BTRFS takes up too much space on the device and I was told by multiple users that F2FS also supports transparent file compression, but I can’t get files to compress and I’m not getting any error messages to try and fix it. Based on what the documentation says, I’m supposed to do something like this:
sudo mkfs.f2fs -f -O extra_attr,inode_checksum,sb_checksum,compression /dev/mmcblk0p1
sudo mount -o compress_algorithm=zstd,compress_extension=* /dev/mmcblk0p1 '/home/j/mountpoint/128mb'
chattr -R +c '/home/j/mountpoint/128mb'
The device will mount like this but files aren’t compressing when added, nor are they compressed if using the last command after they’ve been moved.*
I’m rewriting the old portion for clarification:
In Windows, there’s a file/folder option called “Compress contents to save disk space”. What it does is it compresses the files, as the name suggests, but leaves them accessible as though they aren’t. This doesn’t really have much of a benefit on newer storage devices but on older storage devices, in addition to saving space, it allows files to potentially read faster.
As I have some old storage devices that I want to run games from, I think this will be a great option to have if I could find something similar for Linux. I tried looking online myself but search engines are terrible and I couldn’t find anything though them. So, I decided to post about this here, to see if anyone knows of anything I could try.
F2FS seems to do what you want, it’ll reserve the original size of your file but compress what’s actually written. Performance numbers might be massively inflated if your writes don’t saturate the cache in RAM.
I’ve used BTRFS on SD cards before and it’s mostly fine, but it will struggle massively if over 90% full, or if you have a < 1 GB volume and are, say, frequently updating a handful of files that together take up more than half its capacity. Mostly due to the CoW mechanism, it needs some headroom to make a copy of whatever files are being modified.
I’ve done some testing with f2fs and it does seem like what I want to go with but I can’t get compression working. Someone else linked me to this page but for some reason the command on that page gives an error message saying “Error: Failed to get the device stat!”. I already asked another user but assuming I can get it working, how would I change the command they provide to enable zstd compression. From what I’ve read, zstd is the compression method I want to be using.
I figured out what I did wrong but now I’m getting a different error message. It’s saying “Error: Wrong features compress_algorithm=zstd” even though the page I was given says it should be possible. Any ideas on what I should do to fix that?
This is at file system level… Checkout btrfs and zfs, I am quite positive both can compress like you want.
Never used this feature myself, so cannot be more specific.
Also, there are some read-only compressed filesystems for Linux that you can also use, they offer best compression but data is read only.
Okay, I’m having a bit of a problem. I tried the smallest device first but it’s too small for btrfs and I can’t figure out how to format devices in zfs. Unless xfs is the same as zfs, the option isn’t available in gparted and mkfs gives an error saying that the zfs file doesn’t exist. If it’s possible to install zfs through apt, “apt search zfs” gives a lot of results for zfs.
How small is that device??? Didn’t even knew btrfs has a minimum size.
After for zfs, you probably have to install some software or kernel patches
It’s 128 MB, brtfs needs at least 256 MB. Also yes, I’m aware that I’d need to install something to use zfs but I don’t know where to look to find out what I’d need to install.
Damn, where did you even find such a small drive? The last time I used such a small device was the USB stick I brought to highschool in 2009. Even the free giveaway USB drives I have are at least 2GB. You probably have more RAM than storage?
If it’s a flash memory (sd card, usb stick, ssd, etc), you could try f2fs, it’s very light, and it supports compression and is meant specifically for that kind of devices (well, more for ssds).
But judging your experience from your comments, I suggest you don’t delve into niche filesystems until you have more experience with Linux, especially for something like 128MB. I especially suggest you avoid zfs for now.
It is an sd card and I did just format it to f2fs but how do I enable the compression? It does seem much faster that ext4 but it seems to also take up more space on the sd card, as I’ve lost about 25% of the free space after formatting it, so I’d like to enable the compression.
As for zfs, I have been using Linux for over a few years at this point but I’ve mostly been avoiding some of the more complicated stuff. So if it is more complicated than what I originally thought, then I’ll avoid that for now as well.
So, first of all, there is no gui for this, that I’m aware of, so you will have to do it from terminal. Second, on f2fs, compression works that you don’t enable compression for a folder, instead you mount the drive with compression enabled, and new files will be compressed automatically.
So what you need is to set up your disk to be mounted with compression. There are many paths you can follow here. If you want your drive to be (almost) permanently connected, the easiest way is to use “/etc/fstab”. If you want to use it as a regular SD card, mounting and ejecting it from your file explorer etcetera, then you should go here and learn how to have udisks2 mount your device with compression, which should be what your desktop environment uses to mount drives. I suggest you set that up for your specific device, and not for all f2fs devices. Good luck.
You can look up other useful f2fs options on the arch wiki. I suggest you add all those options that reduces writes to your disk and improve durability (like lazytime).
You should use zstd as compression algorithm, and because this is a slow and small drive, you can crank up the level of compression.
If you manage to pull this off, the next time you install a (bigger and faster) drive on your pc, you can try to look into zfs.
I read the page you linked to but I have no idea what I’m supposed to do with it. If it’s talking about using the “mount” command, I’ve already used that and even though I specifically enabled zstd as the compression method, files aren’t being compressed. I know this because I’m using a file that’s slightly larger than the free space but it’s still giving errors about not having enough space.
Huh. My computer allows me to format a 128MB image file with brtfs. It won’t do it at 64MB though.
How are you trying to format the device? I got that error message when using gparted.
truncate -s 128M fs.img parted fs.img mklabel gpt mkpart primary btrfs 1MiB 100% quit sudo losetup --find --partscan --show fs.img sudo mkfs.btrfs /dev/loop0p1You should be able to skip the loop device stuff and work on an actual device instead. Seems to me the limit is somewhere between 64M and 128M.
Edit: But as edinbruh said, maybe try f2fs if it’s a flash device, that’s probably a bit more lightweight?! And since I don’t know what you’re doing… If it’s embedded stuff and you’re alright with read-only, you might want to use squashfs.
I got brtfs working on a different storage device but I can’t tell if I set it up properly or not. I’m using the documentation that another user posted but it’s a bit confusing to me and I need to test it more.
For my 128 MB sd card, I am trying f2fs because that’s working. It does seem to write much faster than it did with ext4 but it seems to take up more space and it now has about 25% less free space, and it doesn’t seem to compress files. I’ll have to do more testing before I determine whether this what I go with or not.
As for squashfs, I probably don’t want that.
Xfs and zfs are two different filesystems.
Just download more RAM.
https://btrfs.readthedocs.io/en/latest/Compression.html
You know this takes 5 minutes with web search to find out? Or ask Le Chat.
I can tell that you didn’t read my post before commenting. But regardless, I’ll have to try that later as the device I’m testing first is too small for btrfs and I’m currently trying zfs first before I try one of my other storage devices.
Your skill of knowing what people did is as good as your skill of doing research I see.
the device I’m testing first is too small for btrfs
Yes, I can read what OP wrote in response to my message. This information wasn’t provided in the post. What is your point?
that you write things i’m trying to understand the relevance of, like “This information wasn’t provided in the post.” and “the first comment points to btrfs”. The Reddit link you gave also points towards btrfs as well as very undetailed mentions of zfs. ze says “i’ve tried btrfs and it doesn’t work so i’m looking into zfs”, and you reply “use btrfs use btrfs or look into zfs”, a message whose helpfulness I struggle to understand.
The relevance is that it’s easy to find answer to OP’s question on the Internet. Despite OP’s claims in the post or his remarks in his answer. And as it turns out, the device wasn’t too small far btrfs I guess.
well what you said after zir reply didn’t demonstrate any of that
Not sure that’s relevant, but I’m playing around with Linux on some obsolete Windows tablets with cheap eMMC disks (one of which is broken, so I replaced it with a no name microSD card, plus USB drive), and I format my disks to f2fs, which theoretically should help with both keeping the disks for longer, and accessing the data faster.
I’ll have to test that out more later but I did format the sd card to f2fs and it did seem to write files very fast when I did a simple test. The only issue I’m seeing is that it has about 25% less space than it did when I had it formatted for ext4, is that normal for f2fs?
Make sure you’re actually filling the volume, and also keep in mind reporting may be different (with or without filesystem index metadata, etc)
I’m just going by what Linux Mint’s file manager says and it’s saying that the used space is about 40 MB with f2fs while it was about less than half of that when it was formatted to ext4.
What’s a RAM drive? I have 8 GB of RAM and I have ZRAM enabled, so I should have enough RAM for most of the files I’d consider using it for.
RAM drive is when you hold the entire file system in RAM, it’s used for stuff like Linux Live CD boot with no writable storage but you can use it for anything.
Is it read-only, or does it write the files at some point?
Both are possible but read only is more common
FYI digging through documentation reminded me of squashfs. It does what it sounds like. But it’s read only, so you would have to use an overlay FS to cache changes and then overwrite the whole squashfs volume at once to sync changes.
There’s an interesting project called DwarFS. I have it on my todolist to check it out but as I understand it you basically create a compressed read-only archive that is mountable and readable just like any other disk - https://github.com/mhx/dwarfs . Maybe this is something for your use case?
Maybe for some things but because I’m using it for games, the “read-only” might limit what games I can use this with. I might still try it out though.
Unfortunately I haven’t tested it so I cannot fully advise on that but I have seen some “repacks” on the high seas where a whole wine prefix with a game already installed was compressed to a single
dwarwhich you can mount and play with configs and saves in your ~.Funnily enough it’s often used in the pirate scene to distribute games. But only as archives.
The equivalent would be either zfs or btrfs compression. Transparent to applications, you don’t have to do anything special other than enable it.
As I mention in another comment, I’m having a bit of a problem. I tried the smallest device first but it’s too small for btrfs and I can’t figure out how to format devices in zfs. Unless xfs is the same as zfs, the option isn’t available in gparted and mkfs gives an error saying that the zfs file doesn’t exist. If it’s possible to install zfs through apt, “apt search zfs” gives a lot of results for zfs.
How small is your smallest device? BTRFS doesn’t have a minimum size, but practically probably 50-100mb is just about doable before even just setting things up get complex. Having said that though, it’s copy-on-write and has overhead as a result, so may not function well below 1gb.
ZFS meanwhile really won’t work well below probably 8gb. It’s also copy-on-write but with a lot more overhead due to how it works. It really works best on big drives and filesystems.
If your old storage is in the mb range, then really neither will help you achieve what you want.
BTRFS and ZFS do offer the same benefits as NTFS with regard to compression and speeding up some slower devices (due to lowering the actual read/writes needed to achieve the same result as the data is compressed into a smaller space and decompressed rapidly by the PC in memory), but NTFS can go be used on much smaller disk sizes due to how it works. BTRFS and ZFS are designed and optimised with other benefits in mind. And NTFS compression isn’t well supported in Linux.
How small are these devices? I think the other problem is that neither BTRFS nor zfs really are suitable for removable devices, and definitely not for ones smaller than probably 8Gb at the very least.
Unlike NTFS which is just a file system, both BTRFS and ZFS do volume management too, so it’s not just a single partition thing; they prefer to take over an entire volume and manage everything.
So while they’re the closest filesystem with NTFS-like transparent compression……they don’t match exactly.
I also hazard to guess if the devices you’re using are too small to accept a BTRFS formatted volume, no amount of compression is going to be enough to fit what you need.
If you just want to play with a bunch of small old devices……maybe play with LVM and small RAID arrays and configurations instead. You can the build a bigger volume out of a bunch of those disks together and then put a BTRFS or zfs volume on them. Can be fun to experiment and learn with anyway.
PM me if you need help with ZFS. I’ve gotten quite good at setting it up.
For ZFS check the instruction for your distribution here: https://openzfs.github.io/openzfs-docs/Getting Started/index.html
I apparently already have it installed but mkfs still thinks that zfs isn’t available. Am I supposed to use something else?
You’ll need to read the documentation to understand its concepts first. You don’t use
mkfsbutzpoolto create a pool then usezfsto create the fs on it. ZFS is usually used with some sort of raid configuration so most doc will show this but you can create a pool with no replication using a single drive.
I just want to say that if you are going to either resort to belittling me
That’s like a red rag to a bull
you haven’t fully read the post before commenting
You could’ve just tldr’d it. Everyone skims nowadays
I’m not going to respond to your comment
Well that was a big fat lie, wasn’t it.
https://wiki.gentoo.org/wiki/SquashFS
This is frequently used for USB drive purposes.
I would use that but I’d prefer something that isn’t read-only. Since I’m using it to run games, I’m going to need write access for a lot of them.
I can try that but I can’t find any instructions on how to format a device for squashfs. Can you link me to something I can use?
The Gentoo link I provided had instructions. Use the stack exchange for the read-write.
Good luck!
sudo nano /etc/fstab
Then replace “defaults” with “compress=zstd” on your desired partitions.
IMO stick with btrfs. Also, the storage space between file systems is exactly the same, given the same hardware.
I’m only seeing the boot drive and the swapfile even though the SD card is mounted and working. Do I have to manually add a new entry somewhere in this file?
Also, no it is different. If I format the SD card to BTRFS it only has 40MB of free space, while it has about 80MB of free space when formatted to F2FS. I’d be getting double the free space if I can get compression working in F2FS.
The fstab file is used to define how disk partitions or remote file systems are mounted into your computer. Removable drives such as USB drives or SD cards are not shown there, because if they were, your system would complain at boot that it can’t find the requested USB drive.
About F2FS having double the storage space of btrfs. In all honesty, it doesn’t make sense to me. Do you mean it shows you potentially being able to store 80 megabytes if you can get compression working or does it just show 80 megabytes instead of 40, after formatting the file system?
What do you mean, btrfs “takes too much space”? That’s not a sentence that belongs to file system stuff in any way.
The disk is the space. You make it usable with a fs, for which you can partition portions of it.And also, i’ve had bad experiences with f2fs long-term stability.
When I format the SD card to BTRFS, it only has about 40MB of free space available. But when I format the SD card to F2FS, it has about 80MB of free space available.
https://btrfs.readthedocs.io/en/latest/mkfs.btrfs.html
Have you tried
--metadata singleand--mixedas well as mounting with-o nospace_cache?I tried that but it wont mount if I add nospace_cache. It’ll mount with:
sudo mount -o compress=zstd:15 /dev/mmcblk0p1 '/home/j/mountpoint/128mb'But if I add nospace_cache to this command, it’ll give me an error saying “mount: bad usage”.
You may have to clear it first.
Check dmesg output for usually more verbose mount errors.
Also I’d use compress-force and test out the other compression algorithms with your data set.
Even though the “nospace_cache” option isn’t working, the rest of what you mentioned worked fine and there’s about 120MB of free space now, which is most of the device. I’m not sure if nospace_cache is worth trying to get since BTRFS only uses about 5MB of the device now. Also, with the files I was testing, I was able to get almost 250MB onto the device, which is better than what I think I would have gotten with F2FS.
Anyways, thanks for helping me.
Mind you that “free space” figures with compression, with btrfs and with such tiny file systems are tricky at best. You’d have to test all options by actually copying your data set first.
What are you saving on that drive? Many data file formats already have compression of their own and don’t benefit much from file system compression. So if this is for media files, for example, it’s likely to add CPU overhead without a big benefit in transfer speed.
ZFS is not installed by default with most Linux distributions due to its license. It’s something you install after the os. Btrfs should work, but I see some discussion online of 128 or 256MB minimum volume size.
I’m using it for games, I know it’s not really recommended to do that but I’m doing it anyways. I’m trying btrfs on one of my devices but I may need zfs for one of the others, however, I don’t know how to install it.
“*Edit: I have figured out how to use BTRFS and enable what it calls “transparent file compression”, and I’m going to use that on most of my old storage devices.”
If the old data is meant to be archived then maybe do not mess with it that way and if you want it compressed use 7zip and other similar programs that are known and well tested. Also they should be stored on HDDs instead of SSDs, longer shelf life. If you have issues on your personal PC that is and lacking context, this question could be more advanced for NAS or servers which generally people do not know much about.
My “I don’t need to save space, I want it for a different purpose” tee shirt is raising a lot of questions answered by my “I don’t need to save space, I want it for a different purpose” tee shirt.
The Compress attribute has been in even ext3 since day 1. I’ve never tried it, though.
If you are talking about what I think you’re talking about, that doesn’t do what I need it to. That just compresses the file(s) like a normal compression tool. I need something that will allow me to continue accessing the files as if they’re not compressed while keeping them compressed.
