It is a wonderful project, I got a little rush of nostalgia just from the mention of GW-BASIC.

Make sure you research what the “right way” is to approach some of the difficult problems you will run into; you will learn more that way instead of needing to struggle with figuring out how to pick out where the end of the current loop is or whatever. It will be more enjoyable and manageable I think, and also you will level up your skill.

I do think you should leave the emojis out of the summary, people will think you had ChatGPT write the description lol.

Good luck! It sounds like an endeavor. It will probably not get some wide adoption, as you mention there are plenty of these tools already, but that is not the reason to do it.

Let me revise that statement to - it’s better in every metric (compression speed, compressed size, feature set, most importantly decompression speed) compared to all other compressors I’m aware of, apart from xz and bz2 and potentially other non-lz compressors in the best compression ratio aspect.

Your Cloudflare post literally says “a new compression algorithm that we have found compresses data 42% faster than Brotli while maintaining almost the same compression levels.” Yes, I get that in some circumstances where compression speed is important, this might be very useful. I don’t see the point in talking further in circles anymore, thank you for the information.

You must be living in a different bubble than me then, because I see zstd used everywhere, from my Linux package manager, my Linux kernel boot image, to my browser getting served zstd content-encoding by default

Clearly a different bubble lol.

What distro are you using that uses zstd? Both kernel images and packages seem like a textbook case where compressed size is more important than speed of compression… which would mean not zstd. And of course I checked, it looks like NixOS uses bz2 for kernel images (which is obviously right to me) and gzip (!) for packages? Maybe? I’m not totally up to speed on it yet, but it sort of looks that way.

I mean I see the benchmarks, zstd looks nice. I checked this:

https://tools.paulcalvano.com/compression-tester/

… on lemmy.world, and it said that lemmy.world wasn’t offering zstd as an option, In its estimate, Brotli is way better than gzip, and sort of equivalent with zstd with zstd often being slightly faster in compression. I get the idea, it sounds cool, but it sort of sounds like some thing that Facebook is pushing that’s of dubious usefulness unless you really have a need for much faster compression (which, to be fair, is a lot of important use cases).

Yeah, I think of bz2 as sort of maximal compression at the cost of slower speed, gzip as the standard if you just want “compression” in general and don’t care that much, and then a little menagerie of higher performance options if you care enough to optimize. The only thing that struck me as weird about what you were saying was claiming it’s better in every metric (instead of it just being a good project that focuses on high speed and okay compression) and a global standard (instead of being something new-ish that is useful in some specific scenarios). And then when I tried both zstd and this other new Facebook thing and they were both worse (on compression) than bz2 which has been around for ages I became a lot more skeptical…

“The party cannot fail, it can only be failed.”

Anyway, like I said I don’t plan to have an extensive debate. I’ll leave you with this:

https://jacobin.com/2021/08/hitler-stalin-pact-nazis-communist-deportation-soviet

Edit: Fun fact, I’ve actually visited Trotsky’s house in exile in Mexico City.

What are you basing this all on?

$ time (cat optimizer.bin | bzip2 > optimizer.bin.bz2)

real	0m4.352s
user	0m4.244s
sys	0m0.135s

$ time (cat optimizer.bin | zstd -19 > optimizer.bin.zst)

real	0m12.786s
user	0m28.457s
sys	0m0.237s

$ ls -lh optimizer.bin*
-rw-r--r-- 1 billy users 76M Oct 20 17:54 optimizer.bin
-rw-r--r-- 1 billy users 56M Oct 20 17:55 optimizer.bin.bz2
-rw-r--r-- 1 billy users 59M Oct 20 17:56 optimizer.bin.zst

$ time (cat stocks-part-2022-08.tar | bzip2 > stocks-part-2022-08.tar.bz2)

real	0m3.845s
user	0m3.788s
sys	0m0.103s

$ time (cat stocks-part-2022-08.tar | zstd -19 > stocks-part-2022-08.zst)

real	0m34.917s
user	1m12.811s
sys	0m0.211s

$ ls -lh stocks-part-2022-08.*
-rw-r--r-- 1 billy users 73M Oct 20 17:57 stocks-part-2022-08.tar
-rw-r--r-- 1 billy users 26M Oct 20 17:58 stocks-part-2022-08.tar.bz2
-rw-r--r-- 1 billy users 27M Oct 20 17:59 stocks-part-2022-08.zst

Are you looking at https://jdlm.info/articles/2017/05/01/compression-pareto-docker-gnuplot.html or something? I would expect Lempel-Ziv to perform phenomenally on genomic data because of how many widely separated repeated sequences the data will have… for that specific domain I could see zstd being a clear winner (super fast obviously and also happens to have the best compression, although check the not-starting-at-0 Y axis to put that in context).

I have literally never heard of someone claiming zstd was the best overall general purpose compression. Where are you getting this?

AOC has caped for Israel and the DNC

Sure she has lol

https://www.wsws.org/en/special/library/foundations-ger/10.html

73 - The KPD had been established as a response to the betrayal of social democracy. But it proved just as unable as the SPD to weld together the working class and lead it into a struggle against the Nazis. A ten-year campaign against “Trotskyism” had politically corroded the party and transformed its leadership into a willing tool of Stalin. It repeated all the opportunist and ultra-left errors, against which Lenin and Trotsky had fought ten years before, and hid its paralysis and fatalism behind radical phrase-mongering. Until 1933, Trotsky tried relentlessly to correct the wrong course of the KPD. His writings on Germany from these years, which fill two thick volumes, prove his genius as a Marxist and political leader. Banished to a remote Turkish island, forced to rely on newspapers and reports from political friends, Trotsky demonstrated an understanding of German events and their internal dynamics that remains unparalleled to this day. He foresaw the events clearly and precisely and developed a convincing alternative to the devastating course of the KPD. The KPD responded not with arguments, but with slanders, violence and the entire weight of the Moscow apparatus.

74 - At the heart of the policy of the KPD was the thesis of social fascism. From the fact that both fascism and bourgeois democracy were forms of capitalist rule, the Comintern drew the conclusion that there was no contradiction between them, not even a relative one. Fascism and social democracy were the same―in the words of Stalin: “not antipodes, but twins”―the social democrats therefore were “social fascists”. The KPD rejected any collaboration with the SPD against the rightwing danger and, in some cases, even went so far as to make common cause with the Nazis―for example, when it supported the referendum initiated by the Nazis in 1931 to bring down the SPD-led Prussian state government. Occasionally it called for “a united front from below”. But this was not an offer to collaborate, but an ultimatum to the SPD members to break with their party.

75 - Trotsky decisively opposed this form of vulgar radicalism. He recalled that Marx and Engels had protested fiercely when Lassalle had called feudal counterrevolution and the liberal bourgeoisie “one reactionary mass”. Now Stalin and the KPD were repeating the same error. “It is absolutely correct to place on the Social Democrats the responsibility for the emergency legislation of Brüning as well as for the impending danger of fascist savagery. It is absolute balderdash to identify Social Democracy with fascism”, he wrote. “The Social Democracy, which is today the chief representative of the parliamentary-bourgeois regime, derives its support from the workers. Fascism is supported by the petty bourgeoisie. The Social Democracy without the mass organizations of the workers can have no influence. Fascism cannot entrench itself in power without annihilating the workers’ organizations. Parliament is the main arena of the Social Democracy. The system of fascism is based upon the destruction of parliamentarianism. For the monopolistic bourgeoisie, the parliamentary and fascist regimes represent only different vehicles of dominion; it has recourse to one or the other, depending upon the historical conditions. But for both the Social Democracy and fascism, the choice of one or the other vehicle has an independent significance; more than that, for them it is a question of political life or death.”[3]

76 - Trotsky fought untiringly for a policy of the united front. This would have made it possible for the KPD to use the contradiction between social democracy and fascism to unite the working class, win the confidence of the social democratic workers and expose the social democratic leaders. In an article written at the end of 1931, entitled “For a Workers’ United Front Against Fascism”, he explained: “Today the Social Democracy as a whole, with all its internal antagonisms, is forced into sharp conflict with the fascists. It is our task to take advantage of this conflict and not to unite the antagonists against us.” One must “show by deeds a complete readiness to make a bloc with the Social Democrats against the fascists” and “understand how to tear the workers away from their leaders in reality. But reality today is―the struggle against fascism.” It was necessary to “help the Social Democratic workers in action―in this new and extraordinary situation―to test the value of their organizations and leaders at this time, when it is a matter of life and death for the working class.”[4]

77 - The refusal of the KPD to accept such a policy led to the German catastrophe.

I won’t say I agree with 100% of the analysis on that page but a lot of that last part of analysis seems completely spot-on to me. And, of course, how Trotsky predicted is exactly how it played out.

Sure.

I still feel like someone throwing criticism at FDR (along the lines of vandalizing AOC’s office because she’s “pro-genocide” if you ignore all the anti genocide she does), because he’s a “liberal” and therefore an enemy and the labor movement were the ones that did all the progress, would be making an error. That is what I am saying.

Policy that benefits the working class does not get handed down from benign rulers, but is something forced out of their hands.

Absolutely completely correct.

Historically, for these measures, liberals have often opposed them, such as the civil rights movement.

Also correct.

I still feel like someone throwing firebombs at FDR (along the lines of vandalizing AOC’s office because she’s “pro-genocide” if you ignore all the anti genocide she does), because he’s a “liberal” and therefore an enemy and the labor movement were the ones that did all the progress, would be making an error. That is what I am saying.

I feel like we had this conversation before at some point lol… I will summarize my point of view on it and then probably dip from it:

• KPD in 1917: We’re going to seize all the guns and overthrow you and make Communism because Communism
• German government + German people and unions + SPD: Fuck no you’re not (gunfire)
• KPD in 1918: Wooooooowwwwwwwwww okay fuck you, I see how it is
• Germany: Hey KPD you still can have a seat in government, you have to get the votes though, no shooting your opposition, no seizing
• KPD: Woooooooowwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww
• SPD in 1932: Hey we’re going to make an alliance with you because this guy is dangerous, we don’t care about the whole “trying to overthrow thing” that happened a generation ago
• KPD: Woooooooooooooooooooooooooooooooooooooooooooooowwwwwwwwwwwwwwwwwwwwwwwwwww
• Hitler: (wins)

And then much later:

• KPD (ones still alive): Hey we’re going to need all the trade unions to do what we want instead of what the workers want, because that’s leftism
• Germany: Lol fuck OFF

That is my summary. I realize you may have a different one but I don’t think I really want to get in an extended argument about it today. But yes, Germany in the 30s is a relevant example on this topic, I absolutely think.

(Oh, also, the KPD was being murdered by the Nazis with the aid of Stalin. The KPD was coordinating with Stalin, and he was selling them out because of course he was. I have absolutely no idea where you got this idea that the SPD was involved in what Hitler was doing to his opposition.)

Haven’t you heard? Shitting on liberals and not voting is all you need to do to defeat fascism. It’s literally the most important thing.

It’s too bad the Weimar Republic didn’t think of that. I feel like if only the German Communists had been spending all their energy shitting on the center, they might have had a pretty good chance of stopping Hitler from coming to power. Well, at least now we have the chance to try again, with the benefit of hindsight, and make sure we do that known successful strategy…

Not really, dude. Civil rights absolutely, social security, kind of, the activists didn’t create the idea but they gave muscle to the labor movement to the point that FDR got elected in the first place and had the momentum so sure, clean air act and clean water act, you must be joking, those were just liberal government things. The things from the “clean air act” end of the spectrum are actually really good examples of why having a functioning government is a good thing even if it means “electoralism,” meaning it can’t all just be people in the streets fighting. You need both sides of the equation: The vigor and blood to push things forward, and then the paper and system to lock it in. Without either side of that, it doesn’t work.

More to the point, stop shitting on people who did good things. If you live in America, you benefit from all of the things on that list. Look for enemies elsewhere. This is the left’s favorite thing, to turn its guns exclusively on its own side, and it’s super good at it.

In terms of instances, there simply isn’t any appetite for the type of instances and community culture you want. You are simply in a minority here.

Severely tempted to code it up and see what the interest level is. IDK, I am lazy also, so let’s see. In any case thank you for your constructive input lol.

I feel like we’ve had a debate on this before.

Are we having a debate? I honestly was not aware if so lol

This would just negate the concept of communities built up by moderators.

Correct. Moderators should not “own” the communication that goes on in “their” communities, they definitely shouldn’t look at people as “their” users as I’ve heard some of them say before. We are just people. We are allowed to say things. The fact that letting people say things even if the moderators don’t want them to, would do damage to their concept, is a flaw with their concept.

Also, in many case, instances have rules before communities. How does that system work here?

To a certain extent, it is cultural. At the end of the day, the instance admins can physically control whatever passes through their server. In the old school Usenet sense, someone who was in that role would never modify someone else’s message. It just was this kind of wild fascism that would never be done except in the most dire circumstances (there were actually arguments about it when spam started cropping up, some people felt like even removing spam was going too far). Now, even someone who doesn’t own the server hardware feels empowered to set “rules” as you say for what people are allowed to say to each other, sometimes very arbitrary and clearly self-serving or etc. In my opinion, success lies somewhere between those two extremes: People generally being able to talk to one another (and the architecture being designed where it’s assumed that they’re allowed to) even if someone else doesn’t like it and wants to make rules against it, but still moderation set up for people who want it to the extent that they want it.

There’s obviously still a need for someone to take responsibility for deleting spam, harassment, or abusive content, and there’s going to be a grey area. I feel like, generally, you can let people control their own feeds and moderation that applies to them, and they will probably decide to configure it in a way where the anti-spam protection is applied to their feed and their weird additional arbitrary rules are not. That’s what I was saying.

The creator doesn’t curate it beyond curating what communities are visible in it.

Yes, I’m aware. I was proposing a new way in which it could work, I know that currently it doesn’t work that way. I don’t even know that the thing I spitballed is the way to do it, just someone asked how it could work, so I spitballed one possible way.

People. What a bunch of bastards.

Make it pull instead of push. Each user has way too little control over their own experience in my opinion. To me from an old-school-internet background, it’s very weird that a moderator can override what comments you’re allowed to see or not allowed to see. I much prefer Bluesky’s model, where you pick your moderators, and someone can’t override you and decide that certain comments you’re not allowed to read just because those comments happened to land within that person’s little domain after they were the first to claim the “worldnews” name for their community or whatever.

How to graft that onto Lemmy is a little bit difficult. It’s just a different model. I mean you could have a list of moderators whose decisions you want to block (similar to your list of users you want to block) – if any of those moderators removed a comment, you can still read it, their decisions just don’t affect your feed in any way. That would be a simple hack, sort of a useful check on their “power” if you want to say it that way, although it’s definitely a little bit rough approach. Probably a more holistic way would be to restructure how content even gets shared around. I haven’t looked at how Piefed does “feeds,” but that might be one good approach; let someone create or share a “politics” feed for example, and it can be a modification of someone else’s feed (“!news@lemmy.world but take out the Trump stuff” or “block these specific annoying users” or “ignore decisions by these two moderators”), so that it’s not a monopoly in terms of who gets to curate and control the content. You could subscribe to !betterpolitics@lemmy.world for example, and it’s just the identical posts sourced from !politics@lemmy.world, but with some users that are widely disliked banned, and then also with certain moderators who consistently make bad decisions disabled. That’s a lot harder to implement of course… IDK, this is just me thinking out loud about solutions I could see, but hopefully it makes some kind of sense.

What do you miss from Reddit?

Activity in niche communities, but that’s changing slowly.

Actually, two other things I do miss from Reddit: In the heyday (and even still to some extent now), it was so massive that you could have whole communities of types of real-world people you would never interact with. There is a subreddit for cops, one for air traffic controllers, one for sex workers, one for working historians to answer the general public’s questions, and so on. Arnold Schwarzenegger and Ghislaine Maxwell had active Reddit accounts. You could come into contact (in their weird text-box-only way) with people you would never come in contact with, and more to the point you could see what their hivemind looked like and their consensus on public issues. I always liked Reddit’s community model better than the twitter “everything goes on the pile” model, because you could have these for-real communities develop, and it was fascinating sometimes to see what they thought of things or watch them in action.

Edit: Oh, the other thing, AMAs of real public figures, similar idea

heavy-handed moderation and echo chambers where any dissenting opinion gets buried

I have bad news for you lol

It is fine, Lemmy is far superior. But, their baffling decision to copy Reddit’s “lords and peasants” model of moderation has led to a lot of the same moderation rot on Lemmy I am sad to say. It’s just in less of a late stage terminal form as it was on Reddit. At least the echo chambers are separate echo chambers, and they can yell across the void at each other. lemmy.ml is pretty much the only community that is severely balkanized to its own isolated community where politics / geopolitics are concerned.

In general, Lemmy is nice because it is more varied. lemmy.world is the most Reddit-like in terms of having a “hivemind,” then there are particular smaller servers with their own cultures going on. It is more quiet but a lot more human in my opinion.

Enjoy.

Yes, Lempel-Ziv is incredibly fast in compression. That’s because it’s a sort of elegant hack from the 1970s that more or less gets lucky in terms of how it can be made to work to compress files. It’s very nice. You said “by almost any metric,” though, not “by compression speed and literally nothing else.” There is a reason web pages default to using gzip instead of zstd for example.

Absolutely no idea what you’re on about with >100 MB. I’ve used bzip2 for all my hard disk backups for about 20 years now, and I think I broke the 100 MB barrier for local storage at some point during that time.

the current state of the art for generic compression by almost any metric

$ ls -lh optimizer*
-rw-r--r-- 1 billy users 76M Oct 19 15:51 optimizer.bin
-rw-r--r-- 1 billy users 56M Oct 19 15:51 optimizer.bin.bz2
-rw-r--r-- 1 billy users 60M Oct 19 15:51 optimizer.bin.zstd

I mean apparently not.

(Lempel-Ziv is not the best compression that’s currently known by a wide margin. It’s very fast and it’s nicely elegant but I would expect almost any modern “next gen compression” to be based on Huffman trees at the very core, or else specialized lossy compression. Maybe I am wrong, I’m not super up to speed on this stuff, but zstd is not state of the art, that much I definitely know.)

Of course this is not better at generic compression because that’s not what it’s for.

They specifically offered csv as an example of a thing it can handle, that’s why I chose that as one of the tests.