As another user mentioned, package managers are specific to distributions rather than DEs. The main difference between them is that they’re developed by the respective distribution teams, but there are some practical differences too. For example, apt supports versioned dependencies while pacman doesn’t because of the different distribution models between Debian and Arch (monolithic vs. rolling release). This affects their dependency resolution strategy with each being better suited for it’s respective distribution.

To address your point about package managers being the main difference between distros, this isn’t quite true. As mentioned, different distros have different distribution models, priorities, and overall biases/opinions that affect the user experience in a variety of ways and make them better suited to different use cases. I would never dream of putting Arch on one of my servers in the same way that you’d probably never catch me installing Debian on my gaming machine.