Even if you have a password for your ssh key, malware on your system can just wait until you enter the password.

My point is that SSH access is very powerful, and effectively means that the security of the SSH server is reduced to the security of the SSH client. If your SSH client is pwned, so is your server. If you have 10 devices each with ssh access to each other, then if any one device is pwned, all devices are pwned as well.

This is not the case for systems designed for file sharing only. For example with syncthing, if one device gets pwned, all it can do is send files to the other devices.