It is possible, what you’re looking for is a reverse proxy: it’s an HTTP server that will listen to the standard ports for HTTP and HTTPS that will redirect traffic to the chosen service based on the domain name or URL.
In your case, every subdomain would point to your VPS’s IP and traffic that’s for mastodon.example.tld will be seemlesly proxied to your Mastodon container.
Do some research on Caddy or Nginx, and I strongly recommend you learn Docker Compose and Docker networking, it will help you make it easier to maintain everything.
PS: CNAME pointing to A record is the way to go. You can do it one better by having a CNAME entry for *.example.tld, so that you don’t have to create a CNAME entry for every new service you deploy, but you better make sure that your reverse proxy won’t proxy requests to an unexpected container when requesting a bogus subdomain.
I’m not familiar with Nextcloud, but from reading the How to use this? section of the README I believe you can run it behind a reverse proxy:
(Emphasis mine, in “Explanation of the command”)
My understanding is you only have to forward traffic from the reverse proxy to the port 8080. It uses a self-signed certificate though, so you might check if the reverse proxy you are using checks certificates signatures for upstream servers.