It’s been years since I had to admin Windows servers, but I was quite impressed with the number of MS products where the install and configuration tools would output the Powershell commands to carry out the changes you’d asked for. It made it quite a lot easier to automate. I’d love to see that paradigm catch on more widely, with the GUI and CLI having the same functionality and the GUI giving you the commands to run.
It depends what you want to do with it. If it’s just for storing files/backups then encrypt them before uploading and make sure the key never goes anywhere near the VPS. If it’s for serving up something like a simple website, you probably care more about data integrity than exfiltration, so make sure you have the security, including selinux or equivalent, locked down, and regularly run integrity checks. If it’s for running something interactive, or where data will be generated or downloaded to the machine, you’re out of luck, there’s no even theoretical way of securing that against an adversary with that much access.