This is a quick, no-nonsense guide on setting up a Git server on a VPS. I am using Debian, but other operating systems should be easy to figure out.
Make a git user:
sudo adduser git
Switch to the git user:
sudo su -l git
Create a .ssh dir in the git user's home directory and make it only accessible by the git user:
mkdir ~/.ssh
chmod 700 ~/.ssh
Create an authorized_keys file in the .ssh folder, and make it accessible only by the git user:
touch .ssh/authorized_keys
chmod 600 `.ssh/authorized_keys`
Create a public/private key pair locally to authenticate a user on a machine to connect to the remote server:
ssh-keygen -t ed25519
And finally copy it into the (remote) git user's .ssh/authorized_keys, using ssh-copy-id or by giving the public key to the server administrator.
Create directories within git's home dir (nested paths are allowed) with the .git extension, for example my-projects/my-repo.git or just my-repo.git.
git init --bare repo.git
There now exists an empty Git repository on the remote server.
The remote can now be added to a local repository:
git remote add origin git@server:my-repo.git
git push -u origin main
Add an entry to your local .ssh/config:
Host myhost
    HostName        example.com
    User            git
    IdentityFile    ~/.ssh/id_ed25519
And connect once without Git to verify the host:
ssh myhost
Done!