Yeah, that's something I considered pretty carefully. The backup server is completely isolated from the rest of the network, it pulls the backups via ssh/rsync using a special user account that only has sudo permissions for the rsync command (and can only authenticate via ssh certificate). The only way to break the backup server from a compromised server would be to replace OpenSSH on the compromised server and then wait for the backup server to connect -- and then try to somehow break rsync.

Thanks for thinking of that though.