Whats the difference betweeen RDP and VPN?

A vpn session encrypts and tunnels traffic between end devices so it can't be read off the wire, this carries an overhead in terms of processing as well as an extra payload, rdp is not tunneled through an encrypted tunnel and is routed normally, therefore it requires less processing but is inherently insecure. If the vpn is too slow it could be worth upgrading your equipment, rdp sessions if not through a vpn should be locked down only to known remote clients.