Learnings on Solaris™: VirtualBox VM RDP encryption

Nowadays encrypting everything seems a requirement.
New SPARC and Intel systems posseses hardware support for off-loading encryption.

VirtualBox can reference a self-signed CA, certificates and private keys for RDP.

We'll be using the PEM (Privacy Enhanced Mail) format for certificates and keys.
It's primarily used by UNIX and consists on a Base64 representation of the binary DER.

PEM's content is delimited by headers and footers, such as:

-----BEGIN CERTIFICATE----- -----BEGIN ENCRYPTED PRIVATE KEY-----

-----END CERTIFICATE----- -----END ENCRYPTED PRIVATE KEY-----

Create a directory to store the X.509 certificates and RSA keys generated by OpenSSL:

$ mkdir -m 0700 ~/.VirtualBox/encryption

$ cd ~/.VirtualBox/encryption 

Create the CA by creating its self-signed certificate:

$ openssl req
-new -x509
-days 365
-extensions v3_ca
-keyout ca_priv_key.pem
-out ca_cert.pem

Generate the RDP server's private key:

$ openssl genrsa 

  -out srv_priv_key.pem

Generate the RDP server's certificate request based on its private key:

$ openssl req 

  -new 

  -key srv_priv_key.pem

-out srv_cert_req.pem

Generate the CA signed RDP server certificate from its certificate request:

$ openssl x509

  -in srv_cert_req.pem 

  -out srv_cert.pem

-CA ca_cert.pem
-CAkey ca_priv_key.pem

-req
-days 365
-setserial 01

Finally, due replacing path for, in my case, /home/.../.VirtualBox/encryption:

$ VBoxManage modifyvm vm1 

  --vrdeproperty="Security/Method=TLS"

  --vrdeproperty "Security/CACertificate=path/ca_cert.pem"

  --vrdeproperty "Security/ServerCertificate=path/srv_cert.pem"

  --vrdeproperty "Security/ServerPrivateKey=path/srv_priv_key.pem"

An UNIX RDP client such as rdesktop can connect as follows:

$ rdesktop -u user -p - rdp_srv:rdp_srv_port

Interestingly, VBoxManage showvminfo vm1 --details insists on displaying encryption as RDP4. If I didn't make any mistake, this is rather odd as Security/Method has been forced to TLS and rdesktop -4 fails.

OpenSSL References:

Monday, July 2, 2012

VirtualBox VM RDP encryption