Xshell supports the Public key user authentication method which is an alternative way of identifying the user to the remote server instead of typing the password.

To use the public key user authentication method, a user generates a key pair consisting of a public key (which everybody is allowed to know) and a private key (which conceal from the rest). Private key is used in the public key authentication process to generate signature and the public key is used by the server to verify the signature. In other words, private key works as an identification of a user. Also, the user should register his public key to the server for getting authenticated.

Generating a key pair

On the Tools menu, click User Key Manager.
Click Generate button to open User Key Generation Wizard.
In the Key Type list, select a proper key type. For the SSH1 protocol, only RSA algorithm is supported. So, select the RSA. For the SSH2 protocol, more than one type of algorithm is supported. Xshell supports both RSA and DSA.
In the Key Length box, type or select a key length. Longer keys provide better security and shorter keys provide better speed. The optimal key length for most applications is 1024 bits.
Click Next to proceed to the key generation step and wait until the key generation process completes. After the key generation process, click Next to enter the user key information.
In the Key Name box, type a key name.
In the Passphrase box, type a passphrase. The passphrase is used in encrypting a private key file.
In the Confirmation box, type the same passphrase you typed in the Passphrase box to confirm your input.
Click Next to register the public key to the server.

Now, all key generation steps are finished, but you must configure the server to accept your public key for authentication. To configure the server, see the following Registering a public key on the server section.

Registering a public key on the server

To put the key pair you have generated in the Generating a key pair section in use, you have to register the public key on your remote account so that the server can authenticate the user with it. Registering a public key varies with the version of SSH protocol and the vendor of SSH server.

SSH1 protocol: Select SSH1 in the Public Key Format list and copy the public key into the file $HOME/.ssh/authorized_keys. (Note: You need to create this file if it does not exist.)
OpenSSH server using SSH2: Select SSH2 - OpenSSH in the Public Key Format list, and copy the public key into the file $HOME/.ssh/authorized_keys2.
ssh.com's SSH server using SSH2: Select SSH2 - IETF SECSH in the Public Key Format list and click Save as... to save the public key into a file. Then copy the public key file to the directory $HOME/.ssh2/ and put the following line into the file $HOME/.ssh2/authorization:

Key mypublickey.pub

where mypublikey.pub is the public key file you have copied.
For other SSH server products: Refer to the SSH server manual from its provider.

Now, you are ready for the public key authentication.

Importing a user key file

If you are already using the public key authentication somewhere else and want to reuse the private key with Xshell, you can import the private key. Currently, Xshell can read RSA private keys for the SSH1 protocol and OpenSSH's RSA/DSA keys for the SSH2 protocol along with Xshell's own user key files.

Follow the steps below to import user keys into Xshell's User Key Manager:

On the Tools menu, click User Key Manager.
Click Import... button to choose the user key file which will be imported.
After choosing the user key file, Passphrase dialog box will show up. (Only if the user key has passphrase setup.)
In the Passphrase box, enter the passphrase of the user key.
Click OK.