The handshake failed due to an unexpected packet format

Most likely your server requires explicit SSL, sometimes also known as TLS. It is called explicit SSL mode because after the connection is established, client explicitly issues a command to the server that initiates SSL/TLS negotiation. This is in contrast to implicit SSL mode where SSL negotiation is initiated just after successful connection. In implicit mode server and client knows to use SSL, because client uses protocol port, that is commonly used for secured traffic.

First try to connect to your server without SSL:

// C#

client.Connect("mail.example.com");

' VB.NET

client.Connect("mail.example.com")

Then, before logging-in, start explicit SSL negotiation. The command name differs for different protocols:

IMAP explicit SSL/TLS negotiation

// C#

client.Connect("mail.example.com");
client.StartTLS();

' VB.NET

client.Connect("mail.example.com")
client.StartTLS()

POP3 explicit SSL/TLS negotiation

// C#

client.Connect("mail.example.com");
client.STLS();

' VB.NET

client.Connect("mail.example.com")
client.STLS()

SMTP explicit SSL/TLS negotiation

// C#

client.Connect("mail.example.com");
client.StartTLS();

' VB.NET

client.Connect("mail.example.com")
client.StartTLS()

StartTLS and STLS methods negotiate security protocol with the server and secure channel using SSL or TLS. Now, your connection is secured.

Please note, that your server may not need SSL/TLS at all. In such case simply use Connect method.

Enabled SSL Protocols

On very rare occasions “handshake failed…” error may indicate that TLS is incorrectly configured on the client machine or on the server.

It is possible to force SSL v3.0 usage instead of TLS in explicit mode:

// C#

client.SSLConfiguration.EnabledSslProtocols = SslProtocols.Ssl3;
client.Connect("mail.example.com");
client.StartTLS();

' VB.NET

client.SSLConfiguration.EnabledSslProtocols = SslProtocols.Ssl3;
client.Connect("mail.example.com");
client.StartTLS();

It is also possible to force SSL v3.0 usage instead of TLS in implicit mode:

// C#

client.SSLConfiguration.EnabledSslProtocols = SslProtocols.Ssl3;
client.ConnectSSL("mail.example.com");

' VB.NET

client.SSLConfiguration.EnabledSslProtocols = SslProtocols.Ssl3;
client.ConnectSSL("mail.example.com");

Self-signed certificates

Remember that you can ignore SSL certificate errors using ServerCertificateValidate event:

// C#

static void Validate(
    object sender,
    ServerCertificateValidateEventArgs e)
{
    const SslPolicyErrors ignoredErrors =
        SslPolicyErrors.RemoteCertificateChainErrors |
        SslPolicyErrors.RemoteCertificateNameMismatch;

    if ((e.SslPolicyErrors & ~ignoredErrors) == SslPolicyErrors.None)
    {
        e.IsValid = true;
        return;
    }
    e.IsValid = false;
}

client.ServerCertificateValidate += Validate;
client.Connect...

' VB.NET

Private Sub ValidateCerificate( _
    ByVal sender As Object, _
    ByVal e As ServerCertificateValidateEventArgs)

    Const ignoredErrors As SslPolicyErrors = _
        SslPolicyErrors.RemoteCertificateChainErrors Or _
        SslPolicyErrors.RemoteCertificateNameMismatch

    If (e.SslPolicyErrors And Not ignoredErrors) = SslPolicyErrors.None Then
        e.IsValid = True
        Return
    End If
    e.IsValid = False
End Sub

AddHandler client.ServerCertificateValidate, AddressOf Validate
client.Connect...