Post-Quantum Key Exchange in Nginx

When testing an HTTPS server, for example with SSL Labs, you may encounter the following warning:

This server does not support PQC (Post-Quantum Cryptography) key exchange.

The first questions that come to mind are: what does this mean, why does a web server need post-quantum cryptography, and are quantum computers already being used to break HTTPS at scale?

In practice, things are somewhat simpler. Quantum computers powerful enough to practically attack modern Internet cryptography do not yet exist. However, the development of quantum computing poses a real long-term threat, primarily to widely used public-key algorithms such as RSA, classical Diffie–Hellman, and elliptic-curve cryptography.

This is why the transition to PQC is starting in advance. This is particularly important because of the harvest now, decrypt later scenario: an attacker can capture encrypted traffic today and attempt to decrypt it in the future once sufficiently powerful computing resources become available.

What Is ML-KEM?

In August 2024, the U.S. National Institute of Standards and Technology (NIST) approved FIPS 203, which defines ML-KEM — Module-Lattice-Based Key-Encapsulation Mechanism.

ML-KEM is not designed to encrypt transmitted data directly. Instead, it is used to securely establish a shared secret between the parties to a connection. TLS then derives symmetric encryption keys from this shared secret and uses them to protect subsequent traffic.

The standard defines three parameter sets:

• ML-KEM-512;
• ML-KEM-768;
• ML-KEM-1024.

For TLS, hybrid schemes are particularly interesting because they combine a classical cryptographic mechanism with a post-quantum one.

One such option is: X25519MLKEM768

It combines:

• X25519 — a modern classical key exchange mechanism;
• ML-KEM-768 — a post-quantum key encapsulation mechanism.

This hybrid approach preserves classical cryptographic protection while also adding protection against potential future quantum attacks.

PQC Support in OpenSSL 3.5

Starting with OpenSSL 3.5, the library includes native support for ML-KEM and hybrid key exchange mechanisms for TLS 1.3.

OpenSSL 3.5 supports, among others:

• X25519MLKEM768;
• SecP256r1MLKEM768;
• SecP384r1MLKEM1024.

Причём X25519MLKEM768 is included in the default group list in OpenSSL 3.5 and has a high priority during TLS 1.3 negotiation.You can check the installed OpenSSL version with:

openssl version

To check the OpenSSL version used by Nginx:

nginx -V 2>&1 | grep -E 'nginx version|OpenSSL'

Starting with Debian 13, a suitable OpenSSL version is included in the standard distribution.

Configuring X25519MLKEM768 in Nginx

When using OpenSSL 3.5, the default setting:

ssl_ecdh_curve auto;

is sufficient in most cases because OpenSSL automatically uses its current list of supported TLS groups. However, if you want to explicitly define the order of the groups, you can configure Nginx as follows:

ssl_ecdh_curve X25519MLKEM768:X25519:prime256v1:secp384r1;

Here:

• X25519MLKEM768 — a hybrid post-quantum key exchange mechanism;
• X25519 — a modern classical mechanism;
• prime256v1 — NIST P-256;
• secp384r1 — NIST P-384.

This list places the post-quantum mechanism first while preserving compatibility with clients that do not yet support it. After changing the configuration, test it:

nginx -t

Then apply the changes:

systemctl reload nginx

How to Check X25519MLKEM768 Support

In OpenSSL 3.5, you can display the available TLS 1.3 groups with:

openssl list -tls1_3 -tls-groups

Now test the HTTPS connection directly:

openssl s_client \
        -connect domain.com:443 \
        -servername domain.com \
        -tls1_3 \
        -groups X25519MLKEM768

If negotiation is successful, the output should contain information similar to:

Negotiated TLS1.3 group: X25519MLKEM768

Do I Need to Replace the SSL Certificate?

No. X25519MLKEM768 is used during the TLS key exchange phase, so the existing certificate can continue to be used.

These are two separate mechanisms:

• the certificate authenticates the server;
• X25519MLKEM768 participates in securely establishing the shared secret for the TLS session.

Therefore, using a standard RSA or ECDSA certificate does not prevent you from using post-quantum key exchange.

What If Upgrading to Debian 13 Is Not Possible?

If the server is running, for example, Debian 12 with OpenSSL 3.0 and upgrading the entire operating system is currently not possible, there are several options.

Option 1. Upgrade the Operating System

This is the preferred option. Debian 13 includes OpenSSL 3.5 with native support for ML-KEM and X25519MLKEM768, so there is no need to install an additional cryptographic provider or rebuild system libraries manually.

Option 2. Use OQS Provider

For OpenSSL 3, the Open Quantum Safe project provides oqs-provider. It allows additional post-quantum algorithms to be loaded through the OpenSSL Provider mechanism without replacing the system libssl. However, this approach should primarily be considered an experimental or temporary solution. The Open Quantum Safe project itself warns that oqs-provider is intended mainly for research and prototyping and is not currently recommended for protecting sensitive production data.

Option 3. Rebuild Nginx with a Different OpenSSL Version

Technically, you can build a separate Nginx version using the required OpenSSL release. For a normal production server, however, this is the most complicated approach because you will have to maintain OpenSSL and Nginx updates and security patches yourself. Unless there is a strong reason to do so, this approach is best avoided.

OQS Provider for Nginx on Debian 12

The remainder of this article describes an experimental approach for connecting OQS Provider specifically to the Nginx process. Importantly, we will not replace the system OpenSSL installation and will not enable an additional provider globally for the entire system.

Installing the Build Tools

Install the required packages:

apt-get update
apt-get install -y \
    build-essential \
    git \
    cmake \
    ninja-build \
    libssl-dev \
    pkg-config

Before continuing, check OpenSSL again:

openssl version

Do not manually replace the system libssl just for this configuration. Doing so may break Nginx, apt, curl, PHP, and other applications that depend on the system OpenSSL installation.

Building liboqs

Go to the source directory:

cd /usr/local/src

Clone the source code:

git clone https://github.com/open-quantum-safe/liboqs.git
cd liboqs

At the time of writing, the current release is liboqs 0.16.0, which also contains security fixes:

git checkout 0.16.0

Configure the build:

cmake \
    -S . \
    -B build \
    -GNinja \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/usr/local \
    -DOQS_DIST_BUILD=ON \
    -DBUILD_SHARED_LIBS=ON
    

Build the library:

ninja -C build

Run the tests:

ctest --test-dir build

If the tests complete successfully, install the library:

ninja -C build install
ldconfig

Verify that the dynamic linker can see it:

ldconfig -p | grep oqs

Building oqs-provider

Now obtain the source code for the OpenSSL provider:

cd /usr/local/src
git clone https://github.com/open-quantum-safe/oqs-provider.git
cd oqs-provider

There is an important detail here: the versions of oqs-provider and liboqs must be compatible.Before deploying this configuration on a production server, check the project's release notes and use a tested combination of versions. Building from the main branch means using the current development code, which may change over time. For a test environment, configure the build as follows:

cmake \
    -S . \
    -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=/usr/local
cmake --build build -j"$(nproc)"

Install the provider:

cmake --install build
ldconfig

Finding oqsprovider.so

Locate the provider library:

find /usr/local /usr/lib \
    -type f \
    -name 'oqsprovider.so' \
    2>/dev/null

Depending on the system architecture and configuration, the path may look like:

/usr/local/lib/x86_64-linux-gnu/ossl-modules/oqsprovider.so

or:

/usr/local/lib64/ossl-modules/oqsprovider.so

You will need the exact path in the next step.

Creating a Separate OpenSSL Configuration for Nginx

It is better not to enable OQS Provider in the system-wide /etc/ssl/openssl.cnf. Instead, create a separate configuration that will be used only by Nginx:

nano /etc/ssl/openssl-nginx-oqs.cnf

Add:

openssl_conf = openssl_init

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
oqsprovider = oqsprovider_sect

[default_sect]
activate = 1

[oqsprovider_sect]
activate = 1
module = /usr/local/lib/x86_64-linux-gnu/ossl-modules/oqsprovider.so

Replace the path in the module directive with the actual path obtained in the previous step.

Checking That OQS Provider Loads Correctly

Run:

OPENSSL_CONF=/etc/ssl/openssl-nginx-oqs.cnf \
    openssl list -providers

Both providers should appear in the output:

Providers:
  default
    name: OpenSSL Default Provider
    status: active

  oqsprovider
    name: OpenSSL OQS Provider
    status: active

Now you can test the Nginx configuration. If you run the normal command:

nginx -t

with the following configuration:

ssl_ecdh_curve X25519MLKEM768:X25519:prime256v1:secp384r1;

the standard OpenSSL installation on Debian 12 may return an error similar to:

SSL_CTX_set1_curves_list("X25519MLKEM768:...") failed

This does not mean that the Nginx configuration itself is incorrect. It means that the additional OpenSSL Provider was not loaded for this particular process. Test Nginx with the custom OpenSSL configuration:

OPENSSL_CONF=/etc/ssl/openssl-nginx-oqs.cnf \
    nginx -t

If everything is configured correctly, the result should be:

syntax is ok
test is successful

Enabling OQS Provider Only for the Nginx Service

Create a systemd override:

systemctl edit nginx

Add:

[Service]
Environment="OPENSSL_CONF=/etc/ssl/openssl-nginx-oqs.cnf"

Check the resulting configuration:

systemctl cat nginx

At the end of the output, you should see:

# /etc/systemd/system/nginx.service.d/override.conf

[Service]
Environment="OPENSSL_CONF=/etc/ssl/openssl-nginx-oqs.cnf"

Reload the systemd configuration:

systemctl daemon-reload

Test Nginx again:

OPENSSL_CONF=/etc/ssl/openssl-nginx-oqs.cnf nginx -t

After a successful test, restart the service:

systemctl restart nginx

Make sure it is running:

systemctl status nginx

Final PQC Test

Now test the connection:

OPENSSL_CONF=/etc/ssl/openssl-nginx-oqs.cnf \
    openssl s_client \
    -connect domain.com:443 \
    -servername domain.com \
    -tls1_3 \
    -groups X25519MLKEM768

If the client and server successfully negotiate the hybrid group, the TLS 1.3 connection will use X25519MLKEM768.

Conclusion

Post-quantum cryptography in HTTPS is no longer purely experimental technology. Starting with OpenSSL 3.5, ML-KEM and the hybrid X25519MLKEM768 mechanism are supported directly by the library, while Debian 13 ships with a suitable OpenSSL version. For a new or upgraded production server, the preferred approach is therefore straightforward:

• use Debian 13 or another operating system with OpenSSL 3.5 or later;
• use a current version of Nginx;
• keep OpenSSL's default TLS group list or explicitly add X25519MLKEM768;
• verify the result using openssl s_client and SSL Labs.

Using OQS Provider on an older system is possible as a temporary or experimental solution. For a production server, however, moving to an OpenSSL version with native support for standardized PQC algorithms is preferable.

If this guide was useful, share it with colleagues and send any corrections or questions.

Reproducing this material on another website without the author’s permission is prohibited.