# DTLS and PQC: UDP Cryptography Goes Quantum-Safe

**Source**: https://quantumsequrity.com/blog/dtls-pqc
**Category**: Network Protocols

---

[← Back to Blog](../../blog.html) Network Protocols

# DTLS and PQC: UDP Cryptography Goes Quantum-Safe

13 min read

When most people hear about secure connections on the internet they think about TLS, the lock icon in the browser, the handshake that protects every web page. TLS lives on top of TCP, which is the reliable connection-oriented transport that retransmits dropped packets and reorders out-of-order data so the application sees a clean byte stream. Many of the most demanding applications cannot afford that overhead. Voice calls, video conferencing, online games, IoT telemetry, and DNS queries run over UDP, the unreliable datagram transport. They cannot wait for TCP to retransmit a packet that arrived two seconds ago. They need to send a packet, accept that some will be lost, and move on.

DTLS, short for Datagram Transport Layer Security, is the protocol that gives those UDP applications the same cryptographic guarantees TLS gives TCP applications. It was specified by Eric Rescorla and Nagendra Modadugu, first published as RFC 4347 in 2006, then revised through RFC 6347 (DTLS 1.2) and most recently RFC 9147 (DTLS 1.3) in April 2022. The latest version aligns DTLS with TLS 1.3 and brings most of the same modernization. Now the question facing protocol designers is how to layer post-quantum cryptography on top of DTLS without breaking the very property that makes DTLS attractive in the first place — small, self-contained datagrams.

This article walks through the structure of DTLS 1.3, the specific challenges PQC introduces, and the work happening at the IETF to make UDP-based cryptography quantum-safe.

## Why DTLS Exists at All

The first thing to understand is why TLS cannot just be used directly over UDP. TLS assumes a reliable, in-order byte stream. The handshake messages can be arbitrarily large. Records can be split across TCP segments. The receiver does not need to worry about message boundaries because TCP delivers everything eventually.

Drop those guarantees and TLS falls apart. A handshake message that does not fit in one UDP packet has no obvious way to reassemble. A retransmission has no acknowledgment scheme. A handshake whose order matters cannot tolerate out-of-order delivery. DTLS solves these problems by building handshake fragmentation, retransmission timers, and explicit sequence numbers into the protocol itself rather than relying on the transport.

Once the handshake completes, encrypted records flow over UDP just like any other datagram. Each record is self-contained: the receiver decrypts it without needing the previous packet. The connection state advances regardless of order, and lost packets simply stay lost.

DTLS is everywhere. WebRTC uses DTLS to bootstrap the keys for SRTP, which encrypts the actual audio and video. CoAP, the constrained application protocol for IoT, uses DTLS for transport security. QUIC originally used DTLS for its key establishment before evolving into its own design. The OpenVPN project's UDP mode is built on DTLS. Many private VPN clients use DTLS as a fallback when TCP is blocked.

## DTLS 1.3 in One Page

DTLS 1.3 mirrors TLS 1.3 closely. The handshake is reduced to one round trip in the common case. The cipher suites are the AEAD ones from TLS 1.3 (AES-128-GCM, AES-256-GCM, ChaCha20-Poly1305). The key schedule uses HKDF with the same Extract-then-Expand pattern. The certificate signing uses the same algorithms (RSA-PSS, ECDSA over P-256/P-384, EdDSA over Ed25519/Ed448).

The differences are mostly mechanical. Each handshake message carries a 16-bit message sequence number and a 24-bit fragment offset and length, so a long ClientHello can be split across multiple datagrams and reassembled by the receiver. The record layer carries an explicit 16-bit or 8-bit sequence number to prevent replay and to let the receiver decrypt out of order. The transcript hash, the running fingerprint of the handshake that prevents downgrade attacks, treats fragmented messages as if they had been sent whole.

A new feature in DTLS 1.3 is the connection ID, RFC 9146, which lets a client roam across IP addresses without breaking the connection. This matters for mobile devices that move between WiFi and cellular and for NAT timeouts that change the apparent source port.

The handshake itself opens with a ClientHello that lists supported groups, supported signature algorithms, and a key share for the chosen group. The server replies with a ServerHello containing its own key share, plus its certificate, a CertificateVerify signature, and a Finished message. The client verifies and sends its Finished. Done.

## The PQC Sizing Problem

Now layer post-quantum on top. The natural way is to replace the elliptic curve key share with an ML-KEM (Kyber) key share, or to use a hybrid that includes both. ML-KEM is standardized in NIST FIPS 203 (August 2024), and its three parameter sets give different size and security tradeoffs.

ML-KEM-512 has an 800-byte public key and a 768-byte ciphertext. ML-KEM-768 has a 1184-byte public key and a 1088-byte ciphertext. ML-KEM-1024 has a 1568-byte public key and a 1568-byte ciphertext. Compare these to X25519, which has a 32-byte public key and produces a 32-byte ciphertext (the encrypted Diffie-Hellman element).

A typical Ethernet path has an MTU around 1500 bytes. After the IP header (20 bytes), the UDP header (8 bytes), and the DTLS record header (around 5 to 13 bytes depending on version and connection ID), about 1450 bytes remain for handshake fragments. ML-KEM-768 fits in two fragments. ML-KEM-1024 plus the rest of the ClientHello (cipher suites, extensions, signature algorithms) often pushes a real handshake to three or four fragments per direction. With low-MTU paths over satellite, HF radio, or some VPN tunnels, you can see five or six fragments.

Each extra fragment is another UDP packet, another chance for loss, another step in the retransmission timer. DTLS handshake retransmissions in real networks already cause noticeable latency. PQC-sized handshakes make this worse.

The signature is even more painful. Classical ECDSA signatures are 64 bytes; ML-DSA-44 (FIPS 204) signatures are 2420 bytes, ML-DSA-65 are 3309 bytes, and ML-DSA-87 are 4627 bytes. A typical certificate today is a few hundred to a few thousand bytes. A PQC certificate using ML-DSA-87 plus its signature plus the chain to the root can easily reach 10 to 15 KB. Sending that over DTLS means dozens of fragments per handshake.

## Strategies for Living With Big Handshakes

The IETF TLS Working Group has been thinking about PQ-DTLS since around 2019 and the conversation has accelerated since the Kyber selection in 2022. Several mitigations have been discussed.

The most direct one is to negotiate fragment sizes carefully. DTLS already has a path MTU discovery mechanism (RFC 9147 Section 4.4) that lets endpoints converge on a working fragment size before sending big handshake messages. Implementations that did not exercise this code path now have to.

Another is to favor smaller PQ parameters wherever possible. ML-KEM-768 is the IETF default for the hybrid X25519+Kyber construction (draft-ietf-tls-hybrid-design) precisely because the size is manageable. ML-KEM-1024 is reserved for cases where stronger security is required.

A third strategy is to compress the certificate chain. RFC 8879 defines TLS certificate compression using algorithms like Brotli or zstd. This works just as well in DTLS and can shave a few kilobytes off the chain.

A fourth idea, still being debated, is to use a smaller signature algorithm in the certificate. Falcon (FN-DSA) signatures are around 660 to 1280 bytes for the standardized parameter sets, much smaller than ML-DSA. The cost is an implementation that depends on floating-point arithmetic, which has historically caused timing side-channel risks. The IETF has debated whether to recommend FN-DSA, ML-DSA, or both for DTLS.

A more aggressive strategy is to skip the in-band certificate altogether. RFC 7250 raw public keys allow a server to send just a public key and the client to verify it through some out-of-band mechanism. This is common in IoT environments and in WebRTC. Removing the certificate and chain saves the bulk of the handshake size.

Finally, there is the experimental work on KEM-based authentication, sometimes called AuthKEM, which replaces the digital signature in the handshake with a second KEM operation. The ciphertexts are smaller than ML-DSA signatures. The IETF AuthKEM draft is still evolving and has not been broadly deployed.

## Replay, Reordering, and Anti-Forgery Under PQC

DTLS 1.3 already has solid replay protection through the explicit sequence numbers and a sliding window at the receiver. PQC does not change any of this. Once the handshake completes, AEAD records are protected by the same AES-GCM or ChaCha20-Poly1305 they always were, both of which are quantum-resistant in the symmetric sense (Grover's algorithm gives at most a square-root speedup, which is dealt with by 256-bit keys).

The handshake itself, however, has new attack surface. An ML-KEM operation has a small probability of decapsulation failure built into the design — for ML-KEM-512 this is around 2^-138, vanishingly small but not zero. Implementations must handle that gracefully without leaking timing information. There has been published research on Kyber side-channel attacks (Ravi et al. 2022, Pessl and Prokop 2021) that exploit imperfect masking in the decryption step. DTLS implementations must use constant-time Kyber.

Another concern is the handshake transcript. PQC ciphertexts and public keys are bigger, so the transcript is bigger, and the cost of computing the transcript hash grows. SHA-256 is fast on modern CPUs but on constrained devices the difference between hashing 200 bytes and hashing 5000 bytes per handshake matters.

For the harvest-now-decrypt-later threat, see [Harvest Now, Decrypt Later](harvest-now-decrypt-later). DTLS sessions that establish keys today using only classical ECDH are vulnerable in the same way as TLS sessions, and the mitigation is the same: deploy hybrid PQ-DTLS as soon as it is available.

## What Is in the Wild Right Now

OpenSSL 3.4, released October 2024, includes experimental support for hybrid X25519+Kyber768 in its TLS 1.3 stack. The DTLS 1.3 path inherits this through shared code but is less battle-tested. wolfSSL has shipped a similar feature flag.

Cisco AnyConnect, Microsoft RDP gateways, and several enterprise VPN vendors have announced internal pilots of PQ-DTLS for their UDP-mode tunnels. Public information is sparse.

WebRTC, the most widely deployed user of DTLS, has not yet flipped on PQ-DTLS. The Chromium bug tracker has tickets open for hybrid handshakes in DTLS-SRTP keying, but as of early 2026 they remain in design. Once it lands, every browser-to-browser voice and video call will get post-quantum confidentiality for the keying step.

CoAP and constrained-IoT devices are a harder problem. Many of those devices have less than 64 KB of RAM total. ML-KEM-512 ciphertexts and public keys do not fit comfortably alongside an application stack on a Cortex-M0. Research into smaller PQ variants and into hardware-accelerated lattice arithmetic on microcontrollers is active.

## How QNSQY Thinks About DTLS

QNSQY itself is a post-quantum cryptography tool, not a transport, but its hybrid construction (ML-KEM + X25519, ML-DSA + Ed25519) tracks the same design principles the IETF is using for hybrid TLS and DTLS. The key derivation step uses HKDF-SHA-256, the same as DTLS 1.3. See [Hybrid Encryption](hybrid-encryption) for the rationale and [ML-KEM Explained](ml-kem-explained) for the lattice math.

If you are building a system that uses DTLS for transport and QNSQY for at-rest, the design lessons carry across: protect the long-term keys, plan for PQC sizing now, and use hybrid until pure PQ has years of operational scrutiny.

## Frequently Asked Questions

### Does DTLS 1.3 require PQC?

No. DTLS 1.3 (RFC 9147) negotiates the same algorithms TLS 1.3 negotiates and PQC support arrives through TLS 1.3 extensions like the hybrid_design draft. Implementations adopt PQC at their own pace.

### Why is fragmentation such a problem?

Each fragment is a separate UDP packet. UDP loses packets. DTLS handshake messages must arrive in order before the receiver can advance. More fragments means more places for a single drop to delay the entire handshake by a retransmission timeout.

### Will ML-KEM-1024 ever be practical for DTLS?

Yes, on paths with consistently large MTU it already works. The trouble is paths that have to traverse the public internet, where any MTU reduction along the way fragments DTLS records below the IP layer (which DTLS hates). Over a private fiber or a curated overlay it is fine.

### What about QUIC?

QUIC has its own integrated key establishment that is conceptually similar to DTLS 1.3 but more tightly bound to the transport. The PQC story is parallel. The IETF QUIC working group is tracking the same hybrid design.

### Is SRTP affected?

SRTP itself uses symmetric AEAD for the actual media so it is quantum-resistant in the symmetric sense. The quantum risk is in the keying step, which today is DTLS-SRTP. PQ-DTLS-SRTP is the pathway to fix it. See [SRTP and PQC](srtp-pqc).

## Sources

1. Rescorla, E., Tschofenig, H., Modadugu, N. "The Datagram Transport Layer Security (DTLS) Protocol Version 1.3." IETF RFC 9147, April 2022. https://datatracker.ietf.org/doc/html/rfc9147
2. Rescorla, E., Modadugu, N. "Datagram Transport Layer Security Version 1.2." IETF RFC 6347, January 2012. https://datatracker.ietf.org/doc/html/rfc6347
3. NIST FIPS 203. "Module-Lattice-Based Key-Encapsulation Mechanism Standard." August 2024. https://csrc.nist.gov/pubs/fips/203/final
4. NIST FIPS 204. "Module-Lattice-Based Digital Signature Standard." August 2024. https://csrc.nist.gov/pubs/fips/204/final
5. Stebila, D., Fluhrer, S., Gueron, S. "Hybrid key exchange in TLS 1.3." IETF draft-ietf-tls-hybrid-design. https://datatracker.ietf.org/doc/draft-ietf-tls-hybrid-design/
6. Tschofenig, H., Eronen, P. "Connection Identifiers for DTLS 1.2." IETF RFC 9146, March 2022. https://datatracker.ietf.org/doc/html/rfc9146
7. Ghedini, A., Vasiliev, V. "TLS Certificate Compression." IETF RFC 8879, December 2020. https://datatracker.ietf.org/doc/html/rfc8879

## Related Articles

- [What Is Post-Quantum Cryptography](what-is-post-quantum-cryptography)
- [ML-KEM Explained: How NIST's Lattice KEM Works](ml-kem-explained)
- [Hybrid Encryption: Why Combining Old and New Crypto Is Stronger](hybrid-encryption)
- [Harvest Now, Decrypt Later](harvest-now-decrypt-later)
- [SRTP and PQC: Real-Time Voice/Video Encryption](srtp-pqc)

---

### Protect Your Data Before Q-Day Arrives

QNSQY's NIST-standardized post-quantum encryption protects files against both current and quantum-era threats.

[Try QNSQY](../../pricing.html)
