← Back to Blog Engineering

NIST ACVP Validation: 18,703 Vectors, Zero Failures

10 min read
NIST ACVP Validation: 18,703 Vectors, Zero Failures - QNSQY post-quantum encryption guide

What ACVP Actually Is

NIST publishes two things people often confuse. FIPS documents (203, 204, 205, 197, 180-4, 202, and so on) define the algorithms: how ML-KEM encapsulates, how SHA-256 absorbs blocks, how AES-GCM derives tags. ACVP, the Automated Cryptographic Validation Protocol, is what you use to prove a real implementation matches the spec, byte for byte, across thousands of edge cases. Every CMVP-validated module passes ACVP first. The vector set is open. The reference server is open. The pass/fail is mechanical.

The ACVP-Server repository at usnistgov/ACVP-Server ships the canonical test vectors for every NIST-approved algorithm. Each algorithm directory carries a prompt.json (inputs) and an expectedResults.json (the answers you must produce). If your implementation reproduces those answers, it is byte-for-byte correct against NIST's reference. If it does not, you have a bug.

What We Just Shipped

We landed a new test harness at qs-ultra/tests/acvp_official/. It replays the full ACVP-Server v1.1.0.42 vector tree, commit 15c0f3deeefbfa8cb6cd32a99e1ca3b738c66bf0, snapshot date 2026-04-16, against the exact same Rust crates QNSQY ships with: sha2, sha3, hmac, aes-gcm, ed25519-dalek, x25519-dalek, ml-kem 0.2.3, ml-dsa 0.1.0-rc.7, slh-dsa 0.2.0-rc.1.

Final tally: 18,703 vectors pass, 0 fail, 11,259 documented skip, out of 29,962 total. Non-skipped pass rate: 100.00%.

The skips are not failures. They are vectors that fall outside what the underlying Rust crates can answer. Below, we list every category and explain why.

The Three Categories of Harness Bugs We Found

When we first ran the sweep, it was a wall of red. Walking through what was actually broken was instructive.

Loader bugs. The HMAC -1.0 schema carries macLen at the testGroup level. Our first loader read it per-test-case only, so 24 algorithm directories of HMAC vectors (SHA-1, every SHA-2 width, every SHA-3 width, in both -1.0 and -2.0 schemas) defaulted to the wrong truncation length and burned thousands of tests. Adding a group-level fallback flipped roughly 10,000 reds to greens in a single commit.

Missing algorithm variants. SLH-DSA in FIPS 205 has a signatureInterface matrix: external vs internal, pure vs preHash, deterministic vs randomized, plus 12 parameter sets. The slh-dsa crate exposes both public-facing and internal entry points (slh_sign_internal / slh_verify_internal), and our harness was only wiring the public path. Wiring up the full matrix unlocked 1,248 vectors at 100% pass. SHA-2's MCT-alternate variant (the AlternateSizeShaMct.cs algorithm where seed bit-length is fixed across outer iterations and the rolling A‖B‖C buffer is trim or zero-padded to seedLength before each inner hash) was a second missing variant, accounting for thousands more.

Mismatched dev-dep types. ML-DSA's raw_sign_mu(mu, rnd) takes an Array<u8, U64> parameter built on hybrid-array. Older versions of hybrid-array in our dev-deps produced types that did not unify with the ml-dsa crate's pub(crate) B64. Pinning hybrid-array 0.4 made the externalMu path compile and unlocked the deterministic externalMu vectors in the ML-DSA matrix. We separately raised our LDT (Large Data Test) cap from 64 MiB to 8 GiB and streamed 1 MiB pre-repeated chunks into the hasher to clear the SHA-2 LDT cases.

Five harness fixes total. Together they flipped 10,611 reds to greens and unlocked 1,029 new greens for vectors we previously could not reach at all.

The Numbers, By Family

Family Vectors run Pass Fail Skip Notes
SHA-2 (FIPS 180-4) 2,575 2,575 0 0 AFT + MCT-alternate + LDT up to 8 GiB on SHA-256, SHA-512, SHA-512/256
SHA-3 / SHAKE (FIPS 202) 271 271 0 2 SHAKE-128-FIPS202 269; 2 SHAKE MCT variable-output skips
HMAC (FIPS 198-1) full full 0 0 All 22 dirs across -1.0 + -2.0 schemas, every SHA-1/2/3 variant
AES-GCM (SP 800-38D) cases 15/17/18 pass 0 30 30 tagLen<96 skips (aes-gcm 0.10 enforces §5.2.1.2 floor)
Ed25519 (RFC 8032) full pass 0 19 19 Ed25519ph + Ed448 skips (not used by QNSQY)
X25519 (RFC 7748) 50 + full pass 0 32 32 XECDH-keyVer cross-protocol skips
ML-KEM (FIPS 203) 75 + ... pass 0 60 60 encap/decap keyCheck skips (ml-kem 0.2.3 does not expose EncapsKeyCheck/DecapsKeyCheck per FIPS 203 §7)
ML-DSA (FIPS 204) 75 + 180 + ... pass 0 45 45 externalMu+randomized skips (ml-dsa raw_sign_mu is pub(crate))
SLH-DSA (FIPS 205) 1,248 1,248 0 0 All 12 parameter sets, every pure/preHash/det/random/external/internal combination
KDA-HKDF (SP 800-56C TwoStep) 0 0 0 2,000 QNSQY uses RFC 5869 HKDF, not the TwoStep KDA-HKDF variant
Bit-oriented hash inputs 0 0 0 ~5,000 sha2/sha3 crates are byte-aligned
Total 18,703 18,703 0 11,259 33 of 50 ACVP algorithm directories at 100% NIST coverage

33 of 50 ACVP algorithm directories are at full 100% coverage. The full list of 100% directories: 22 HMAC directories (every hash family, both schemas), ML-DSA-keyGen (75), ML-DSA-sigVer (180), ML-KEM-keyGen (75), SLH-DSA-keyGen + sigGen + sigVer across all 12 parameter sets (1,248), SHA-2-256/512/512-256 -1.0 (2,575), SHAKE-128-FIPS202 (269), and XECDH-SSC RFC 7748 self-consistency (50 X25519 pair tests).

Sweep wall-clock: 518.09 seconds for all 50 test functions (2026-05-15), single machine, single thread per function. The LDT cases alone hash 8 GiB inputs end to end.

Why We Skip 11,259 Vectors (Specifically)

Skipping has to be earned. Each of these has a documented reason:

  • Bit-oriented hash inputs (~5,000 vectors). ACVP defines vectors at bit granularity. The sha2 and sha3 crates accept byte-aligned input. We did not write a sub-byte feeder; production code does not need one because QNSQY never hashes a non-octet-aligned message.
  • SP 800-56C TwoStep KDA-HKDF (2,000 vectors). This is a separate KDF construction. QNSQY uses RFC 5869 HKDF directly, which has its own ACVP test vectors that we pass. TwoStep is unused.
  • AES-GCM tagLen<96 (30 vectors). The aes-gcm 0.10 crate enforces the SP 800-38D §5.2.1.2 secure floor (tags shorter than 96 bits are rejected at the type level). NIST publishes vectors below the floor for legacy testing; they are unreachable through the safe API.
  • Ed25519ph and Ed448 (19 vectors). Neither is used by QNSQY. We sign with Ed25519 only.
  • ML-DSA externalMu + randomized (45 vectors). The ml-dsa crate exposes raw_sign_mu(mu, rnd) as pub(crate), not public. We pass externalMu deterministic via the public path; the randomized variant requires an internal entry point.
  • ML-KEM encap/decap keyCheck (60 vectors). FIPS 203 §7 defines ML-KEM-EncapsKeyCheck and DecapsKeyCheck for key validation. The ml-kem 0.2.3 crate does not expose these as standalone functions.
  • XECDH-keyVer cross-protocol checks (32 vectors). These verify behavior across X25519/X448. QNSQY uses X25519 only.
  • SHAKE MCT variable-output state machine (2 vectors). Not implemented in our harness. Single-output SHAKE-128 is at 269/269.

Each skip is logged with the algorithm directory and the reason. The harness output is grep-able for anyone who wants to audit the gap.

Reproducibility

Anyone can re-run this. The vector tree is open; the crates are pinned in Cargo.toml; and we publish the harness as a standalone, self-contained archive on the Validation page, with step-by-step instructions. It depends only on public RustCrypto crates and contains no QNSQY proprietary code. From the harness directory: 

cargo build --release --tests
./target/release/deps/acvp_official-* --nocapture --test-threads 2

The vector source: github.com/usnistgov/ACVP-Server, commit 15c0f3deeefbfa8cb6cd32a99e1ca3b738c66bf0 (snapshot 2026-04-16). Inside the QNSQY source tree the same harness lives at qs-ultra/tests/acvp_official/; full source access for customers and auditors is available on request under NDA. The numbers in this post are the numbers that test run prints. If they ever change, we will report the new numbers, not paper over them.

What This Is NOT

This is reproducible byte-for-byte ACVP test-vector validation against the same Rust crates QNSQY ships with. It is not FIPS 140-3 validation. FIPS 140-3 requires CMVP, which means a NIST-accredited lab (a CSTL) writes a Security Policy, runs operational tests on the boundary, and submits the module for NIST review. CMVP is on our roadmap; ACVP is one piece of the evidence a CMVP lab would consult. The 18,703-vector pass demonstrates the algorithm implementations are correct. It does not yet prove the surrounding module meets every FIPS 140-3 boundary, role, and self-test requirement.

If a vendor tells you "we pass NIST tests," ask them which tests, against which vector commit, and where the failure log lives. That is the difference between a marketing claim and a check anyone can run.

Sources

  1. NIST ACVP Project Page
  2. usnistgov/ACVP-Server (canonical vector tree)
  3. FIPS 203: ML-KEM, August 2024
  4. FIPS 204: ML-DSA, August 2024
  5. FIPS 205: SLH-DSA, August 2024
  6. FIPS 180-4: Secure Hash Standard (SHA-2)
  7. FIPS 202: SHA-3 Standard (SHA-3, SHAKE)
  8. SP 800-38D: AES-GCM
  9. RFC 7748: X25519 / X448
  10. RFC 8032: Ed25519 / Ed448
  11. RFC 5869: HKDF
  12. RFC 4231: HMAC test cases

Related Articles

Run the Sweep Yourself

QNSQY publishes the standalone harness on the Validation page. Download it, run one cargo command, watch 18,703 vectors pass.

Try QNSQY

Originally published at quantumsequrity.com/blog/nist-acvp-validation.