LLM Steganography Demo

This demo lets you hide a secret message inside ordinary-looking AI-generated text. The output reads like a normal sentence a language model might produce, but it secretly encodes your message bit by bit.

The practical point is covert communication: two people who share a key can exchange messages that, to anyone watching, look like innocuous GPT-2 text. There's no obvious ciphertext, no encrypted file, and no metadata screaming "this is encrypted." The secret is hidden in plain sight.

It's also a neat demonstration of how much information is packed into every token a language model emits — each token can carry a full secret bit while still looking natural.

This demo uses a half-splitting entropy coding scheme built on top of GPT-2's next-token probability distribution.

1. The secret message is converted to a bit string (UTF-8 → bits).
2. For each secret bit, GPT-2 produces a probability distribution over the entire vocabulary for the next token.
3. Tokens are sorted by probability (descending) and split into two halves at the 50% cumulative probability mark.
4. Bit 0 → the next token is sampled from the first (higher-probability) half; bit 1 → from the second half.
5. A shared-key PRNG (mulberry32) selects the exact token within the chosen half, so the decoder can reproduce the same random draws.
6. The decoder re-runs GPT-2 on the same context, observes which half each cover token fell into, and recovers the bits → original message.

Because both sides share the same model, context, and PRNG seed, the decoder can perfectly reconstruct the hidden bits. The resulting cover text reads like normal GPT-2 output, hiding the secret in plain sight.

To make this tangible, let's use a toy model with a small vocabulary of 11 words so you can see every number. The real scheme works identically, just with vocabularies of ~50,000 tokens and floating-point probabilities.

Setup

The model: A tiny "LLM" with a fixed vocabulary of 11 words: {apple, date, banana, cherry, pie, juice, tart, sauce, for, with, and}. At each step, the model assigns a probability to all 11 words based on the context. The probabilities change as the context grows, and they always sum to 1.0.

Shared context/prompt: "I like to eat" — both Alice and Bob have this. It is not secret, just shared.

Shared secret key: Used to seed a PRNG, producing the random stream: 0.15, 0.62, 0.40, ...

Secret message: The bits 0 1 1 (3 bits).

Encoding (Alice's Side)

Step 1: Alice runs the model

Alice feeds "I like to eat" into the model. After "I like to eat", fruits are the most natural continuation, so they get the highest probabilities:

The probabilities sum to 1.0, and the intervals partition [0, 1).

Step 2: Alice encodes her first secret bit

Alice's first secret bit is 0 → "look in the first half [0.00, 0.50)." The words in that half are apple, date, banana, and cherry. Alice uses her next random number, 0.15, to pick within [0.00, 0.50). The value 0.15 falls in [0.00, 0.20), which is apple.

Alice outputs: "apple" → text so far: "I like to eat apple"

Step 3: Alice runs the model again

Context is now "I like to eat apple". After "apple", food preparations like pie and tart become more likely:

Alice's next bit is 1 → "look in the second half [0.50, 1.00)." Random number 0.62 → scaled: 0.50 + 0.62 × 0.50 = 0.81, which falls in [0.70, 0.86) → tart.

Alice outputs: "tart" → text so far: "I like to eat apple tart"

Step 4: Alice runs the model again

Context: "I like to eat apple tart". After "apple tart", connectors like "with" and "and" become most likely:

Alice's next bit is 1 → second half [0.50, 1.00). Random number 0.40 → scaled: 0.50 + 0.40 × 0.50 = 0.70, falls in [0.60, 0.78) → with.

Alice outputs: "with" → text so far: "I like to eat apple tart with"

Alice is done

Alice sends Bob the text: "I like to eat apple tart with" (plus whatever padding she wants to make it look like a complete sentence). To any observer, this looks like someone generated a sentence about food. Nothing suspicious.

Decoding (Bob's Side)

Bob has the received text, the same model, the same shared context "I like to eat", and the same shared key (so the same random stream: 0.15, 0.62, 0.40, ...).

Step D1: Bob runs the model

Bob feeds "I like to eat" into the model and gets the same distribution Alice got. He sees Alice chose apple, interval [0.00, 0.20). "Which half?" First half [0.00, 0.50) → bit 0 ✓

Step D2: Bob runs the model again

Context: "I like to eat apple". Same distribution as Alice's Step 3. Bob sees Alice chose tart, interval [0.70, 0.86). "Which half?" Second half [0.50, 1.00) → bit 1 ✓

Step D3: Bob runs the model again

Context: "I like to eat apple tart". Same distribution as Alice's Step 4. Bob sees Alice chose with, interval [0.60, 0.78). "Which half?" Second half [0.50, 1.00) → bit 1 ✓

Result

Bob has recovered: 0 1 1 — exactly the message Alice sent.

Key Points

1. The LLM never saw the secret message. The secret bits controlled which word was picked from the distribution. The LLM just produced distributions.
2. The output looks natural. "I like to eat apple tart with" is a perfectly normal sentence. A censor seeing this has no reason to be suspicious.
3. Both sides run the same model with the same context. This is why they get the same distributions. If Bob had a different model or different context, decoding would fail.
4. The shared key provides the random stream. Without it, a censor could run the same model and try to decode. With the key, the censor can't reproduce the random choices.
5. Each token encodes roughly 1 bit in this toy example (2-way split). With arithmetic coding over a 50,000-word vocabulary, you can encode ~2–4 bits per token.
6. The "half" splitting is the simplified version. The real scheme (from the Meteor paper) uses full arithmetic coding, which is more efficient. But the principle is the same: secret bits steer selection within the model's probability distribution.

The direct inspiration for this project is a post by waxwing on Nostr that walks through the core idea of using a language model's next-token probability distribution as a steganographic carrier channel. The post explains how secret bits can steer token selection while preserving the model's output distribution, making the resulting cover text statistically indistinguishable from normal model output.

That post references the academic paper that formalized this approach:

• Meteor: A Simple and Practical Steganographic Protocol for LLMs — this paper introduces the entropy-coding mechanism that makes the output distribution-preserving. Instead of a naive 2-way split, the full scheme uses arithmetic coding over the model's probability distribution, achieving roughly 2–4 bits per token while remaining information-theoretically undetectable to a censor who only observes the channel.

The half-splitting scheme implemented in this demo is a simplified version of Meteor's approach: it guarantees exactly one bit per token, is symmetric (encoder and decoder run the same logic), and produces text that's indistinguishable from normal model output to a casual reader. The full arithmetic-coding version from the paper is more efficient but the core principle is identical — secret bits steer selection within the model's probability distribution.