laantungir/client
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace simple example FAQ with detailed toy-model walkthrough from llm_steganography_explained.md

Browse Source
This commit is contained in:
Laan Tungir
2026-06-28 20:25:51 -04:00
parent d6b167a50b
commit c36cb4a0fe
2 changed files with 219 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
www/js/version.json
View File

@@ -1,5 +1,5 @@
{
"VERSION": "v0.7.59",
"VERSION_NUMBER": "0.7.59",
"BUILD_DATE": "2026-06-29T00:23:01.531Z"
"VERSION": "v0.7.60",
"VERSION_NUMBER": "0.7.60",
"BUILD_DATE": "2026-06-29T00:25:51.063Z"
}

242
www/llm-steganography.html
View File

@@ -333,6 +333,54 @@
margin-bottom: 4px;
}
/* ---------- Example walkthrough ---------- */
.stegoExampleH3 {
font-size: 15px;
font-weight: 700;
color: var(--accent-color);
margin: 16px 0 8px 0;
}
.stegoExampleH4 {
font-size: 14px;
font-weight: 600;
color: var(--primary-color);
margin: 14px 0 6px 0;
}
.stegoExampleOutput {
font-family: var(--font-mono, "SF Mono", "Fira Code", "Consolas", monospace);
background: var(--background-color);
padding: 8px 12px;
border-radius: 6px;
border: 1px solid var(--border-color);
color: var(--accent-color);
font-size: 13px;
margin: 8px 0;
}
.stegoTableWrap {
overflow-x: auto;
margin: 10px 0;
}
.stegoTable {
width: 100%;
border-collapse: collapse;
font-size: 12px;
font-family: var(--font-mono, "SF Mono", "Fira Code", "Consolas", monospace);
}
.stegoTable th,
.stegoTable td {
border: 1px solid var(--border-color);
padding: 4px 8px;
text-align: left;
color: var(--primary-color);
}
.stegoTable th {
background: var(--background-color);
font-weight: 600;
}
.stegoTable tr:nth-child(even) td {
background: var(--background-color);
}
.stegoFooter {
text-align: center;
color: var(--muted-color);
@@ -509,41 +557,183 @@
</button>
<div class="stegoCollapsibleContent stegoHidden">
<p>
Suppose the context is <code>"I like to eat"</code> and your secret
message is <code>"HI"</code>. The letter <code>H</code> is byte
<code>0x48</code> = bits <code>01001000</code>, and <code>I</code> is
<code>0x49</code> = bits <code>01001001</code>. So the encoder needs to
hide 16 bits total.
To make this tangible, let's use a <strong>toy model with a small
vocabulary of 11 words</strong> so you can see every number. The real
scheme works identically, just with vocabularies of ~50,000 tokens and
floating-point probabilities.
</p>
<h3 class="stegoExampleH3">Setup</h3>
<p>
<strong>The model:</strong> A tiny "LLM" with a fixed vocabulary of 11
words: <code>{apple, date, banana, cherry, pie, juice, tart, sauce, for, with, and}</code>.
At each step, the model assigns a probability to all 11 words based on
the context. The probabilities change as the context grows, and they
always sum to 1.0.
</p>
<p>
For the <strong>first bit (0)</strong>, GPT-2 looks at
<code>"I like to eat"</code> and ranks every possible next token by
probability. It splits that ranked list at the 50% cumulative-probability
mark into a "high-prob" half and a "low-prob" half. Since the bit is
<code>0</code>, the encoder samples a token from the <em>high-prob</em>
half — say <code>" pizza"</code>. The cover text is now
<code>"I like to eat pizza"</code>.
<strong>Shared context/prompt:</strong> <code>"I like to eat"</code> —
both Alice and Bob have this. It is not secret, just shared.
</p>
<p>
For the <strong>second bit (1)</strong>, GPT-2 now looks at
<code>"I like to eat pizza"</code> and produces a fresh distribution.
The bit is <code>1</code>, so the encoder samples from the
<em>low-prob</em> half — maybe <code>" and"</code>. Cover text:
<code>"I like to eat pizza and"</code>.
<strong>Shared secret key:</strong> Used to seed a PRNG, producing the
random stream: <code>0.15, 0.62, 0.40, ...</code>
</p>
<p>
This continues for all 16 bits, then a few padding tokens are added so
the text ends naturally. The final cover text might read something like:
</p>
<p style="font-family: var(--font-mono, monospace); background: var(--background-color); padding: 8px 12px; border-radius: 6px; border: 1px solid var(--border-color);">
"I like to eat pizza and pasta with my friends on weekends."
<strong>Secret message:</strong> The bits <code>0 1 1</code> (3 bits).
</p>
<h3 class="stegoExampleH3">Encoding (Alice's Side)</h3>
<h4 class="stegoExampleH4">Step 1: Alice runs the model</h4>
<p>
To anyone else, that's just a normal sentence. But the decoder — who
knows the same context, key, and model — re-runs GPT-2 at each step,
checks which half each token fell into, and recovers the bits
<code>01001000 01001001</code> → <code>"HI"</code>.
Alice feeds <code>"I like to eat"</code> into the model. After "I like
to eat", fruits are the most natural continuation, so they get the
highest probabilities:
</p>
<div class="stegoTableWrap">
<table class="stegoTable">
<thead><tr><th>Word</th><th>Probability</th><th>Interval</th></tr></thead>
<tbody>
<tr><td>apple</td><td>0.20</td><td>[0.00, 0.20)</td></tr>
<tr><td>date</td><td>0.15</td><td>[0.20, 0.35)</td></tr>
<tr><td>banana</td><td>0.10</td><td>[0.35, 0.45)</td></tr>
<tr><td>cherry</td><td>0.05</td><td>[0.45, 0.50)</td></tr>
<tr><td>pie</td><td>0.20</td><td>[0.50, 0.70)</td></tr>
<tr><td>juice</td><td>0.12</td><td>[0.70, 0.82)</td></tr>
<tr><td>tart</td><td>0.08</td><td>[0.82, 0.90)</td></tr>
<tr><td>sauce</td><td>0.05</td><td>[0.90, 0.95)</td></tr>
<tr><td>for</td><td>0.03</td><td>[0.95, 0.98)</td></tr>
<tr><td>with</td><td>0.015</td><td>[0.98, 0.995)</td></tr>
<tr><td>and</td><td>0.005</td><td>[0.995, 1.00)</td></tr>
</tbody>
</table>
</div>
<p>The probabilities sum to 1.0, and the intervals partition [0, 1).</p>
<h4 class="stegoExampleH4">Step 2: Alice encodes her first secret bit</h4>
<p>
Alice's first secret bit is <code>0</code> → "look in the <strong>first
half</strong> [0.00, 0.50)." The words in that half are apple, date,
banana, and cherry. Alice uses her next random number, <code>0.15</code>,
to pick within [0.00, 0.50). The value <code>0.15</code> falls in
[0.00, 0.20), which is <strong>apple</strong>.
</p>
<p class="stegoExampleOutput">Alice outputs: "apple" → text so far: "I like to eat apple"</p>
<h4 class="stegoExampleH4">Step 3: Alice runs the model again</h4>
<p>
Context is now <code>"I like to eat apple"</code>. After "apple", food
preparations like pie and tart become more likely:
</p>
<div class="stegoTableWrap">
<table class="stegoTable">
<thead><tr><th>Word</th><th>Probability</th><th>Interval</th></tr></thead>
<tbody>
<tr><td>apple</td><td>0.15</td><td>[0.00, 0.15)</td></tr>
<tr><td>date</td><td>0.10</td><td>[0.15, 0.25)</td></tr>
<tr><td>banana</td><td>0.08</td><td>[0.25, 0.33)</td></tr>
<tr><td>cherry</td><td>0.07</td><td>[0.33, 0.40)</td></tr>
<tr><td>for</td><td>0.06</td><td>[0.40, 0.46)</td></tr>
<tr><td>with</td><td>0.04</td><td>[0.46, 0.50)</td></tr>
<tr><td>pie</td><td>0.20</td><td>[0.50, 0.70)</td></tr>
<tr><td>tart</td><td>0.16</td><td>[0.70, 0.86)</td></tr>
<tr><td>juice</td><td>0.08</td><td>[0.86, 0.94)</td></tr>
<tr><td>sauce</td><td>0.04</td><td>[0.94, 0.98)</td></tr>
<tr><td>and</td><td>0.02</td><td>[0.98, 1.00)</td></tr>
</tbody>
</table>
</div>
<p>
Alice's next bit is <code>1</code> → "look in the <strong>second half</strong>
[0.50, 1.00)." Random number <code>0.62</code> → scaled:
<code>0.50 + 0.62 × 0.50 = 0.81</code>, which falls in [0.70, 0.86) →
<strong>tart</strong>.
</p>
<p class="stegoExampleOutput">Alice outputs: "tart" → text so far: "I like to eat apple tart"</p>
<h4 class="stegoExampleH4">Step 4: Alice runs the model again</h4>
<p>
Context: <code>"I like to eat apple tart"</code>. After "apple tart",
connectors like "with" and "and" become most likely:
</p>
<div class="stegoTableWrap">
<table class="stegoTable">
<thead><tr><th>Word</th><th>Probability</th><th>Interval</th></tr></thead>
<tbody>
<tr><td>apple</td><td>0.10</td><td>[0.00, 0.10)</td></tr>
<tr><td>date</td><td>0.08</td><td>[0.10, 0.18)</td></tr>
<tr><td>banana</td><td>0.07</td><td>[0.18, 0.25)</td></tr>
<tr><td>cherry</td><td>0.06</td><td>[0.25, 0.31)</td></tr>
<tr><td>pie</td><td>0.08</td><td>[0.31, 0.39)</td></tr>
<tr><td>juice</td><td>0.06</td><td>[0.39, 0.45)</td></tr>
<tr><td>for</td><td>0.05</td><td>[0.45, 0.50)</td></tr>
<tr><td>tart</td><td>0.06</td><td>[0.50, 0.56)</td></tr>
<tr><td>sauce</td><td>0.04</td><td>[0.56, 0.60)</td></tr>
<tr><td>with</td><td>0.18</td><td>[0.60, 0.78)</td></tr>
<tr><td>and</td><td>0.22</td><td>[0.78, 1.00)</td></tr>
</tbody>
</table>
</div>
<p>
Alice's next bit is <code>1</code> → second half [0.50, 1.00). Random
number <code>0.40</code> → scaled: <code>0.50 + 0.40 × 0.50 = 0.70</code>,
falls in [0.60, 0.78) → <strong>with</strong>.
</p>
<p class="stegoExampleOutput">Alice outputs: "with" → text so far: "I like to eat apple tart with"</p>
<h4 class="stegoExampleH4">Alice is done</h4>
<p>
Alice sends Bob the text: <strong>"I like to eat apple tart with"</strong>
(plus whatever padding she wants to make it look like a complete
sentence). To any observer, this looks like someone generated a
sentence about food. Nothing suspicious.
</p>
<h3 class="stegoExampleH3">Decoding (Bob's Side)</h3>
<p>
Bob has the received text, the same model, the same shared context
<code>"I like to eat"</code>, and the same shared key (so the same
random stream: <code>0.15, 0.62, 0.40, ...</code>).
</p>
<h4 class="stegoExampleH4">Step D1: Bob runs the model</h4>
<p>
Bob feeds <code>"I like to eat"</code> into the model and gets the
<strong>same</strong> distribution Alice got. He sees Alice chose
<strong>apple</strong>, interval [0.00, 0.20). "Which half?" First half
[0.00, 0.50) → <strong>bit <code>0</code></strong> ✓
</p>
<h4 class="stegoExampleH4">Step D2: Bob runs the model again</h4>
<p>
Context: <code>"I like to eat apple"</code>. Same distribution as Alice's
Step 3. Bob sees Alice chose <strong>tart</strong>, interval [0.70, 0.86).
"Which half?" Second half [0.50, 1.00) → <strong>bit <code>1</code></strong> ✓
</p>
<h4 class="stegoExampleH4">Step D3: Bob runs the model again</h4>
<p>
Context: <code>"I like to eat apple tart"</code>. Same distribution as
Alice's Step 4. Bob sees Alice chose <strong>with</strong>, interval
[0.60, 0.78). "Which half?" Second half [0.50, 1.00) →
<strong>bit <code>1</code></strong> ✓
</p>
<h3 class="stegoExampleH3">Result</h3>
<p>
Bob has recovered: <code>0 1 1</code> — exactly the message Alice sent.
</p>
<h3 class="stegoExampleH3">Key Points</h3>
<ol>
<li><strong>The LLM never saw the secret message.</strong> The secret bits controlled which word was picked from the distribution. The LLM just produced distributions.</li>
<li><strong>The output looks natural.</strong> "I like to eat apple tart with" is a perfectly normal sentence. A censor seeing this has no reason to be suspicious.</li>
<li><strong>Both sides run the same model with the same context.</strong> This is why they get the same distributions. If Bob had a different model or different context, decoding would fail.</li>
<li><strong>The shared key provides the random stream.</strong> Without it, a censor could run the same model and try to decode. With the key, the censor can't reproduce the random choices.</li>
<li><strong>Each token encodes roughly 1 bit</strong> in this toy example (2-way split). With arithmetic coding over a 50,000-word vocabulary, you can encode ~2–4 bits per token.</li>
<li><strong>The "half" splitting is the simplified version.</strong> The real scheme (from the <a href="https://eprint.iacr.org/2021/686" target="_blank" rel="noopener">Meteor paper</a>) uses full arithmetic coding, which is more efficient. But the principle is the same: secret bits steer selection within the model's probability distribution.</li>
</ol>
</div>
</div>