amethyst/.claude/skills/event-store-semantics/SKILL.md

---
name: event-store-semantics
description: Quartz IEventStore/SQLite store contract (STORE-* rules) and NIP-50 search: filter semantics, replaceable/deletion/expiry writes, counts, FTS, which kinds are SearchableEvent. Use for store parity, geode, or making a kind searchable.
---

# Event store semantics and NIP-50 search

The SQLite `EventStore` (`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/`)
is the reference implementation of a Quartz event store. The filesystem store
(`quartz/src/jvmMain/.../nip01Core/store/fs/`, held to parity by
`quartz/src/jvmTest/.../store/fs/FsParityTest.kt`), geode and external engines (e.g. a
Vespa-backed store) reimplement its observable behavior. Every behavior is a numbered rule
in **`references/store-rules.md`** so a divergence is a lookup. When you change store
behavior, update that rule and its changelog in the same PR.

## Key files (under `nip01Core/store/`)

| Concern | File |
|---|---|
| Public contract (KDoc is normative) | `IEventStore.kt`, `RejectionReason.kt` |
| High-level store (pool + planner) | `sqlite/EventStore.kt`, `sqlite/SQLiteEventStore.kt` |
| Filter → SQL, ordering, limits, counts | `sqlite/QueryBuilder.kt` |
| k-way merge fast path (feed shapes) | `sqlite/MergeQueryExecutor.kt` |
| Schema, tag hashing, immutability | `sqlite/EventIndexesModule.kt`, `sqlite/TagNameValueHasher.kt`, `sqlite/SeedModule.kt` |
| Replaceable / addressable supersession | `sqlite/ReplaceableModule.kt`, `sqlite/AddressableModule.kt` |
| NIP-09 / NIP-40 / NIP-62 / ephemeral | `sqlite/DeletionRequestModule.kt`, `sqlite/ExpirationModule.kt`, `sqlite/RightToVanishModule.kt`, `sqlite/EphemeralModule.kt` |
| NIP-50 FTS | `sqlite/FullTextSearchModule.kt` |
| Index / feature toggles | `sqlite/IndexingStrategy.kt` (client default); geode's `geode/src/main/kotlin/com/vitorpamplona/geode/RelayIndexingStrategy.kt` (relay preset) |
| Operations (concurrency, pragmas, maintenance) | `sqlite/README.md` |

Executable spec: `quartz/src/commonTest/.../nip01Core/store/sqlite/` (`BasicTest`,
`ReplaceableTest`, `AddressableTest`, `DeletionTest`, `ExpirationTest`, `RightToVanishTest`,
`SearchTest`, `SearchRelevanceOrderTest`, `NoFullTextSearchTest`, `MergeQueryCorrectnessTest`,
`TagMergeCorrectnessTest`, `QueryAssemblerTest`, `SnapshotIdsForNegentropyTest`,
`InsertOutcomeClassificationTest`) plus `nip01Core/relay/filters/FilterMatcherTest`. A test
beats a rule; fix the rule.

## Rule index (full text in `references/store-rules.md`)

- **F01** `since`/`until` inclusive · **F02** ids/authors exact 64-hex only, no prefix match ·
  **F03** tag values OR within a name, AND across names; `tagsAll` = every value ·
  **F04** only single ASCII-letter tag names indexed (`isIndexableTagName`) ·
  **F05** `#d` matched on `event_headers.d_tag`, use it only via plain `tags` ·
  **F06** tag matching is 64-bit hash, no post-verify · **F07** multi-filter = UNION, limit
  per filter, no global limit · **F08** `created_at DESC`; same-second order unspecified
  unless `useAndIndexIdOnOrderBy` · **F09** merge fast path returns the same set ·
  **F10** empty filter matches all, but `delete(Filter())` deletes nothing ·
  **F11** empty lists are invalid input · **F12** `limit = 0` → none · **F13**
  `FilterMatcher` is separate and simpler.
- **W01/W02** replaceable/addressable: newest wins, tie → lowest id, stale version rejected
  `REPLACED` (`OK false`) · **W03** ephemeral acked, never stored · **W04** expired rejected
  at insert · **W05** expiry not enforced at query time (sweep) · **W06** gift-wrap owner =
  `p` recipient · **W07** rows immutable · **W08** batch: savepoint per row · **W09** failures
  classified by querying the DB, not driver text.
- **D01** kind-5 by id, same owner, no time check · **D02** by address, `created_at <=` ·
  **D03** cross-author kind-5 inert · **D04** re-insert blocking · **D05** kind-5 can delete a
  kind-5 (known quirk) · **D06/D07** NIP-62 vanish relay-scoped, strict delete / inclusive block
  · **D08** manual deletes create no blocking.
- **C01** count = deduped match set honoring per-filter limits.
- **S01** search extensions stripped before FTS · **S02** bm25 relevance order · **S03** search
  ANDs with structure · **S04** FTS5 `MATCH` grammar · **S05** FTS off → search matches nothing
  · **S06** deferred FTS catches up before a search.
- **N01** negentropy snapshots use query semantics.

Kind classes everywhere: replaceable = 0, 3, 10000–19999; ephemeral = 20000–29999;
addressable = 30000–39999.

## Searchable events (NIP-50 indexing surface)

`nip50Search/SearchableEvent.kt`:

```kotlin
interface SearchableEvent {
    fun indexableContent(): String                                   // write path: stores call it once per insert
    fun forEachIndexableField(visitor: IndexableFieldVisitor) { … }  // read path: allocation-free cache scans (EventSearchMatcher)
    fun indexableSeparator(): String = "\n"                          // what indexableContent joins with
}
```

A kind is searchable iff its class implements this **and** is wired into `EventFactory` (the
stores probe by kind). The set is `nip50Search/SearchableKinds.kt` `ALL` — 201 kinds today,
one row each in **`references/searchable-kinds.md`** with the exact `indexableContent()`
expression.

Implementation rules:

- Plain text out: human fields joined with `"\n"` (a few metadata kinds use `" "` and override
  `indexableSeparator`). JSON-content kinds (kind 0, stalls, channel info) parse first and join
  the fields, never the raw JSON. Markdown goes in raw.
- Never throw, never null — a throw aborts the insert transaction. Use `?.let { … } ?: ""`.
- Public data only: never index NIP-44 payloads or private tags.
- Kinds that join several fields should override `forEachIndexableField` to visit the same
  fields in the same order (the golden test rejoins them and compares).
- Eligibility: human-authored text (titles, bodies, names, descriptions). Reactions, follow
  lists, relay lists and zaps (minus the receipt's comment) stay out.

**MANDATORY when you add/remove `SearchableEvent` or change an `indexableContent()` body:**

1. Update `references/searchable-kinds.md` in the same PR — external search engines mirror it
   at pin bumps.
2. List the kind in `SearchableKinds.ALL` (`SearchableKindsTest` sweeps kinds 0–65535 through
   `EventFactory` and fails otherwise) and register the class in `EventFactory`, or
   `FullTextSearchModule.isSearchableKind` and the reindex scan never see it.
3. Regenerate `quartz/src/jvmTest/resources/indexable-content.golden`
   (`IndexableContentGoldenTest`).
4. Existing databases don't reindex themselves: old rows keep their text until
   `IEventStore.reindexFullTextSearch()` runs (its KDoc is the contract); app-side, schedule the
   resumable overload after shipping.

Surprises: kind 9735 indexes the embedded zap request's comment; kinds 0/31990 index many
profile fields space-joined; kind 30063 indexes release notes only for NIP-82 releases (NIP-51
`content` may be encrypted); poll kinds append each option label.

## How the stores index

- **SQLite** (`FullTextSearchModule.kt`): contentless FTS5 table `event_fts`, `rowid` =
  `event_headers.row_id`, kept in sync by a delete trigger; default `unicode61` tokenizer;
  queries are always a bound `event_fts MATCH ?`, ordered by bm25 `rank` then `created_at DESC`
  (rules S01–S06).
- **Filesystem** (`jvmMain/.../store/fs/FsIndexer.kt`, `FsSearchTokenizer.kt`): tokenizes
  `indexableContent()` itself, approximating `unicode61`; the same tokenizer runs on queries.

## NIP-50 client side

- `nip50Search/SearchQuery.kt` — `SearchQuery.parse(search)` splits `terms` from
  `extensions`. A token is an extension iff it looks like `lowercasekey:value` with the value not
  starting with `//` (URLs stay text); duplicate keys keep the last; unknown keys are kept
  (`extension(key)`). Typed: `includeSpam`, `domain`, `language`, `sentiment`, `nsfw`.
  `stripExtensions()` / `Filter.strippingSearchExtensions()` is what the built-in stores apply:
  unsupported extensions are ignored, so an extensions-only search is unconstrained. A store
  that implements extensions gets the raw string and should parse it with `SearchQuery.parse`.
- `SearchRelayListEvent` (`nip50Search/SearchRelayListEvent.kt`) — **kind 10007**, the user's
  search relays (public + NIP-44 private tags; not itself searchable). Used by
  `commons/.../commons/actions/SearchActions.kt`; defaults in
  `commons/.../commons/account/AccountBootstrapEvents.kt`.
- Not the same thing: `commons/.../commons/search/SearchQuery.kt` is an app-level local-feed
  query model.

Search tests: `commonTest/.../nip50Search/SearchQueryTest.kt`, `SearchableKindsTest.kt`,
`jvmTest/.../nip50Search/IndexableContentGoldenTest.kt`, the sqlite `SearchTest` /
`SearchRelevanceOrderTest` / `NoFullTextSearchTest`, and `jvmTest/.../store/fs/FsSearchTest.kt`.

## For parity implementers

Cite rule ids in divergence notes ("diverges from STORE-D05: we no-op deletion-of-a-deletion").
F06 (hash-based tags) and F08 (unordered same-second ties) are where the reference itself has
slack, so a byte-for-byte diff there may not be your bug.