mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-09-14 04:45:07 +00:00
master
275
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DanConwayDev
|
b990d2189b |
docs(maintainers): point to shared protocol guide
The maintainer implementation guide moved out of the ngit client repository so implementations need a stable cross-project authority. Link the canonical ngit-docs source and published guide from agent guidance and the relay architecture, and update the repository-state decision to name that guide as its model. This assumes the source and published locations recorded by ngit's move commit are canonical. Runtime behavior and the existing relay-specific model description are deliberately unchanged. Validated with git diff --cached --check and a repository-wide stale-reference scan. |
||
|
DanConwayDev
|
3f8157693f |
docs: point GRASP references at Nostr Git
Move current GRASP specification links from GitHub to GitWorkshop. Link audit output to its exact pinned specification commit. Use NIP-05 cloning for ngit-grasp and ngit.dev for the ngit homepage. Leave the archived migration link unchanged as a historical record. Validated with rustfmt and the 54-test grasp-audit library suite. |
||
|
DanConwayDev
|
fd7c6bb851 |
refactor(auth): make current maintainer authority explicit
The v3.0.1 authorization fix is intentionally small. Follow it with a separate structural pass so the implementation and documentation express the present-tense maintainer model directly instead of leaving the security behavior hidden behind owner-oriented names and repeated raw-tag interpretation.
Parse indexed roles once into a current-only snapshot of active maintainers, active lead targets, and announcement-author activity. Preserve detailed lead-resolution failures internally while policy callers continue to fail closed, distinguish selected authorization coordinates from physical owner views, and name broad announcement admission as discovery rather than authority.
Keep history relevant only while deriving current activity and retain active leads only for selected-coordinate resolution. Preserve the v3.0 public API through compatibility projections and deprecated aliases; this commit is not intended to change the authorization outcome established by
|
||
|
DanConwayDev
|
650cfb57ee |
fix(auth): root state validity at active lead
The v3 role-tag work filtered invited and ended listed maintainers, but every announcement author still seeded a repository view. A removed forwarding maintainer or an author with an ended self-role could therefore keep publishing authoritative kind 30618 state through their own coordinate. Parity-only role parsing could also treat malformed histories as active. Validate numeric role boundaries and the final-end-only defer sentinel, resolve each selected coordinate through one valid active M path to a terminal self-M, and only then seed the reciprocal membership fixpoint. Legacy and explicitly leadless views continue to root at the selected author while that author remains active; incomplete, ambiguous, and cyclic explicit paths fail closed. This changes current state and maintainer-scoped authorization only. Historical event authorization and membership mutation workflows remain outside the relay-side patch. Validated with cargo test --lib (902 tests), cargo test --test state_authorization (53 tests), and cargo clippy --tests -- -D warnings. |
||
|
DanConwayDev
|
8698c10102 |
Merge #66c9386e: Add portable deployment paths and managed hosting temp…
Add portable deployment paths and managed hosting templates nostr:nevent1qqsxdjfcdmr88n4pe79h4st43kvkwzssekuwepmsl7dt3ze2x0a27scpz3mhxue69uhhyetvv9ujumn8d96zuer9wc5fxfyx PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 PR description: Define one durable single-writer deployment contract, then implement it with a non-root Docker image, Compose and Caddy, a generic systemd unit, and separate NixOS, Linux, and Proxmox guides. Add Railway, Render, and Fly.io templates that share the same /data layout, plus bounded public verification and a container replacement test that proves the relay identity survives. The dedicated container e2e workflow expects the ngit-ci KVM runner to opt in to exposing its disposable guest Docker socket; it fails clearly if that prerequisite is absent. |
||
|
DanConwayDev
|
527bfeced5 |
docs(storage): distinguish legacy incomplete histories
Motivation: Production v3 rehearsal found unmarked missing ancestors that the previous shallow-fetch wording could misattribute to a narrow pre-release bug. Approach: Describe incomplete legacy object graphs without inferring provenance, retain the shallow marker as a specific historical case, and document a minimal maintainer-bundle recovery procedure. Remove stale test comments claiming current purgatory fetches are shallow. Correctness: Current fetches have no depth limit, migration logged the affected graphs as pre-existing, and an unmarked missing parent is not evidence of the marked January fallback behavior. Excluded scope: No integrity policy, object import automation, migration behavior, or production data is changed. Validation: Ran git diff --check and reviewed the production sweep evidence; tests were intentionally skipped because this commit changes documentation and comments only. |
||
|
DanConwayDev
|
22bbd48537 |
docs(deploy): add host-specific production paths
The production guide was NixOS-only despite presenting itself as the general deployment entry point, and its examples referenced an unavailable GitHub source and a hardening control the module does not set. Turn the entry point into an environment chooser, preserve the corrected NixOS material in its own guide, add a hardened generic systemd unit and repeatable Linux installation, document the preferred unprivileged Proxmox layout, and update repository navigation and architecture references. Each path assumes the shared deployment contract from the container change. Kubernetes automation, remote host mutation, and changes to the existing NixOS module are deliberately excluded. Validated the canonical Git remote with git ls-remote, parsed and scored the systemd unit with systemd-analyze, checked all new deployment-guide links, removed trailing whitespace, scanned the staged diff for key-shaped nsec values, and ran git diff --check. |
||
|
DanConwayDev
|
cbe48d9ce2 |
docs: define owner-view PR ref authority
Motivation:
The authorization-integrity implementation now recognizes exact standard clone endpoints as an alternative owner-view authority, but the v3 changelog and migration architecture still described only maintainer-overlap propagation.
Approach:
Document both accepted owner-view paths and explicitly state that foreign services, different coordinates, URL suffixes, and /prs/ endpoints do not satisfy the standard endpoint rule. Preserve the existing GRASP-06 warning and operator requirement to inspect manual findings.
Correctness assumptions:
The accepted event database remains authoritative and GRASP-06 contributor views continue to use their separate signer plus exact /prs/ coordinate predicate.
Excluded scope:
This changes no runtime behavior, deployment pin, release tag, or production repository data.
Validation:
git diff --check; review against the remotely built and production-validated
|
||
|
DanConwayDev
|
5d2ba5462e |
feat(security): support scoped integrity validation
Motivation: Production release-candidate validation must be able to exercise the new storage and event-authorization checker on selected identifier families without immediately sweeping thousands of repositories. The existing manual command also covered storage only, which made its name and operator workflow misleading. Approach: Apply one validated startup identifier scope to both background passes, with an empty scope retaining the secure all-family default and unmatched names counted as failures. Extend durable manual requests so check-only mode compares refs without mutation and --repair applies the same safe authorization reconciliation after storage repair. Expose the scope consistently through CLI/env, the NixOS module, examples, operator docs, architecture notes, and the v3 security warning. Correctness assumptions: Accepted State, PR, and PR Update events remain authoritative, active precisely-scoped purgatory entries remain valid in-flight exceptions, and unexplained PR refs remain preserved for manual inspection. A scoped pass proves only the named identifiers; full v3 assurance still requires removing the scope and completing the default sweep. Excluded scope: This does not tag v3, alter migration behavior, update the production deployment, or delete unexplained refs. It also does not make the manual request synchronous; the live worker continues to consume durable requests. Validation: - cargo clippy --all-targets --locked -- -D warnings - cargo test --lib --locked (895 passed) - focused scoped-selection and non-mutating reconciliation tests - resource-safe NixOS module evaluation of startupIntegrityIdentifiers |
||
|
DanConwayDev
|
cc5a3cbb0e |
fix(security): reconcile served refs with authorized events
Motivation: the v3 storage migration preserves legacy refs exactly, so structural Git integrity alone cannot detect refs written through the pre-v3 GRASP-06 path traversal. Operators need an online, post-migration answer without extending the production outage. Approach: add a second startup pass that derives owner refs from the NIP-01-preferred State of the confirmed maintainer set and PR refs from accepted PR/PR Update events, including exact GRASP-06 and active-purgatory scoping. Fetch missing expected objects through the hardened repair path, repair unambiguous drift, and preserve unexplained refs with bounded manual-inspection logs. Correctness: authoritative events are refreshed while holding the family lease, ref updates use compare-and-swap, and anything changed since the initial online snapshot is left untouched. This assumes the accepted event database and existing membership/GRASP-06 predicates are authoritative. The offline migration is deliberately unchanged, and unexplained PR refs are not auto-deleted because they may be evidence. Validation: cargo fmt --all -- --check; cargo clippy --workspace --all-targets -- -D warnings; cargo test --locked; cargo test --lib --locked (892 passed after the final race guard). |
||
|
DanConwayDev
|
c0a023c7f2 |
fix(sync): keep maintainer repository inboxes historical
The earlier commit in this PR made every accepted repository owner's and declared maintainer's read/unmarked inboxes ordinary persistent live repository sources. Combined with unbounded per-author relay lists this drove the soak's connection growth: merely owning a repository let an author's advertised inventory imply long-lived live subscriptions. The 2026-08-19 investigation showed one bot-owned repository contributing 983 relay URLs through exactly this path. Owner/maintainer repository scope is now historical-only. The live inbox_repositories tier and its merge into derive_targets are removed entirely, and the repository-scoped overlay moves onto the existing paced, byte-bounded mailbox history workers using each author's bounded, sanitized read/unmarked inbox selection. Repository coordinates and known roots are still probed there, so a root or status stored only on a maintainer mailbox is still discovered - on the recurring history cycle instead of a persistent subscription. Correctness assumptions: inbox_roots is the pre-existing root-author live tier and is deliberately untouched, so accepted root authors keep live coverage on their own bounded inboxes. Participant mailboxes were already history-only. public_repository_mailbox_scope still empties repository scope in private mode before any overlay is built, and the per-relay independent mailbox worker behavior is unchanged. Excluded scope: no changes to instance-own-relay or repository-advertised relay coverage, no event-triggered drains, negentropy routing, or scheduling changes; those remain follow-up work. Validation: cargo fmt, cargo clippy --all-targets (clean), cargo test --lib (884 passed), cargo test --test sync proactive_sync (4 passed; the owner scenario now proves the inbox is probed through the history path, that a later root still arrives via historical refresh, and that the inbox never enters ordinary live sync). |
||
|
DanConwayDev
|
9347394a2e |
fix(sync): bound canonical NIP-65 relay selections
One hostile kind 10002 list could previously contribute an unbounded number of mailbox targets, and extraction collapsed relays into HashSet<String> of raw RelayUrl::to_string() output. Raw trailing-root-slash keys never matched the canonical connection map, so those relays occupied inventory while their mailbox probes deferred forever (2026-08-19 investigation, sections 4-5 and 8: 389 of 1,648 unique corpus strings were stalled root-slash variants, and one bot author supplied 983 relay URLs). Extraction now preserves published tag order, canonicalizes each accepted URL with the existing canonical_relay_key semantics, deduplicates by canonical key (widening disagreeing duplicate markers to unmarked, preserving NIP-65 union semantics from a single slot), and takes at most four relays per author and purpose: read/unmarked for live inbox coverage, write/unmarked for identity refresh, and write/unmarked preferred with read-only backfill for participant history. The ceiling is a hard internal constant, not configuration: NIP-65 already guides users toward two to four relays per category and this is a defensive resource boundary. Correctness assumptions: every downstream mailbox key space (author_inboxes, author_mailboxes, overlays, probe scheduling, cursor and due-time maps) is populated exclusively from these extraction functions plus the already canonical operator fallback set, so canonicalizing at the extraction chokepoint makes every key space consistent with the canonical connections map. Selections stay deterministic because hygiene, canonicalization and dedup all run before the four-slot cut in published order. Excluded scope: no shuffled rotation, liveness weighting, NIP-66 ranking, host-level identity merging, whole-list rejection, or operator setting. Owner/maintainer live-versus-historical placement is unchanged here and is addressed by the next commit. Validation: cargo fmt, cargo clippy --all-targets (clean), cargo test --lib (885 passed), cargo test --test sync proactive_sync (4 passed, including the root-author scenario now advertising its inbox in trailing-root-slash form to prove the previously stalled variant reaches its connection end to end). |
||
|
DanConwayDev
|
7b49af2933 |
fix(sync): drain mailbox history per relay
The global mailbox rotation made productive relays wait behind the full discovered relay inventory between every filter group. Fast relays therefore took hours to traverse a cycle even when each individual request completed in seconds. Keep one worker per relay and immediately continue that relay after a successful partial cycle. Admit new relay workers one per maintenance pass and retain a 32-worker process-wide safety ceiling so peer-controlled NIP-65 inventories cannot create unbounded response accumulation. Failure backoff, completed-cycle refresh, request pacing, pagination, subscription-ledger accounting, persistence policy, and connection retirement remain unchanged. This deliberately does not add physical connection sharding or reduce mailbox discovery scope. Validated with cargo fmt --check, cargo clippy --all-targets --all-features -- -D warnings, the full cargo test workspace suite, focused owner-inbox live-sync coverage, and git diff --check. |
||
|
DanConwayDev
|
9d68414912 |
fix(sync): keep repository inboxes live
Repository owner and declared-maintainer read inboxes are authoritative sources for roots that may never reach a repository relay. Serving them only through a periodic history rotation leaves newly published roots and root-only status events unnecessarily stale. Map each accepted public Full repository into its owners and maintainers read or unmarked NIP-65 inbox targets, and merge every locally known repository root into the same target. The ordinary GRASP-02 target pipeline therefore keeps canonical repository and root filters in EssentialCore coverage, while compatibility variants remain ahead of recursive descendant fan-out. Write-only outboxes and non-root participant mailboxes retain the bounded history probe. This assumes NIP-65 read and unmarked relays are suitable inbox sources. Private mode continues to suppress repository coordinates, normal admission and persistent deletion tombstones remain authoritative, and this commit deliberately does not add permanent participant or write-only mailbox subscriptions or introduce cross-relay precedence over root-author inboxes. Validated with the full 819-test library suite, every Sync+ integration scenario, the complete sync integration binary (102 passed, one ignored), cargo fmt --check, and Clippy across all targets and features with warnings denied. Two unrelated timing tests failed once under the full workspace load, then passed individually and in the complete normal-parallel sync binary rerun. |
||
|
DanConwayDev
|
39149f93f8 |
fix(sync): backfill repository mailbox history
Historical repository roots and root-only descendants can exist on an accepted owner or maintainer's NIP-65 mailbox without appearing on a declared repository relay. Root-participant provenance alone cannot discover that first root or query a known root against an unrelated maintainer mailbox. Associate accepted Full repositories with their owners and declared maintainers, derive a repository-scoped history overlay from retained relay lists, and add every known root in that repository to the mailbox rotation. Reuse the existing byte-bounded, paced, single-flight fetch path and reschedule when root inventory grows. Correctness depends on repository admission remaining authoritative and on every fetched event continuing through the normal write policy, persistence pipeline, and global NIP-09/NIP-62 tombstones. Participant-only mailboxes retain exact root provenance. Deliberately excluded are permanent maintainer live subscriptions, repository-coordinate expansion in private mode, and any change to deletion or retention behavior. Validated with nix develop -c cargo test, nix develop -c cargo fmt -- --check, and nix develop -c cargo clippy --all-targets --all-features -- -D warnings. |
||
|
DanConwayDev
|
5d33b5498a |
Merge #e7f970af: feat(nip34): reciprocal maintainer membership with ind…
feat(nip34): reciprocal maintainer membership with indexed M/m role tags nostr:nevent1qqsw07ts4uns9j2squ9tcs3v7wwjm4ca7yqfsl907erjna0nenl9tvgpz3mhxue69uhhyetvv9ujumn8d96zuer9wc9x7rz5 PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 CoverNote: Implements the refined NIP-34 maintainers model (nips `e0edeef`) as three layered commits, replacing the earlier single-commit revision: 1. **Reciprocal membership (deprecated `maintainers` tag).** A listed pubkey is only *invited* until their own announcement for the same identifier lists back a confirmed maintainer. Invited maintainers' state events are no longer authorized, but their announcements are still fetched, accepted and synced — that is how the relay notices acceptance — and an acceptance re-applies stored state events (`reapply_stored`) so the owner's repository re-points without another push. 2. **Indexed `M`/`m` role tags.** Active role-tag entries are the primary maintainer listing and override the deprecated `maintainers` tag. The lead/co-maintainer distinction deliberately carries no meaning for this service. Ended entries (four-plus, even element count) are ignored entirely: role history only ever concludes that a pubkey is *no longer* a maintainer, never grants retroactive authority. Role-less announcements with a `u` tag (subordinate fork) cannot acknowledge an invitation. 3. **Moderator role (`o`) documented as deliberately unhandled.** Moderators only affect status-event authority, which this relay does not enforce; docs-only commit. Each commit builds and passes its relevant suites independently; the full workspace suite passes (two pre-existing load flakes pass in isolation). |
||
|
DanConwayDev
|
2262adea35 |
feat(nip34): consolidate duplicate role-tag histories instead of rejecting
The NIP allows a pubkey one tag of each role letter; a second tag under the same letter is out of spec. Rejecting such announcements dropped otherwise-valid membership data over a formatting slip, and clients that merge histories tag-by-tag can plausibly emit them. Tolerate them instead: histories for a pubkey are consolidated, and since this service only evaluates current activity - never time-scoped historic authority - consolidation reduces to "a maintainer while any `M`/`m` entry is active", exactly the rule already applied across letters for role transitions. Duplicate-rejection tests become consolidation tests: an ended entry plus an active one keeps the pubkey a maintainer (including the author's own self-entries), all-ended duplicates end it, and duplicate moderator tags still grant no maintainership. Validation: nostr::events and git::authorization unit tests and the state_authorization suite pass. |
||
|
DanConwayDev
|
e9a54678d5 |
feat(nip34): treat the moderator role tag as a role without maintainership
Follow the clarified NIP-34 indexed roles model (nips 986edd1): the `o` (moderator) tag is a role tag, so its presence suppresses the deprecated `maintainers` fallback, and a self-`o` entry is a self-role, so a moderator-only author is not implicitly a maintainer and their state events are not authorized. `o` listings grant no maintainership and create no maintainer invitations. Like `M`/`m`, a pubkey may appear in at most one `o` tag; a duplicate rejects the announcement. No further handling is needed: moderators are only empowered to have their status events (kinds 1630-1633) treated as authoritative, and this relay does not reject status events from non-maintainers, so the role's own authority requires no enforcement. Moderators never publish authoritative repository state. Deliberately deferred: announcements from moderators are not walked for the `M`/`m` assignments they might carry (the NIP says role combinations beyond self-plus-lead SHOULD be avoided unless the author is `M`), and moderator membership gets no reciprocal-confirmation treatment. Both only matter if status-event authority is ever enforced. Validation: nostr::events and git::authorization unit tests and the state_authorization suite pass. |
||
|
DanConwayDev
|
d0a23caf5c |
feat(nip34): parse indexed M/m role tags as the primary maintainer listing
Follow the indexed repository roles format from NIP-34 (nips 986edd1): `M` (lead) and `m` (co-maintainer) tags are now the primary maintainer listing, and their presence means the deprecated `maintainers` tag is ignored entirely. The lead / co-maintainer distinction carries no meaning for this service, so both collapse into one maintainer set. Role tags may record history as alternating start/end timestamps; a tag is currently active when it has fewer than four elements or an odd number of elements. Ended entries are ignored entirely: role history is only consulted to conclude that a pubkey is no longer a maintainer, never to grant time-scoped retroactive authority over historic events. A pubkey may appear in one `M` and one `m` tag to record a role transition and remains a maintainer while either entry is active; a second tag under the same letter is malformed and rejects the announcement. RepositoryAnnouncement::listed_maintainers() - already the single source for the listed maintainer set since the reciprocal-membership commit - now prefers active role-tag entries over the deprecated tag, so state authorization, replacement detection, the maintainer exception, sync discovery and the dependency walkers all pick up the new format through the sites switched to it here. Two refinements to membership follow from the format: - An announcement using role tags acknowledges its author via an active self-entry, or implicitly: per NIP-34 an author who appears in no role tag is a maintainer for the repository's entire history. Only an ended self-entry means the member left, which takes precedence over assignments in other announcements and is distinct from merely being invited (author_has_left). - A `u` (subordinate fork) tag has no effect on maintainership: the author of a role-less announcement asserts maintainership with or without it. Correctness assumption: authorization remains namespace-scoped, so the owner of a repository namespace stays authorized for it regardless of their own role history; role history only ends the authority of listed maintainers. Conflicting listings across announcements resolve as the union of confirmed members' active listings, matching the NIP's current-role rule; the NIP's owner-first precedence applies only to conflicting records of past roles, which this service never evaluates. Scope deliberately excluded: the moderator role tag (`o`) is handled in a follow-up commit. Validation: nostr::events and git::authorization unit tests, state_authorization suite (including new role-tag acceptance and ended-role rejection tests) and the sync invitation tests all pass. |
||
|
DanConwayDev
|
96cb60501a |
feat(nip34): require reciprocal announcements before maintainer state is authorized
Follow the reciprocal membership rule from the refined NIP-34 maintainers model (nips 781590b): a pubkey listed as a maintainer is only *invited* until its own announcement for the same identifier lists back an existing confirmed maintainer. State events from invited maintainers are no longer authorized; previously any pubkey reachable through recursive `maintainers` expansion was authorized without ever acknowledging the role. compute_membership replaces get_maintainers_recursive as the single membership computation: a fixpoint over announcements that confirms a listed pubkey once their own announcement lists back a confirmed maintainer. The owner is always a confirmed maintainer of their own repository, since announcing a repository in their namespace is what creates it on this service. collect_authorized_maintainers keeps its signature so all state-authorization call sites pick up the new semantics unchanged. Invited maintainers' announcements are deliberately still fetched, accepted (maintainer exception) and walked by the sync dependency machinery - the reciprocal announcement is exactly how the relay notices an invitation was accepted. To complete that flow, an acceptance stored via the maintainer exception is now pre-saved and stored state events are re-applied (new reapply_stored flag on process_state_event) so the newly confirmed maintainer's latest state re-points the owner's repository without another push; without this the stored state short-circuited as a duplicate and the invitation flow stalled. Remove the dead pre-membership helpers (AuthorizationContext, find_latest_state_for_announcement, find_latest_authorized_state, is_latest_state) that encoded the superseded semantics and had no other callers. Tests updated: invited state is rejected until acceptance (tests/state_authorization.rs), the invitation sync test now asserts the owner's refs are withheld until the invitee accepts, and grasp-audit maintainer fixtures publish reciprocal announcements. Scope deliberately excluded: the indexed `M`/`m` role tags and the moderator role from nips 986edd1 land in follow-up commits; this commit changes membership semantics for the deprecated `maintainers` tag only. Validation: git::authorization unit tests, state_authorization suite, sync invitation tests and grasp-audit lib tests all pass. |
||
|
DanConwayDev
|
eca818041c |
docs(storage): define the v2 to v3 migration boundary
Explain that identifier-family storage is an automatic, one-way on-disk migration and that rollback requires the pre-upgrade Git and relay-data snapshot. Keep the operator procedure focused on actions they must actually take. Document the precise untagged development window that could create server-side shallow repositories, the missing-push fallback that triggered it, and v3's non-degrading automatic repair behavior. No tagged v1 or v2 release shipped that fetch behavior. This change deliberately leaves on-demand integrity commands in the technical explanation rather than making them a required upgrade step. Validation: git diff --check and historical commit/tag inspection. |
||
|
DanConwayDev
|
56d705e2bb |
fix(storage): persist backup retirement before journal removal
Motivation: writing Retired before recursive deletion makes retries safe, but removing the journal durably without fsyncing the backup directory changes could leave a resurrected backup and no journal after power loss. Approach: fsync the immediate backup parent after remove_dir_all, then fsync each parent after pruning an empty child directory. The journal is removed only after those directory-entry changes are durable. Non-empty or already-absent ancestors stop optional pruning; unexpected pruning errors are logged without undoing the already-durable backup removal. Correctness assumptions: the migration root and journals reside on the same durable Git-data filesystem, and directory fsync provides the persistence boundary for entry removal. A failure before journal removal retains Retired state for an idempotent restart. Deliberately excluded: this does not change which backups qualify for retirement or the content-addressed quarantine layout. Validation: all migration unit tests pass, including complete and Retired restart states, successful parent pruning, retained backups, and content quarantine. Formatting and diff checks pass. |
||
|
DanConwayDev
|
05f9b10810 |
fix(storage): quarantine unindexed packs by content
Motivation: a later migration of the same repository path can encounter a different unindexed or damaged pack using a filename already present in quarantine. Treating an existing filename as identical would delete the new payload with its backup. Approach: stream each unindexed pack through SHA-256 and place it beneath a digest directory while retaining its original filename. Identical retries converge on one path; differing bytes use distinct paths. An existing destination is rehashed and any mismatch fails closed. Correctness assumptions: quarantine and migration storage share a filesystem, so rename remains atomic. SHA-256 paths are derived before the source moves, and an existing path is trusted only after its content verifies against that digest. Deliberately excluded: directory-deletion fsync ordering is handled separately, and quarantined raw packs remain operator-managed without automatic indexing. Validation: an end-to-end regression migrates two replacement repositories at the same path with different unindexed payloads under the same pack filename and verifies that both survive. Formatting and diff checks pass. |
||
|
DanConwayDev
|
59e9693b9e |
fix(storage): require healthy families before retirement
Motivation: the object-superset gate compared only names returned by cat-file. Git can enumerate a corrupt loose object by filename, so an already-migrated family could appear to contain a healthy backup object even when that backup held the only readable copy. Approach: complete the per-view and backup-superset checks first, then run the permanent identifier-family integrity inspection once for every ready backup in the family. Retirement proceeds only when that report is healthy. Missing history, corrupt objects, broken refs, invalid alternates, shallow views, and pack errors retain the backups and are reported for the ordinary asynchronous repair worker. Correctness assumptions: structural verification errors still abort an active migration, while a successfully inspected but unhealthy family does not block startup because its accepted clone sources are only available to the post-start integrity worker. Old completed migrations retain unverifiable backups without enforcing stale ref snapshots. Deliberately excluded: quarantine naming and deletion durability are separate review fixes, and this commit does not add another legacy repair path. Validation: the migration test group passes, including a regression where OID enumeration sees an empty corrupt family object while the legacy backup retains a healthy copy. Formatting and diff checks pass. |
||
|
DanConwayDev
|
28e885da63 |
feat(storage): recover full closure for shallow legacy repositories
Motivation: a legacy repository carrying a server-side shallow marker is a deliberate truncation. Treating it like a complete repository would let the new rolling backup retirement delete its legacy copy while the family lacks the truncated history, and the marker itself must not become permanent family state. Approach: migration records the marker in the view journal, copies it onto the installed thin view so clients keep seeing exactly what the legacy repository served, and excludes the backup from retirement until the family holds the complete reachable closure of the backup's refs. Closure recovery is the existing integrity repair: inspect_family now reports marked views as shallow_views (unhealthy), fsck already reports the truncated parents as missing objects, repair fetches the closure from accepted clone URLs, and removes the marker once the closure verifies. The backup then retires on a later launch. An unrecoverable closure retains the backup and reports the family as unresolved without ever blocking startup, since recovery depends on remote servers that are only reachable after the listener starts. Correctness assumptions: the closure check walks rev-list --objects over backup or view ref tips that exist in the family, skipping blob-target refs whose closure is their own presence and tips that are pre-existing missing objects reported separately. Client-requested shallow clones and fetches are an upload-pack protocol feature and are unaffected. Deliberately excluded: retrofitting shallow markers onto views migrated by earlier releases (their families heal through the same missing-object repair) and any change to outbound fetch depth behavior. Validation: new tests cover migration of a real depth-1 clone (backup retained, family reported unresolved, marker preserved, shallow client clone still served), retirement with marker removal and a full clone after closure recovery, integrity reporting of marked views, and repair removing or keeping the marker depending on closure completeness. cargo fmt, strict clippy, and the full test suite pass. |
||
|
DanConwayDev
|
e8c017aa41 |
feat(storage): retire migration backups family by family
Motivation: launch migration kept every legacy repository under .grasp/migration/backups until an operator deleted them, so converting a storage root needed disk for roughly two complete copies of the Git data. Approach: after the last view of an identifier family is installed, each of that family's backups passes a retirement gate before the next family migrates: the family must contain every Git-readable backup object (cat-file --batch-all-objects superset), every affected view must be a correctly wired thin view whose refs and HEAD match its journal snapshot, and every family pack must be indexed and pass git verify-pack. A durable 'retired' journal state is fsynced before deletion starts, so an interrupted deletion resumes without re-verification; the journal itself is removed once the backup is gone. Peak migration overhead is now bounded by the family in flight. Correctness assumptions: the gate deliberately does not reuse the family integrity inspection, which reports pre-existing missing objects and broken ref targets as unhealthy; an object that was never in the backup cannot be preserved by keeping the backup, and such damage stays visible through the integrity worker and integrity-check --repair. Packs without an index are Git-invisible and cannot be vouched for by the superset proof, so they are quarantined under .grasp/migration/unindexed-packs before deletion. Any gate failure during an active migration is fail-closed and keeps the backup. Journals from a migration that committed long ago are handled leniently: their ref snapshots are stale once the server has served traffic, so snapshot equality is skipped, unverifiable backups (including those whose migrated view was later deleted by repository lifecycle) are retained with a warning as operator-managed rollback material, and completed journals whose backups were already removed manually are compacted away so such installations keep starting normally. Deliberately excluded: unreachable-object pruning inside families, any automatic deletion of unverifiable backups, and S3 storage remain out of scope. Validation: new unit tests cover successful retirement of multiple views in one family, gate failure before verification preserving the backup, interrupted retirement resuming after restart (complete and retired journal states), completed installations with manually deleted backups, and fail-open retention when the family cannot prove it supersedes a stale backup. cargo fmt, cargo clippy --workspace --all-targets -D warnings, and the full cargo test suite pass. |
||
 DanConwayDev
|
d73f2a3276 |
Merge #4584cec7: Route Git traffic through identifier families
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsytpxwcu3rtwvqntrk64dnekh6xf9czgv8sedtd8rd95fsugc53xcwp9h3l PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 CoverNote: Completes the local identifier-family storage model after the prerequisite storage-primitives PR was merged. - Routes owner and `/prs/` reads, pushes, and proactive fetches through a shared `(object format, identifier)` object family. - Lets related repositories satisfy reachable SHA wants and advertises retained family base refs, avoiding repeat uploads of objects already stored by the server. - Migrates legacy repositories deterministically on launch while retaining rollback backups and preserving incomplete refs and readable objects. - Adds one permanent family integrity/healing engine for packs, object connectivity, view alternates, and ref targets. It fetches exact missing OIDs from clone URLs in accepted announcements through the existing hardened outbound path, rechecks the family, and logs unresolved damage at `ERROR`. - Runs that engine asynchronously after migration and exposes `ngit-grasp integrity-check --identifier <id> [--repair]` through a durable live-process request queue. Migration does not get a separate recovery subsystem: it performs the structural conversion, then hands the resulting family to the ordinary steady-state checker. Unindexed legacy packs remain in the rollback backup. Garbage collection, legacy backup archaeology, and S3 storage remain out of scope. Testing on gitnostr.com: the already-installed storage version makes structural migration a no-op, but the startup integrity pass still runs unconditionally, so this is a valid test of the permanent steady-state path. To prove remote self-healing, use a sacrificial identifier whose accepted announcement lists a second Git server containing the same reachable object; snapshot its family and views, move one verified loose object into quarantine, invoke `integrity-check --repair` or restart, and verify the repair log, restored object, `git fsck`, and a fresh clone. This does not re-test the first legacy-to-family transition; that transition should remain covered by the migration fixtures or a disposable pre-migration data copy. Do not remove the production migration marker to force a rerun. |
||
|
DanConwayDev
|
c68ae50e13 |
feat(storage): run family integrity repair
Newly migrated and steady-state identifier families need the same healing path, but remote availability must not gate relay startup. Start one background pass after migration and database initialization, repair through accepted clone URLs, and emit a bounded error for every unresolved family. Add a durable identifier-scoped request queue and an integrity-check operator command so the live process performs manual checks under the existing family leases. Check-only and repair requests cover every object format and owner or /prs/ view for the identifier, and repeated requests coalesce safely. Document the family-level integrity boundary and make migration's handoff deliberately small: structural conversion remains fail-closed, then the ordinary new-model pass handles pre-existing damage. This does not add legacy backup archaeology, garbage collection, periodic remote repair, or S3 behavior. Validated with cargo fmt, strict locked workspace Clippy, the complete locked workspace test suite, focused request-worker tests, and the command help path. |
||
|
DanConwayDev
|
1ad06bae22 |
feat(http): support path-mounted GRASP services
Allow operators to share an authority by configuring a normalized public base path. Route HTTP, WebSocket, Git, and GRASP-06 traffic beneath that prefix and carry the path through service validation, generated relay identity, self-sync, NIP-11 icons, and landing links. Treat NIP-05 as a domain-root capability: path-mounted services neither expose the well-known document nor advertise NIP-05 in NIP-11, and generated owner profiles omit the nip05 field. Exact relay URL matching keeps sibling mounts distinct while clone URLs may descend below the configured prefix. This assumes reverse proxies preserve the configured public prefix. Proxy-side prefix stripping remains deliberately out of scope. Validation: - nix develop -c cargo test - nix develop -c cargo clippy --all-targets -- -D warnings - cargo fmt --all -- --check - git diff --check - nix-instantiate --parse nix/module.nix |
||
|
DanConwayDev
|
4834a11b54 |
feat(storage): activate local families during launch
Motivation: routing new Git traffic through identifier families is not a complete deployment boundary for an existing server. Legacy owner and /prs repositories must be converted before runtime recovery, synchronization, or HTTP handling can safely assume thin views. Approach: run a versioned, fsynced, restart-safe migration during RelayServer startup. Build a verified union containing every reachable and unreachable object for each identifier family, retain original repositories as rollback backups, atomically install exact refs/HEAD views, and make archive restoration plus purgatory revival preserve the family layout. Correctness assumptions: startup has exclusive process ownership of the Git root; validated identifiers are safe migration path components; a migration error fails launch closed; successful local receive finalization releases its family lease before recovery can re-enter it. Focused recovery tests cover crash resume, unreachable objects, archive restoration, deletion transitions, and revived views. Deliberately excluded: S3 dependencies, backend configuration, cache eviction, remote durability, garbage collection, and automatic backup deletion. Those remote-storage concerns remain one later opt-in PR. Validation: nix develop -c cargo check --all-targets; cargo test --lib git::migration::tests; cargo test --lib extending_soft_expired_announcement_recreates_a_thin_view; cargo test --test nip09_blacklist_ops; cargo test --test nip09_state_cascade; cargo test --test nip09_recovery; cargo fmt --all -- --check; git diff --check. |
||
|
DanConwayDev
|
f084f7e973 |
feat(storage): route Git traffic through identifier families
Motivation: owner repositories and GRASP-06 contributor routes currently store or copy the same Git objects independently, forcing clients to upload data the relay already has. Approach: create new repositories as thin ref views, direct receive-pack and proactive fetch writes into the shared identifier family, retain accepted tips, and expose family bases as anonymous receive negotiation haves. Existing legacy repositories deliberately remain self-contained until the startup migration layer lands. Correctness assumptions: repositories only share objects when their validated identifier and object format match. A process-wide per-family lease serializes object-producing operations, while the terminal receive-pack flush remains behind ref validation and post-push processing. Excluded scope: remote S3 durability, cache eviction, garbage collection, and conversion of existing repositories are separate stack layers. Validation: cargo check --all-targets; cargo test --test git_response_streaming; cargo test --test grasp06_pr_hosting; plus repository_creation, git_clone, and storage/purgatory targeted suites from the preceding review. |
||
|
DanConwayDev
|
927c38f059 |
Merge #e88acf4b: Design identifier-family Git object storage
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsw3zk0fdhqeummycnmmpzls0zwjzxmkfgnqgqnsje2hhdvvsgewksq2g90s PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 PR description: Defines identifier-scoped object families shared by owner repositories and /prs/ views. The design accepts allowReachableSHA1InWant for related repositories, keeps local storage as the default deduplicated backend, makes S3 opt-in, and deliberately excludes garbage collection so deletion rollback remains recoverable. |
||
|
DanConwayDev
|
f632ce7de7 |
docs: define administration and configuration vision
Motivation: Operators need a coherent path from declarative production configuration to approachable single-binary administration. Maintainers also need scoped private analytics without service-wide authority. Approach: Define a NIP-98-authenticated, NIP-86-compatible control plane with namespaced extensions, an embedded React client, a provenance-aware settings registry, and a lower-priority SQLite override store. Use repository stats plus changerelaydescription as the first read/action vertical slice. Correctness assumptions: Command-line, credential, environment, NixOS-rendered, and dotenv sources remain authoritative over database overrides. Repository authorization reuses current recursive maintainer rules, and lifecycle mutations continue through existing serialization facades. Deliberately excluded: This documents direction only. It does not implement the API, UI, database, quota accounting, or live reconfiguration. It also does not promise that every setting will become API-writable. Validation: Ran git diff --cached --check and verified every local link introduced by the new vision and index entries resolves. A broader scan found an unrelated pre-existing broken documentation link, which is left outside this atomic change. |
||
|
DanConwayDev
|
c6b79ccbd9 |
docs(storage): define identifier-family object model
Motivation: Related owner repositories and GRASP-06 submission routes currently duplicate large Git histories and force clients to upload objects already held by the service. Approach: Specify identifier-and-object-format families, thin per-route views, anonymous alternate negotiation, local-first durability, opt-in S3 hydration, append-only recovery roots, and a crash-safe startup migration. Link the accepted design from the explanation index and architecture overview. Correctness assumptions: Git alternates expose family base tips as anonymous .have entries, and allowReachableSHA1InWant across related repositories is an accepted disclosure. No object is deleted until rollback-aware GC is designed. Deliberately excluded: This commit changes documentation only; family storage, S3 transport, and migration are implemented in later commits in the stack. Validation: git diff --cached --check |
||
|
DanConwayDev
|
b50883d9f0 |
Merge #2f5ad1ca: fix(logging): align operational severity with actionab…
fix(logging): align operational severity with actionability nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsz7kk3e29ceup4fzccjxq48r6vxu4f3jd54anfurevch0vp6lfmvq9fke8q PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 PR description: Production evidence showed the default info stream was dominated by dependency chatter and peer-controlled detail: gitnostr emitted 1,008 per-repository discovery records in a 3,501-line window, the US relay emitted 232 in 972 lines, and the German relay emitted 2,288 discoveries plus 2,236 warnings in 11,472 lines. Rejected historical announcements dominated the warnings, while all three services were intentionally running NGIT_LOG_LEVEL=info rather than debug. This PR scopes bare levels to ngit-grasp with dependencies held at warn, while preserving explicit EnvFilter expressions. It moves individual repository, event, filter, WebSocket, HTTP disconnect, and missing-object probe diagnostics to debug; retains aggregate sync outcomes and capability fallback at info; retains transient cooldowns at warn; and keeps internal database, policy, service, and subprocess failures at error. Unsupported NIP-77 is reported once per connection rather than once per filter. Configuration docs, the NixOS module, the example environment, monitoring guidance, and changelog are updated together. Tests that use rejection diagnostics as an ordering barrier explicitly opt into application debug logging. A scheduler-sensitive checkpoint-expiry test encountered during the package build now simulates downtime through persisted state instead of a fixed sleep; production behavior is unchanged. Validation: - cargo fmt --check and git diff --check - focused logging, HTTP, Git, self-subscriber, capability-gate, downtime, and maintainer-reprocessing tests - all 795 library tests passed in the development suite - all nine maintainer-reprocessing tests passed sequentially after opting their fixtures into debug - the 2,500-event REQ concurrency stress test passed in isolation after an earlier host-contention timeout - nix build .#ngit-grasp --no-link passed, including the pinned-toolchain 795-test package gate |
||
|
DanConwayDev
|
36634dc326 |
fix(private-repos): restore self-subscription in private mode
The SelfSubscriber feeds the sync manager's repository index from the service's own accepted events. It dialled our own public WebSocket endpoint with an unauthenticated client, which a private instance's NIP-42 gate correctly refused: the gate runs in the HTTP layer, ahead of LocalRelay, and cannot distinguish our own process from any other anonymous dialler. Announcements accepted at runtime therefore never reached the sync index until a restart rebuilt it from the database, stalling proactive sync and the dynamic membership derived from accepted relay owners — worst on exactly the private-to-private mirroring GRASP-08 exists to enable. Attach the subscriber to the embedded relay in-process instead. A custom WebSocketTransport hands LocalRelay one end of an in-memory duplex pair and keeps the other, so the client gets an ordinary relay session with the same framing, subscription handling, and post-save broadcast, minus the listener, the auth gate, and the network round trip. This replaces the loopback dial in both modes: the public-mode feed is identical in content and strictly more reliable, and it removes a self-directed reconnect loop. Chosen over attaching the relay owner key as a NIP-42 authenticator plus adding the owner pubkey to the effective member set. That alternative works, but widens the member set and the authenticated surface to solve a problem that is not authentication: there is no remote party here. The in-process route needs no key, no membership entry, and no configuration, and nothing reaches the subscriber that the relay did not already accept and persist. Correctness assumptions: LocalRelay applies no NIP-42 or query policy of its own — private-mode access control lives entirely in the HTTP layer — so an in-process session is exactly a local session, not a bypassed remote one. Both ends speak raw WebSocket framing over the duplex with no HTTP upgrade, matching take_connection's Role::Server. The session consumes one connection permit, as the loopback dial did. The GRASP-08 regression test no longer restarts the relay over persistent LMDB: it publishes an announcement at runtime and asserts sync connections to both referenced relays, which is only possible if the live feed reached the index. Verified to fail against the previous implementation (60s deadline, no connections) and pass with this one. req_concurrency's source relay now applies the production outbound target policy. Its scenario lists a proxy URL in the announcement, and with a reliable live feed the source discovers that URL — a distinct host:port that happens to front itself — as an event-directed sync target and opens its own REQ traffic through the proxy, contending for a budget the test means to measure for the syncing relay alone. That behavior is pre-existing and was already reachable after a restart; only its timing changed. The policy keeps the source scenery without weakening the assertion. Deliberately excluded: neg_concurrency shares that topology but passes unchanged, so its fixture is left alone; the duplicate NIP-11 fetch between the pre-dial preflight probe and the post-connect hint fetch is untouched. Validation: cargo clippy --all-targets -D warnings; cargo test --lib (793 passed); cargo test --test private_mode --test sync --test outbound_policy --test purgatory_sync (255 passed, including the full sync suite under parallel load). req_concurrency's startup-burst test passed 4/4 isolated runs after the fixture change. |
||
|
DanConwayDev
|
e741d9ddc4 |
docs(explanation): document disclosure guidance for private services
Teams asking how a private service receives CVE and security reports had no documented answer, inviting ad-hoc workarounds that could weaken the access boundary. Adds a "Disclosure and outside contributions" section to the GRASP-08 design doc: the security contact is discoverable via the public NIP-11 contact field and the NIP-05 root identity; reports arrive as NIP-17 encrypted DMs on the maintainers' public mailbox relays; collaboration access is granted by adding the reporter to NGIT_PRIVATE_MEMBERS; and a non-member submission lane is deliberately not implemented because it would reopen the unauthenticated write surface GRASP-08 exists to close (mirroring the GRASP-06 incompatibility rationale). Doc-only change; no behavior is added or altered. Validated against the discovery surfaces the implementation actually exposes (unauthenticated NIP-11 and NIP-05 in private mode). |
||
|
DanConwayDev
|
9da74ac03f |
docs(explanation): document the GRASP-08 outbound policy matrix
The GRASP-08 design doc listed outbound authentication as follow-up scope; the preceding commits implemented it, and architecture docs are living documents that must describe what was built. Replaces the follow-up bullet with an "Outbound authentication and sync policy" section presenting the decided matrix: NIP-42 answered everywhere with the relay owner key (skipped and immediately parked without one), restricted refusals terminal via the policy-refusal machinery, GRASP-08-advertising peers parked pre-dial on public instances versus fully credentialed (NIP-42 plus repository-root NIP-98 on Git fetches) on private ones, and missing NIP-11 treated as an ordinary relay. Records the rationale that NIP-42 is identification rather than confidentiality, and the derived-membership tightening to GRASP-08-advertising relays. The architecture doc's GRASP-08 section gains the matching outbound paragraph and membership caveat. Doc-only change; multi-service fleet and encrypted kind-10318 discovery deliberately remain follow-up scope. Validated by reading the rendered markdown against the implemented behavior in src/sync/mod.rs, src/sync/relay_connection.rs, and src/purgatory/sync/context.rs. |
||
|
DanConwayDev
|
7b7b098b8e |
fix(logging): scope bare levels to the application
Production NGIT_LOG_LEVEL=info currently enables INFO records from every dependency as well as ngit-grasp, obscuring application health with SDK and protocol chatter. Expand bare levels into an explicit warn dependency baseline plus the requested ngit_grasp level. Preserve full EnvFilter expressions verbatim so operators retain precise temporary diagnostics, and document the severity contract consistently across source, NixOS, the environment example, and reference material. This commit deliberately does not reclassify individual application call sites; those changes remain separately reviewable. Validated with cargo fmt --check and the focused logging unit tests (4 passed). |
||
|
DanConwayDev
|
7334e04b88 |
feat(identity): publish relay owner events
The relay-owner key is intended to double as the service identity used by ngit-ci, but clients could only discover it through HTTP and owner-authored coordinator events without repository roots failed admission. Sign a minimal kind-0 NIP-05 bot profile at startup — per NIP-24 `name` is always set, here to the scheme-less public URL — alongside a single unmarked kind-10002 relay entry. An operator-customized profile is kept and never overwritten, and no identity event — locally stored or freshly generated — is published before the local database and at least one user-index relay have been successfully checked for that kind, so a database wiped and reseeded during an index outage can never displace a customized profile surviving on the indexes. Every send is preceded by a per-relay re-check: an identity found on an index relay is adopted locally, where replaceable-event semantics keep the newest copy, and is never overwritten, so publication only fills gaps on index relays that individually confirm they hold none; propagating a profile update onto an index that already has one is left to the operator's own client. A kind with no local copy is not even seeded until a reachable user-index relay confirms it holds no identity of that kind. Publication never blocks startup, retries transient failures with a capped backoff, and stops on terminal protocol rejections. With no user-index relays configured, missing kinds are seeded locally right away. In private mode (GRASP-08) identity events are seeded and served locally but never published, so a private relay does not advertise its existence. Trust valid owner-signed events only for kinds without a dedicated admission policy, such as ngit-ci coordinator advertisements that carry no repository root tag. Owner-signed NIP-34 announcements, state events, and PRs run the normal announcement validation, ref alignment, and purgatory git-data handling like any other author, and the relay's own kind 0/10002 identity is always accepted. The NIP-09/NIP-62 deletion gate still runs before any owner acceptance, so replaying a retracted owner event cannot undo its tombstone, and owner deletion/vanish requests keep their lifecycle handling. Because the event blacklist cannot block the owner key, rotating the key is the only remediation if it is compromised; this is documented. NGIT_DOMAIN is assumed to be the documented bare public authority: loopback authorities use ws and other hosts advertise wss, with bare IPv6 authorities bracketed. The test fixture reuses relay-owner keys across TestRelay::restart, and gains caller-provided keys, a private-mode member setup, and explicit user-index relay lists; MockRelay can start pre-seeded so a "recovering" index deterministically holds prior state. Identity retry intervals honor the existing NGIT_TEST fast-timer convention. No new configuration switches or ngit-ci changes are included. Includes rustfmt fixes for src/nostr/policy/announcement.rs and tests/private_mode.rs, which arrived on master unformatted and would otherwise fail the workspace format gate. Validated with rustfmt, strict workspace Clippy, the relay_identity and private_mode integration binaries, and the full workspace test suite. Individual sync/grasp06 integration tests fail intermittently under parallel full-suite load, each passing in isolation and on rerun; the same intermittent failures reproduce on origin/master without these changes. |
||
|
DanConwayDev
|
a3bfd023b2 |
feat(private-repos): gate announcement admission on membership
In GRASP-08 private mode, accepted kind-30617 announcements expand hosting and — via their referenced relays' NIP-11 owners — the effective member set itself. Admission never checked the announcement author, so a member could submit a third-party-signed announcement (or sync could import one from an operator-configured source) and mint membership for pubkeys no member ever chose. AnnouncementPolicy now rejects a repository announcement whose author is not a current effective member, using the shared PrivateAccess set (configured members plus admitted relay owners) threaded from server startup through create_relay and Nip34WritePolicy. The check runs at the top of AnnouncementPolicy::validate, the single choke point every announcement arrival path (direct publish, sync import, purgatory entry) funnels through, and uses the existing AnnouncementResult rejection machinery so operators observe these rejections like any whitelist rejection. Public-mode behavior is unchanged (the gate is None outside private mode) and no configuration was added: the gate is implied by NGIT_PRIVATE_MODE. Correctness assumptions: membership is evaluated against the live set at admission time via PrivateAccess::contains, not re-derived; state events (kind 30618) stay governed by GRASP-01 maintainer rules; and removal is non-retroactive — repositories admitted while their author was a member remain hosted until the operator curates them. Deliberately excluded: outbound authentication when syncing from other private services, which remains a follow-up in the design doc. Validation: cargo clippy --all-targets -D warnings clean; lib tests (783), private_mode (52, incl. new member/non-member admission integration test), nip34_announcements (60), repository_creation (47), and purgatory (55) suites all pass. |
||
|
DanConwayDev
|
44d79c38a0 |
docs(explanation): document GRASP-08 private service design
GRASP-02/03/05/06 each have a dedicated explanation document, but the merged GRASP-08 single-service implementation was described only by a section in architecture.md. Add the missing design document covering the implemented behavior: the fail-closed configuration, the indistinguishable empty 401 contract, the repository-scoped NIP-98 profile and why it deviates from generic NIP-98, NIP-42 authentication outside the embedded relay, service-wide membership with dynamic accepted-relay-owner admission, and the trust model including non-retroactive removal. Index the document from the explanation README. The follow-up scope section records what is deliberately not yet implemented (membership-gated announcement admission, outbound authentication, fleet orchestration); implementation commits that close those gaps must update it. Documentation only; no behavior change. Validation: proofread against the merged implementation in src/private/, src/http/mod.rs, src/git/mod.rs, and src/sync/mod.rs. |
||
|
DanConwayDev
|
6b4d585bf0 |
feat(private-repos): admit accepted relay owners
GRASP-08 requires the service whitelist to include NIP-11 owners of relays referenced by accepted announcements. Static configuration alone would incorrectly deny those relay operators and make repository synchronization between private services unusable. Carry the owner field through the existing once-per-session NIP-11 limit fetch, retain the latest owner per canonical relay, and reconcile the shared inbound access set from configured members plus owners whose relays appear in Full repository sync entries. StateOnly purgatory entries are excluded because unaccepted peer input must not grant access. Reconciliation runs on connection completion and the existing five-second maintenance cadence; unchanged sets do not churn authenticated sessions. This adds no connection, request, subscription, or configuration option. Multi-service fleet orchestration and encrypted kind-10318 discovery remain separate future work. Validation: cargo check --all-targets passed; focused private authentication tests passed (7/7); unit coverage verifies owner parsing without a NIP-11 limitation object, configured/accepted/unrelated membership selection, and no generation change for an identical replacement. Full suite validation follows on the completed two-commit stack. |
||
|
DanConwayDev
|
d7766dace7 |
feat(private-repos): require GRASP-08 service authentication
Private repository events and Git objects must not be readable merely because an endpoint is reachable. Add an opt-in, fail-closed single-service mode that requires NIP-42 authentication before bridging WebSocket traffic and a repository-scoped GRASP-08 NIP-98 credential before serving Smart HTTP. The Git credential signs the canonical repository root with method GET and is reusable across the standard Git endpoints for its 60-second validity window. Authentication precedes repository lookup, every failure returns the same empty 401 challenge, canonical paths cannot escape the Git root, browser clients can inspect the challenge through CORS, and ordinary GRASP-01 push authorization remains authoritative after authentication. The canonical public origin is operator-controlled so reverse proxies cannot influence signed identity. This commit deliberately implements one configured private service and excludes dynamic relay-owner membership, encrypted kind-10318 client discovery, and multi-service fleet orchestration. Validation before consolidation: cargo check --all-targets passed on current master. Focused unit coverage exercises credential reuse, signature/member/URL/method rejection, canonical origin and repository paths, fail-closed configuration, authentication framing, and empty indistinguishable failures. |
||
|
DanConwayDev
|
18eddeedb5 |
Merge #4e732958: feat(sync): discover participant mailboxes
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsyuueftzc68d5a3x5d0pecs66y0nacv4pc87kkcfw499adyl5t3ycxqgtta PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 CoverNote: ## What this adds Repository conversations can continue on relays used only by a non-root participant. For example, a reaction to an issue may be stored on one relay, while a reply to that reaction exists only on the reaction author's NIP-65 write relay. Root-author relay discovery and recursive reference queries cannot find that reply unless the participant's mailbox is also searched. This PR adds bounded, historic participant-mailbox coverage: 1. Preserve exact repository-root provenance while deriving the accepted recursive descendant frontier. 2. Treat authors of accepted replies, comments, reactions, zaps and other descendants as thread participants. 3. Accept kind `10002` relay lists only for repository owners, maintainers, root authors and those accepted participants. 4. Derive each participant's NIP-65 read, write and unmarked mailbox relays. 5. Query those mailboxes only for the repository roots and accepted descendant IDs associated with that participant. 6. Pass every returned event through the ordinary write policy, deletion/replacement rules and persistence pipeline. This discovers replies to reactions and other indirect descendants without fetching a participant's unrelated notes or trusting their relay list to bypass repository policy. ## Bounds and scheduling - The existing `NGIT_SYNC_RECURSIVE_DESCENDANT_LIMIT` bounds which indirect descendant IDs remain query roots. Each direct event that tags a repository root has its own bounded branch; the default is 500. - Once a branch reaches that bound, later indirect descendants cannot add more participant authors or query roots through that branch. - Exact root provenance prevents roots from one repository leaking into another participant query. - Participant mailbox coverage is historic only. It does not add permanent non-root participant subscriptions or expand the existing live-sync tier. - Filters use the existing byte-bounded grouping and paginated `RelayConnection::fetch_events` path, including request pacing, background priority, subscription-ledger capacity and EOSE/CLOSED handling. - At most one mailbox worker runs per relay, and at most one new due relay is started per maintenance pass. - Relays progress independently: there is no global mailbox lane, cross-relay success condition or shared completion counter. ## Restart and failure behavior - A probe starts only after both the WebSocket and the sync actor's connection lifecycle are ready. - Ready due relays are preferred, so old unreachable mailbox sources cannot starve connected work. - Completed mailbox workers are handled before new connection results, so a large startup connection queue cannot delay cursor progress or resource release. - Successful filters advance an in-memory, stable-sorted cursor immediately. Completing the last filter schedules the next historic rotation after 24 hours. - Failed filters release their relay worker and retry after five minutes without blocking other relays. - On restart, accepted roots, participants and kind `10002` ownership are reconstructed from LMDB. Filter cursors deliberately restart from the first current group; they are best-effort coverage, not durable exactly-once state. The implementation reuses the existing relay transport. It does not add a mailbox-specific protocol state machine, pending-batch purpose, CLOSE API, NIP-42 retry system, watchdog or connection-queue coordinator. ## Coverage and production evidence The audit motivating this change found 72 events visible through gitworkshop but absent from the pre-change production relay across the `gitworkshop` and `ngit` repositories. The persistent canary served 29 exact misses: six kind-1 notes, three reposts, ten reactions, six comments, two issues, one zap receipt and one repository-follow event. The exact candidate `de5fa6625aa5693afcde7335a572456680aef9df` activated on the isolated archive at 2026-08-14 19:23:49 UTC. Restart reconstruction found 4,291 accepted roots, 403 participant authors and 351 mailbox relays. During the recorded soak: - 63 mailbox filters started and all 63 reached terminal handling: 59 successes and four bounded failures. - Failures received the intended five-minute retry and did not block successful progress on other relays. - `relay.ngit.dev`, `nostr.land`, `nostr.mom`, `haven.danconwaydev.com/inbox` and other relays advanced independently. - `nostr.azzamo.net` completed all six filter groups; the final terminal reported `completed_cycle=true` and scheduled the next probe in 86,400 seconds. - The archive remained active with zero restarts and no process panic or fatal error. The temporary archive test override of `NGIT_SYNC_RECURSIVE_DESCENDANT_LIMIT=2` has been removed. Both the archive and public gitnostr.com have no override and therefore use the default of 500. Public gitnostr.com was not restarted or changed by the archive deployment. ## Validation and review state - Exact tested head: `de5fa6625aa5693afcde7335a572456680aef9df`. - One commit; 1,157 additions and 140 deletions across 11 files. - 767 library tests passed in the exact Nix release build locally and on the production host. - All three proactive Sync+ integration scenarios pass together, including a child found only through a participant reaction author's write mailbox. - Strict all-target Clippy, formatting, diff checks, Nix flake evaluation and a conflict-free merge-tree check against current `master` passed. - One remote build attempt hit the pre-existing `test_entries_expired_during_downtime` timing flake; its source is unchanged by this PR, and the unchanged candidate passed all 767 tests on retry before activation. Recommended for merge. The archive demonstrates successful and failed terminal paths, per-relay independence, restart reconstruction, cursor advancement and a complete historic-to-24-hour-refresh rotation under real startup load. |
||
|
DanConwayDev
|
de5fa6625a |
feat(sync): probe participant NIP-65 mailboxes
Repository conversations can continue in the read or write relays advertised by authors of accepted replies, reactions, zaps, and other descendants. Root-author inbox discovery alone therefore leaves valid indirect descendants undiscovered. Carry exact accepted-root provenance through the existing bounded descendant frontier, admit identity events only for those participants, and derive read/write mailbox ownership from accepted kind 10002 events. Probe one byte-bounded historic filter at a time with the existing per-relay fetch_events pagination, pacing, ledger, timeout, policy, and persistence paths. Starts are paced globally, but progress and terminal state remain independent per relay. Require both an established socket and the sync actor committed lifecycle before starting a probe. Prefer lifecycle-active due relays while falling back to the existing oldest-due dial order, so hundreds of unavailable mailbox sources cannot starve already-ready work. Drain completed mailbox probes before accepting more connection results so a busy startup queue cannot delay cursor progress or resource release. Production canaries exposed these startup conditions without requiring cross-relay coordination. The recursive descendant limit bounds which indirect IDs remain query roots; direct root references remain complete. Mailbox filter cursors are intentionally best-effort in-memory state: restart reconstructs ownership from LMDB and safely begins historic coverage again. This does not add permanent participant live subscriptions or a cross-relay completion coordinator. Validated with cargo check, strict all-target Clippy, 767 library tests including lifecycle and ready-selection regressions, and the three proactive Sync+ integration scenarios, including a write-mailbox child that references only a participant reaction. |
||
|
DanConwayDev
|
7bca6e2482 |
feat(http): serve NIP-05 root identity
Relay operators already have a stable Nostr identity for NIP-11 and NIP-42, but clients cannot resolve the domain itself to that key. Serve the NIP-05 root identifier so _@domain identifies a root-mounted relay without another key source. Add an exact root-level /.well-known/nostr.json route for GET and HEAD, serialize the relay-owner public key as lowercase hex under names._, retain permissive CORS, and give this route precedence over NIP-11 content negotiation. Derive NIP-11 support from the relay request path: only / advertises NIP-05, while nested relay paths omit it. Keep the changelog, architecture, README, and relay-owner configuration descriptions synchronized. The path check assumes the HTTP request path is the relay's public mount path; a future explicit base-path setting should remain the source for this decision if proxy routing hides that path. Named identities and recommended relay-hint mappings remain outside this change. Validated with 760 library tests, the root/non-root NIP-05 server integration test, strict all-target Clippy, cargo fmt, git diff checks, and Nix module parsing. |
||
|
DanConwayDev
|
9360d419b3 |
feat(sync): bound recursive related frontiers
Repository conversations can continue through parent-only event and address references, but recursively following an unbounded social thread can pull unrelated activity into proactive sync. Extend the existing descendant rotation transitively while bounding each subtree rooted at an event that directly tags a repository root. Derive branch membership deterministically from the accepted local database on each reconciliation tick. A branch below the configurable frontier limit contributes its event IDs and coordinates to the existing live-or-REQ+EOSE machinery; a full branch contributes no further child-query seeds. Rebuild auxiliary live coverage when the frontier shrinks so stale child filters are actually closed. Because LMDB is the checkpoint, restart needs no new durable state and source relay loops remain independent. The default limit is 500 and direct root-tagging events do not consume it. The limit is intentionally soft: replies to requests already in flight can still be stored, but cannot extend a full branch. The existing eight-generation traversal bound remains. Exact receive-time admission and cross-relay coordination are deliberately excluded. Validated with 758 library tests, six focused frontier tests, a limit-2 historic/restart integration test, the serialized sync suite (95 passed, 1 ignored, 2 unrelated timing failures that passed isolated reruns), strict workspace Clippy, cargo fmt, and nix flake check --no-build. |
||
|
DanConwayDev
|
08d38fdeae |
fix(sync): retry related policy orphans
Repository comments, reactions, zap requests, and other related events can arrive before the event or repository address that makes them admissible. Sync previously discarded these restricted policy orphans, making arrival order a permanent coverage gap. Persist full dependency-sensitive related events in a bounded seven-day index, match later accepted references in both directions, and reprocess an iterative bounded closure. Keep cached dependency failures pending in exact-ID hydration and expose retained count through the fixed-cardinality state metric. The index is capped at 1,024 events, 8 MiB total, 128 KiB per event, and 512 attempts per trigger. Recursive remote discovery and participant mailbox probing remain separate stacked changes. Validated with 754 library tests, focused forward/backward/persistence/eviction and policy reprocessing tests, formatting, and clippy across all targets with warnings denied. |
||
|
DanConwayDev
|
39e1f692c0 |
fix(sync): account for hydration outcomes
Historic missing-event recovery treated relay delivery as success even when admission or persistence left the event unservable. That could promote a relay to healthy while repository coverage was still incomplete and gave operators no way to distinguish transport, policy, and storage gaps. Classify sync processing into saved, duplicate, purgatory, tombstone, bounded rejection, and persistence outcomes. Keep dependency-sensitive and transient failures pending during exact-ID recovery, export phase-scoped hydration counters, and document the operational queries. Permanent invalid/blocked events and events already owned by the rejected dependency index remain terminally accounted. Recursive frontier expansion and participant relay discovery are deliberately excluded for stacked follow-up changes. Validated with 750 library tests, the censored-relay historic recovery scenario, all sync-metrics integration tests, formatting, and clippy across all targets with warnings denied. |