mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-09-14 04:45:07 +00:00
master
320
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DanConwayDev
|
59a37b660a |
test: cancel proxy connections when fixtures stop
Stopping accept loops left detached relay/proxy sessions alive. Track HTTP, WebSocket upgrade and forwarding tasks under their owning fixture, cancel them on shutdown, and drain cancellation before explicit stop returns. Preserve censoring, rate limits, authentication and simulated disconnect behavior. Add regressions that observe a live protocol exchange before asserting the connection closes on stop, all under bounded deadlines. Validation: fixture lifecycle checks passed for censoring, REQ/NEG limiting, flapping and setup-drop relays. Auth-gating and upload proxy shutdown regressions passed through relay_identity's common helper tests. Assisted-by: Codex (GPT-6) |
||
|
DanConwayDev
|
3c77e76a1f |
test(git): own request tasks and avoid pipe backpressure deadlocks
Detached HTTP tasks could outlive stopped Git fixtures. Sequential stdin and output handling could deadlock when a child filled stdout or stderr before consuming input. Own connection tasks, cancel Git children with their request, and drain both outputs concurrently with stdin. Keep fixture HTTP semantics and repository layout unchanged. A bound owned listener replaces the unnecessary startup grace loop. Explicit stop joins cancelled tasks while synchronous drop aborts the owner. Validation: all seven shared SmartGitServer tests passed, including live connection shutdown and one-MiB concurrent pipe traffic. Assisted-by: Codex (GPT-6) |
||
|
DanConwayDev
|
cf2ec36a68 |
test(sync): wait for observable fixture readiness
Fixed startup and promotion sleeps could expire before asynchronous work completed. Wait with bounded deadlines for event visibility and an actual connected gauge; connection attempts and health do not prove readiness. Preserve URL schemes when constructing metrics endpoints, retain ownership of unavailable endpoints, and remove stopped sources instead of replacing them with a live placeholder. Keep immediate assertions where completion already guarantees the result. Production sync policy is unchanged. Validation: shared sync helper tests passed; relay_identity and the fixture lifecycle target compiled and passed with the revised common helpers. Full sync and deletion integration execution remains host validation. Assisted-by: Codex (GPT-6) |
||
|
DanConwayDev
|
75da9a8446 |
test: preserve subprocess and recovery fixture listeners
Releasing a reservation before spawn or recovery allowed unrelated tests to take the same port. Transfer the socket across exec through a private Unix test-only protocol, retaining a parent copy across Grasp restarts. Validate the inherited descriptor and keep it out of Git and SSH descendants. Offline identity fixtures retain an accept-and-close endpoint until recovery. Mock relay listeners and upgraded connections remain owned until shutdown. Use real HTTP or SDK readiness instead of grace sleeps; startup errors in the owned-listener path are not retried. Normal server startup and deployment configuration are unchanged. The capability probe permits older external harnesses to remain compatible. Validation: listener adoption unit tests passed; relay_identity passed with recovery coverage, MockRelay shutdown passed, and a real subprocess restart plus a cross-repository ngit harness smoke test passed. Full platform builds remain host validation. Assisted-by: Codex (GPT-6) |
||
|
DanConwayDev
|
1187cc062e |
test(git): synchronize streaming assertions with explicit gates
Sleeping fake Git processes and assumed HTTP frame boundaries made streaming checks depend on scheduler timing. Hold fake Git behind an owned loopback gate until the test observes the expected prefix; collect bytes independently of frame splits and explicitly release subprocess progress. Keep terminal-flush, promotion and cleanup ordering assertions. Gate waits have bounded deadlines; no production streaming code changes are included. Validation: all three git_response_streaming tests passed. Assisted-by: Codex (GPT-6) |
||
|
DanConwayDev
|
3f8125fdaf |
test(sync): simulate downtime in checkpoint integration test
The persistence integration test sleeps 75 ms between 50 ms hot-cache and
100 ms cold-index expiry deadlines. Snapshot writes and host scheduling
can consume the remaining margin, legitimately expiring both tiers.
A sandboxed nixpkgs build exposed this after the streaming tests passed.
Set the persisted saved_at timestamp back two minutes and use minute/week
retention periods. This follows the synthetic-checkpoint approach already
used by the rejected-index unit tests in
|
||
|
DanConwayDev
|
68347956ed |
Merge #e924d285: feat(cli): export deterministic documentation schema
nostr:nevent1qqswjfxjs5ckkaqrnnz64ds0uj5csfpqukdncffgdw0cgup844e85mspz3mhxue69uhhyetvv9ujumn8d96zuer9wc0ewdc3 PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 PR description: Documentation builds need a source-owned view of ngit-grasp commands and runtime configuration without copying Clap help or operator defaults into another repository. Add an early hidden __docs-export dispatch that serializes schema-v1 command metadata, deduplicated public environment options, runtime constraints, dotenv behavior, and relay-owner secret precedence. Clap remains authoritative for reflectable facts; a narrow explicit registry covers validation and the skipped secret. Correctness assumes Config::validate relationships continue to be mirrored in the constraint registry when they change. Secret source metadata intentionally contains locations and precedence only, never loaded values. This does not publish artifacts, generate ngit.dev pages, expose internal test switches, model conditional Clap requirements that stable reflection cannot expose, or change normal relay and maintenance-command behavior. Validated with cargo fmt --check, targeted unit and startup-isolation integration tests, all-target/all-feature clippy with warnings denied, deterministic process-output comparison, and jq schema assertions. |
||
|
DanConwayDev
|
3f8157693f |
docs: point GRASP references at Nostr Git
Move current GRASP specification links from GitHub to GitWorkshop. Link audit output to its exact pinned specification commit. Use NIP-05 cloning for ngit-grasp and ngit.dev for the ngit homepage. Leave the archived migration link unchanged as a historical record. Validated with rustfmt and the 54-test grasp-audit library suite. |
||
|
DanConwayDev
|
fd7c6bb851 |
refactor(auth): make current maintainer authority explicit
The v3.0.1 authorization fix is intentionally small. Follow it with a separate structural pass so the implementation and documentation express the present-tense maintainer model directly instead of leaving the security behavior hidden behind owner-oriented names and repeated raw-tag interpretation.
Parse indexed roles once into a current-only snapshot of active maintainers, active lead targets, and announcement-author activity. Preserve detailed lead-resolution failures internally while policy callers continue to fail closed, distinguish selected authorization coordinates from physical owner views, and name broad announcement admission as discovery rather than authority.
Keep history relevant only while deriving current activity and retain active leads only for selected-coordinate resolution. Preserve the v3.0 public API through compatibility projections and deprecated aliases; this commit is not intended to change the authorization outcome established by
|
||
|
DanConwayDev
|
650cfb57ee |
fix(auth): root state validity at active lead
The v3 role-tag work filtered invited and ended listed maintainers, but every announcement author still seeded a repository view. A removed forwarding maintainer or an author with an ended self-role could therefore keep publishing authoritative kind 30618 state through their own coordinate. Parity-only role parsing could also treat malformed histories as active. Validate numeric role boundaries and the final-end-only defer sentinel, resolve each selected coordinate through one valid active M path to a terminal self-M, and only then seed the reciprocal membership fixpoint. Legacy and explicitly leadless views continue to root at the selected author while that author remains active; incomplete, ambiguous, and cyclic explicit paths fail closed. This changes current state and maintainer-scoped authorization only. Historical event authorization and membership mutation workflows remain outside the relay-side patch. Validated with cargo test --lib (902 tests), cargo test --test state_authorization (53 tests), and cargo clippy --tests -- -D warnings. |
||
|
DanConwayDev
|
c1ebbf1aae |
feat(cli): export deterministic documentation schema
Documentation builds need a source-owned view of ngit-grasp commands and runtime configuration without copying Clap help or operator defaults into another repository. Add an early hidden __docs-export dispatch that serializes schema-v1 command metadata, deduplicated public environment options, runtime constraints, dotenv behavior, and relay-owner secret precedence. Clap remains authoritative for reflectable facts; a narrow explicit registry covers validation and the skipped secret. Correctness assumes Config::validate relationships continue to be mirrored in the constraint registry when they change. Secret source metadata intentionally contains locations and precedence only, never loaded values. This does not publish artifacts, generate ngit.dev pages, expose internal test switches, model conditional Clap requirements that stable reflection cannot expose, or change normal relay and maintenance-command behavior. Validated with cargo fmt --check, targeted unit and startup-isolation integration tests, all-target/all-feature clippy with warnings denied, deterministic process-output comparison, and jq schema assertions. |
||
|
DanConwayDev
|
527bfeced5 |
docs(storage): distinguish legacy incomplete histories
Motivation: Production v3 rehearsal found unmarked missing ancestors that the previous shallow-fetch wording could misattribute to a narrow pre-release bug. Approach: Describe incomplete legacy object graphs without inferring provenance, retain the shallow marker as a specific historical case, and document a minimal maintainer-bundle recovery procedure. Remove stale test comments claiming current purgatory fetches are shallow. Correctness: Current fetches have no depth limit, migration logged the affected graphs as pre-existing, and an unmarked missing parent is not evidence of the marked January fallback behavior. Excluded scope: No integrity policy, object import automation, migration behavior, or production data is changed. Validation: Ran git diff --check and reviewed the production sweep evidence; tests were intentionally skipped because this commit changes documentation and comments only. |
||
|
DanConwayDev
|
c0a023c7f2 |
fix(sync): keep maintainer repository inboxes historical
The earlier commit in this PR made every accepted repository owner's and declared maintainer's read/unmarked inboxes ordinary persistent live repository sources. Combined with unbounded per-author relay lists this drove the soak's connection growth: merely owning a repository let an author's advertised inventory imply long-lived live subscriptions. The 2026-08-19 investigation showed one bot-owned repository contributing 983 relay URLs through exactly this path. Owner/maintainer repository scope is now historical-only. The live inbox_repositories tier and its merge into derive_targets are removed entirely, and the repository-scoped overlay moves onto the existing paced, byte-bounded mailbox history workers using each author's bounded, sanitized read/unmarked inbox selection. Repository coordinates and known roots are still probed there, so a root or status stored only on a maintainer mailbox is still discovered - on the recurring history cycle instead of a persistent subscription. Correctness assumptions: inbox_roots is the pre-existing root-author live tier and is deliberately untouched, so accepted root authors keep live coverage on their own bounded inboxes. Participant mailboxes were already history-only. public_repository_mailbox_scope still empties repository scope in private mode before any overlay is built, and the per-relay independent mailbox worker behavior is unchanged. Excluded scope: no changes to instance-own-relay or repository-advertised relay coverage, no event-triggered drains, negentropy routing, or scheduling changes; those remain follow-up work. Validation: cargo fmt, cargo clippy --all-targets (clean), cargo test --lib (884 passed), cargo test --test sync proactive_sync (4 passed; the owner scenario now proves the inbox is probed through the history path, that a later root still arrives via historical refresh, and that the inbox never enters ordinary live sync). |
||
|
DanConwayDev
|
9347394a2e |
fix(sync): bound canonical NIP-65 relay selections
One hostile kind 10002 list could previously contribute an unbounded number of mailbox targets, and extraction collapsed relays into HashSet<String> of raw RelayUrl::to_string() output. Raw trailing-root-slash keys never matched the canonical connection map, so those relays occupied inventory while their mailbox probes deferred forever (2026-08-19 investigation, sections 4-5 and 8: 389 of 1,648 unique corpus strings were stalled root-slash variants, and one bot author supplied 983 relay URLs). Extraction now preserves published tag order, canonicalizes each accepted URL with the existing canonical_relay_key semantics, deduplicates by canonical key (widening disagreeing duplicate markers to unmarked, preserving NIP-65 union semantics from a single slot), and takes at most four relays per author and purpose: read/unmarked for live inbox coverage, write/unmarked for identity refresh, and write/unmarked preferred with read-only backfill for participant history. The ceiling is a hard internal constant, not configuration: NIP-65 already guides users toward two to four relays per category and this is a defensive resource boundary. Correctness assumptions: every downstream mailbox key space (author_inboxes, author_mailboxes, overlays, probe scheduling, cursor and due-time maps) is populated exclusively from these extraction functions plus the already canonical operator fallback set, so canonicalizing at the extraction chokepoint makes every key space consistent with the canonical connections map. Selections stay deterministic because hygiene, canonicalization and dedup all run before the four-slot cut in published order. Excluded scope: no shuffled rotation, liveness weighting, NIP-66 ranking, host-level identity merging, whole-list rejection, or operator setting. Owner/maintainer live-versus-historical placement is unchanged here and is addressed by the next commit. Validation: cargo fmt, cargo clippy --all-targets (clean), cargo test --lib (885 passed), cargo test --test sync proactive_sync (4 passed, including the root-author scenario now advertising its inbox in trailing-root-slash form to prove the previously stalled variant reaches its connection end to end). |
||
|
DanConwayDev
|
9d68414912 |
fix(sync): keep repository inboxes live
Repository owner and declared-maintainer read inboxes are authoritative sources for roots that may never reach a repository relay. Serving them only through a periodic history rotation leaves newly published roots and root-only status events unnecessarily stale. Map each accepted public Full repository into its owners and maintainers read or unmarked NIP-65 inbox targets, and merge every locally known repository root into the same target. The ordinary GRASP-02 target pipeline therefore keeps canonical repository and root filters in EssentialCore coverage, while compatibility variants remain ahead of recursive descendant fan-out. Write-only outboxes and non-root participant mailboxes retain the bounded history probe. This assumes NIP-65 read and unmarked relays are suitable inbox sources. Private mode continues to suppress repository coordinates, normal admission and persistent deletion tombstones remain authoritative, and this commit deliberately does not add permanent participant or write-only mailbox subscriptions or introduce cross-relay precedence over root-author inboxes. Validated with the full 819-test library suite, every Sync+ integration scenario, the complete sync integration binary (102 passed, one ignored), cargo fmt --check, and Clippy across all targets and features with warnings denied. Two unrelated timing tests failed once under the full workspace load, then passed individually and in the complete normal-parallel sync binary rerun. |
||
|
DanConwayDev
|
39149f93f8 |
fix(sync): backfill repository mailbox history
Historical repository roots and root-only descendants can exist on an accepted owner or maintainer's NIP-65 mailbox without appearing on a declared repository relay. Root-participant provenance alone cannot discover that first root or query a known root against an unrelated maintainer mailbox. Associate accepted Full repositories with their owners and declared maintainers, derive a repository-scoped history overlay from retained relay lists, and add every known root in that repository to the mailbox rotation. Reuse the existing byte-bounded, paced, single-flight fetch path and reschedule when root inventory grows. Correctness depends on repository admission remaining authoritative and on every fetched event continuing through the normal write policy, persistence pipeline, and global NIP-09/NIP-62 tombstones. Participant-only mailboxes retain exact root provenance. Deliberately excluded are permanent maintainer live subscriptions, repository-coordinate expansion in private mode, and any change to deletion or retention behavior. Validated with nix develop -c cargo test, nix develop -c cargo fmt -- --check, and nix develop -c cargo clippy --all-targets --all-features -- -D warnings. |
||
|
DanConwayDev
|
d0a23caf5c |
feat(nip34): parse indexed M/m role tags as the primary maintainer listing
Follow the indexed repository roles format from NIP-34 (nips 986edd1): `M` (lead) and `m` (co-maintainer) tags are now the primary maintainer listing, and their presence means the deprecated `maintainers` tag is ignored entirely. The lead / co-maintainer distinction carries no meaning for this service, so both collapse into one maintainer set. Role tags may record history as alternating start/end timestamps; a tag is currently active when it has fewer than four elements or an odd number of elements. Ended entries are ignored entirely: role history is only consulted to conclude that a pubkey is no longer a maintainer, never to grant time-scoped retroactive authority over historic events. A pubkey may appear in one `M` and one `m` tag to record a role transition and remains a maintainer while either entry is active; a second tag under the same letter is malformed and rejects the announcement. RepositoryAnnouncement::listed_maintainers() - already the single source for the listed maintainer set since the reciprocal-membership commit - now prefers active role-tag entries over the deprecated tag, so state authorization, replacement detection, the maintainer exception, sync discovery and the dependency walkers all pick up the new format through the sites switched to it here. Two refinements to membership follow from the format: - An announcement using role tags acknowledges its author via an active self-entry, or implicitly: per NIP-34 an author who appears in no role tag is a maintainer for the repository's entire history. Only an ended self-entry means the member left, which takes precedence over assignments in other announcements and is distinct from merely being invited (author_has_left). - A `u` (subordinate fork) tag has no effect on maintainership: the author of a role-less announcement asserts maintainership with or without it. Correctness assumption: authorization remains namespace-scoped, so the owner of a repository namespace stays authorized for it regardless of their own role history; role history only ends the authority of listed maintainers. Conflicting listings across announcements resolve as the union of confirmed members' active listings, matching the NIP's current-role rule; the NIP's owner-first precedence applies only to conflicting records of past roles, which this service never evaluates. Scope deliberately excluded: the moderator role tag (`o`) is handled in a follow-up commit. Validation: nostr::events and git::authorization unit tests, state_authorization suite (including new role-tag acceptance and ended-role rejection tests) and the sync invitation tests all pass. |
||
|
DanConwayDev
|
96cb60501a |
feat(nip34): require reciprocal announcements before maintainer state is authorized
Follow the reciprocal membership rule from the refined NIP-34 maintainers model (nips 781590b): a pubkey listed as a maintainer is only *invited* until its own announcement for the same identifier lists back an existing confirmed maintainer. State events from invited maintainers are no longer authorized; previously any pubkey reachable through recursive `maintainers` expansion was authorized without ever acknowledging the role. compute_membership replaces get_maintainers_recursive as the single membership computation: a fixpoint over announcements that confirms a listed pubkey once their own announcement lists back a confirmed maintainer. The owner is always a confirmed maintainer of their own repository, since announcing a repository in their namespace is what creates it on this service. collect_authorized_maintainers keeps its signature so all state-authorization call sites pick up the new semantics unchanged. Invited maintainers' announcements are deliberately still fetched, accepted (maintainer exception) and walked by the sync dependency machinery - the reciprocal announcement is exactly how the relay notices an invitation was accepted. To complete that flow, an acceptance stored via the maintainer exception is now pre-saved and stored state events are re-applied (new reapply_stored flag on process_state_event) so the newly confirmed maintainer's latest state re-points the owner's repository without another push; without this the stored state short-circuited as a duplicate and the invitation flow stalled. Remove the dead pre-membership helpers (AuthorizationContext, find_latest_state_for_announcement, find_latest_authorized_state, is_latest_state) that encoded the superseded semantics and had no other callers. Tests updated: invited state is rejected until acceptance (tests/state_authorization.rs), the invitation sync test now asserts the owner's refs are withheld until the invitee accepts, and grasp-audit maintainer fixtures publish reciprocal announcements. Scope deliberately excluded: the indexed `M`/`m` role tags and the moderator role from nips 986edd1 land in follow-up commits; this commit changes membership semantics for the deprecated `maintainers` tag only. Validation: git::authorization unit tests, state_authorization suite, sync invitation tests and grasp-audit lib tests all pass. |
||
 DanConwayDev
|
d73f2a3276 |
Merge #4584cec7: Route Git traffic through identifier families
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsytpxwcu3rtwvqntrk64dnekh6xf9czgv8sedtd8rd95fsugc53xcwp9h3l PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 CoverNote: Completes the local identifier-family storage model after the prerequisite storage-primitives PR was merged. - Routes owner and `/prs/` reads, pushes, and proactive fetches through a shared `(object format, identifier)` object family. - Lets related repositories satisfy reachable SHA wants and advertises retained family base refs, avoiding repeat uploads of objects already stored by the server. - Migrates legacy repositories deterministically on launch while retaining rollback backups and preserving incomplete refs and readable objects. - Adds one permanent family integrity/healing engine for packs, object connectivity, view alternates, and ref targets. It fetches exact missing OIDs from clone URLs in accepted announcements through the existing hardened outbound path, rechecks the family, and logs unresolved damage at `ERROR`. - Runs that engine asynchronously after migration and exposes `ngit-grasp integrity-check --identifier <id> [--repair]` through a durable live-process request queue. Migration does not get a separate recovery subsystem: it performs the structural conversion, then hands the resulting family to the ordinary steady-state checker. Unindexed legacy packs remain in the rollback backup. Garbage collection, legacy backup archaeology, and S3 storage remain out of scope. Testing on gitnostr.com: the already-installed storage version makes structural migration a no-op, but the startup integrity pass still runs unconditionally, so this is a valid test of the permanent steady-state path. To prove remote self-healing, use a sacrificial identifier whose accepted announcement lists a second Git server containing the same reachable object; snapshot its family and views, move one verified loose object into quarantine, invoke `integrity-check --repair` or restart, and verify the repair log, restored object, `git fsck`, and a fresh clone. This does not re-test the first legacy-to-family transition; that transition should remain covered by the migration fixtures or a disposable pre-migration data copy. Do not remove the production migration marker to force a rerun. |
||
|
DanConwayDev
|
1ad06bae22 |
feat(http): support path-mounted GRASP services
Allow operators to share an authority by configuring a normalized public base path. Route HTTP, WebSocket, Git, and GRASP-06 traffic beneath that prefix and carry the path through service validation, generated relay identity, self-sync, NIP-11 icons, and landing links. Treat NIP-05 as a domain-root capability: path-mounted services neither expose the well-known document nor advertise NIP-05 in NIP-11, and generated owner profiles omit the nip05 field. Exact relay URL matching keeps sibling mounts distinct while clone URLs may descend below the configured prefix. This assumes reverse proxies preserve the configured public prefix. Proxy-side prefix stripping remains deliberately out of scope. Validation: - nix develop -c cargo test - nix develop -c cargo clippy --all-targets -- -D warnings - cargo fmt --all -- --check - git diff --check - nix-instantiate --parse nix/module.nix |
||
|
DanConwayDev
|
4834a11b54 |
feat(storage): activate local families during launch
Motivation: routing new Git traffic through identifier families is not a complete deployment boundary for an existing server. Legacy owner and /prs repositories must be converted before runtime recovery, synchronization, or HTTP handling can safely assume thin views. Approach: run a versioned, fsynced, restart-safe migration during RelayServer startup. Build a verified union containing every reachable and unreachable object for each identifier family, retain original repositories as rollback backups, atomically install exact refs/HEAD views, and make archive restoration plus purgatory revival preserve the family layout. Correctness assumptions: startup has exclusive process ownership of the Git root; validated identifiers are safe migration path components; a migration error fails launch closed; successful local receive finalization releases its family lease before recovery can re-enter it. Focused recovery tests cover crash resume, unreachable objects, archive restoration, deletion transitions, and revived views. Deliberately excluded: S3 dependencies, backend configuration, cache eviction, remote durability, garbage collection, and automatic backup deletion. Those remote-storage concerns remain one later opt-in PR. Validation: nix develop -c cargo check --all-targets; cargo test --lib git::migration::tests; cargo test --lib extending_soft_expired_announcement_recreates_a_thin_view; cargo test --test nip09_blacklist_ops; cargo test --test nip09_state_cascade; cargo test --test nip09_recovery; cargo fmt --all -- --check; git diff --check. |
||
|
DanConwayDev
|
f084f7e973 |
feat(storage): route Git traffic through identifier families
Motivation: owner repositories and GRASP-06 contributor routes currently store or copy the same Git objects independently, forcing clients to upload data the relay already has. Approach: create new repositories as thin ref views, direct receive-pack and proactive fetch writes into the shared identifier family, retain accepted tips, and expose family bases as anonymous receive negotiation haves. Existing legacy repositories deliberately remain self-contained until the startup migration layer lands. Correctness assumptions: repositories only share objects when their validated identifier and object format match. A process-wide per-family lease serializes object-producing operations, while the terminal receive-pack flush remains behind ref validation and post-push processing. Excluded scope: remote S3 durability, cache eviction, garbage collection, and conversion of existing repositories are separate stack layers. Validation: cargo check --all-targets; cargo test --test git_response_streaming; cargo test --test grasp06_pr_hosting; plus repository_creation, git_clone, and storage/purgatory targeted suites from the preceding review. |
||
|
DanConwayDev
|
1a2c3a8cfb |
test(sync): expose rejection ordering barriers
Two maintainer-reprocessing tests wait on per-event rejection diagnostics, but the logging severity merge demoted those records to debug while their relay fixtures retained the info default. The bounded waits therefore failed deterministically even though reprocessing remained healthy. Start only the two affected fixtures through the existing debug-enabled helper. This preserves production-like logging for unrelated tests and keeps the observable ordering barriers introduced to replace fixed sleeps. This assumes the rejection diagnostic remains available at debug. Production behavior, global test logging, cache configuration, and the stacked storage PR lint failures are deliberately excluded. Validated with both focused integration tests under the normal environment, cargo fmt --all -- --check, and git diff --check. |
||
|
DanConwayDev
|
1d55c69dc7 |
test(sync): isolate NEG proxy source traffic
The NEG concurrency proxy records counters across all client connections. Once live self-subscription became reliable, the permissive archive source discovered the proxy URL in its own announcement and ran negentropy through the same measurement point, making the test evidence include scenery traffic. Reuse the production-outbound-policy archive source fixture already used by REQ. Its loopback event-directed targets are rejected while the syncing relay bootstrap remains operator-configured and allowed, preserving end-to-end coverage and attributing proxy counts only to the relay under test. Correctness assumes this integration topology remains loopback-only. Deliberately excluded: production sync behavior and the REQ scenario are unchanged. Validation: nix develop -c cargo fmt --check; cargo test --test sync sync::neg_concurrency::startup_historic_sync_stays_within_relay_neg_concurrency_limit; cargo test --test sync sync::req_concurrency::startup_historic_sync_stays_within_relay_req_concurrency_limit. |
||
|
DanConwayDev
|
b50883d9f0 |
Merge #2f5ad1ca: fix(logging): align operational severity with actionab…
fix(logging): align operational severity with actionability nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsz7kk3e29ceup4fzccjxq48r6vxu4f3jd54anfurevch0vp6lfmvq9fke8q PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 PR description: Production evidence showed the default info stream was dominated by dependency chatter and peer-controlled detail: gitnostr emitted 1,008 per-repository discovery records in a 3,501-line window, the US relay emitted 232 in 972 lines, and the German relay emitted 2,288 discoveries plus 2,236 warnings in 11,472 lines. Rejected historical announcements dominated the warnings, while all three services were intentionally running NGIT_LOG_LEVEL=info rather than debug. This PR scopes bare levels to ngit-grasp with dependencies held at warn, while preserving explicit EnvFilter expressions. It moves individual repository, event, filter, WebSocket, HTTP disconnect, and missing-object probe diagnostics to debug; retains aggregate sync outcomes and capability fallback at info; retains transient cooldowns at warn; and keeps internal database, policy, service, and subprocess failures at error. Unsupported NIP-77 is reported once per connection rather than once per filter. Configuration docs, the NixOS module, the example environment, monitoring guidance, and changelog are updated together. Tests that use rejection diagnostics as an ordering barrier explicitly opt into application debug logging. A scheduler-sensitive checkpoint-expiry test encountered during the package build now simulates downtime through persisted state instead of a fixed sleep; production behavior is unchanged. Validation: - cargo fmt --check and git diff --check - focused logging, HTTP, Git, self-subscriber, capability-gate, downtime, and maintainer-reprocessing tests - all 795 library tests passed in the development suite - all nine maintainer-reprocessing tests passed sequentially after opting their fixtures into debug - the 2,500-event REQ concurrency stress test passed in isolation after an earlier host-contention timeout - nix build .#ngit-grasp --no-link passed, including the pinned-toolchain 795-test package gate |
||
|
DanConwayDev
|
36634dc326 |
fix(private-repos): restore self-subscription in private mode
The SelfSubscriber feeds the sync manager's repository index from the service's own accepted events. It dialled our own public WebSocket endpoint with an unauthenticated client, which a private instance's NIP-42 gate correctly refused: the gate runs in the HTTP layer, ahead of LocalRelay, and cannot distinguish our own process from any other anonymous dialler. Announcements accepted at runtime therefore never reached the sync index until a restart rebuilt it from the database, stalling proactive sync and the dynamic membership derived from accepted relay owners — worst on exactly the private-to-private mirroring GRASP-08 exists to enable. Attach the subscriber to the embedded relay in-process instead. A custom WebSocketTransport hands LocalRelay one end of an in-memory duplex pair and keeps the other, so the client gets an ordinary relay session with the same framing, subscription handling, and post-save broadcast, minus the listener, the auth gate, and the network round trip. This replaces the loopback dial in both modes: the public-mode feed is identical in content and strictly more reliable, and it removes a self-directed reconnect loop. Chosen over attaching the relay owner key as a NIP-42 authenticator plus adding the owner pubkey to the effective member set. That alternative works, but widens the member set and the authenticated surface to solve a problem that is not authentication: there is no remote party here. The in-process route needs no key, no membership entry, and no configuration, and nothing reaches the subscriber that the relay did not already accept and persist. Correctness assumptions: LocalRelay applies no NIP-42 or query policy of its own — private-mode access control lives entirely in the HTTP layer — so an in-process session is exactly a local session, not a bypassed remote one. Both ends speak raw WebSocket framing over the duplex with no HTTP upgrade, matching take_connection's Role::Server. The session consumes one connection permit, as the loopback dial did. The GRASP-08 regression test no longer restarts the relay over persistent LMDB: it publishes an announcement at runtime and asserts sync connections to both referenced relays, which is only possible if the live feed reached the index. Verified to fail against the previous implementation (60s deadline, no connections) and pass with this one. req_concurrency's source relay now applies the production outbound target policy. Its scenario lists a proxy URL in the announcement, and with a reliable live feed the source discovers that URL — a distinct host:port that happens to front itself — as an event-directed sync target and opens its own REQ traffic through the proxy, contending for a budget the test means to measure for the syncing relay alone. That behavior is pre-existing and was already reachable after a restart; only its timing changed. The policy keeps the source scenery without weakening the assertion. Deliberately excluded: neg_concurrency shares that topology but passes unchanged, so its fixture is left alone; the duplicate NIP-11 fetch between the pre-dial preflight probe and the post-connect hint fetch is untouched. Validation: cargo clippy --all-targets -D warnings; cargo test --lib (793 passed); cargo test --test private_mode --test sync --test outbound_policy --test purgatory_sync (255 passed, including the full sync suite under parallel load). req_concurrency's startup-burst test passed 4/4 isolated runs after the fixture change. |
||
|
DanConwayDev
|
8b69971593 |
Merge #0a6a0d75: feat(private-repos): authenticate outbound sync betwee…
feat(private-repos): authenticate outbound sync between GRASP-08 services nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsq56sdwh0cccm69uqln5ar58hf38xssqwpzs9fz87kfdtlv3x45qqr462cs PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 PR description: Implements the outbound half of GRASP-08 private-service support: how an instance behaves as a sync client toward relays that demand authentication or advertise GRASP-08. Policy: (1) every sync connection answers NIP-42 AUTH challenges with the relay owner key when available, on public and private instances alike; after successful authentication the refused subscription is retried once, and without an owner key authentication is skipped and auth-demanding subscriptions park immediately. (2) A restricted: rejection after valid authentication is terminal and parks subscription work via the existing policy-refusal machinery, so there are no retry storms. (3) A relay whose NIP-11 supported_grasps advertises GRASP-08 is recognised with a pre-dial NIP-11 fetch that re-runs the outbound target policy, keeping the SSRF gate intact: a public instance parks such a relay without ever opening the WebSocket (no AUTH exchange, nothing to launder), while a private instance treats it as a peer and attaches the GRASP-08 repository-root NIP-98 credential (60-second validity, minted fresh per git subprocess) to purgatory Git fetches from that host, both signed with the relay owner key. (4) Relays with missing or unreadable NIP-11 remain ordinary sync targets. (5) Membership tightening: the NIP-11 owner of a relay referenced by an accepted announcement is only admitted as a derived member when that relay also advertises GRASP-08; configured NGIT_PRIVATE_MEMBERS are unaffected. NIP-42 here is identification, not confidentiality: our pubkey is already published via NIP-11 and NIP-05, and a private instance authenticating outbound discloses its identity consistently with the public-discovery stance of GRASP-08. Also adds disclosure guidance for private services: security contact via NIP-11 contact and the NIP-05 root identity, reports as NIP-17 encrypted DMs on the public mailbox relays of maintainers, collaboration access by adding the reporter to NGIT_PRIVATE_MEMBERS, and deliberately no non-member submission lane. Validation: cargo fmt --check, cargo clippy --all-targets -D warnings, cargo test --lib (792 passed), and the private_mode, sync, outbound_policy, purgatory_sync, nip11_document and relay_identity integration suites all pass. |
||
|
DanConwayDev
|
bd371b17a4 |
test(logging): opt ordering fixtures into debug logs
Maintainer-reprocessing integration tests use the rejected event ID in the relay log as an observable barrier before they mutate reciprocal repository state. Demoting per-event rejection diagnostics to debug made those barriers invisible at the production info default. Add an explicit debug-level TestRelay option and enable it only for fixtures that observe rejection diagnostics. The ordinary fixture retains the production info default, while short-hot-cache fixtures select debug because all current callers rely on that barrier. This deliberately preserves the existing bounded log wait instead of adding sleeps or changing production severity. Tests that do not inspect debug diagnostics remain unchanged. Validated with cargo fmt --check, git diff --check, and all nine maintainer-reprocessing integration tests running sequentially. |
||
|
DanConwayDev
|
502c74c975 |
feat(private-repos): attach GRASP-08 credentials to private outbound Git fetches
A private instance mirroring another private service could open the authenticated WebSocket but had no way to fetch the git data behind the peer's NIP-98 gate, so purgatory promotion of mirrored repositories never completed. This is the missing half of zero-configuration private mirroring. Approach: a new Grasp08Peers registry (canonical host:port keys with explicit ports, since loopback tests co-host several services on one address) is populated by the sync manager whenever a session's NIP-11 advertises GRASP-08 and drained when it stops. The purgatory fetch path consults it per URL and, for confirmed peers, attaches the GRASP-08 repository-root credential via `http.extraHeader` on the hardened git command. Headers are minted fresh before each subprocess because the peer enforces a 60-second validity window that a long batch fetch must not outlive; signing failures warn once and degrade to unauthenticated fetches. Both the registry and the signing keys exist only when the server runs in private mode, so a public mirror can never present credentials. Correctness assumptions: `extract_domain` drops the port and is therefore unsuitable as the registry key; `peer_key` derives host:port from the URL itself on both the relay-URL (write) and clone-URL (read) sides. The `http.extraHeader` addition does not conflict with the `credential.helper=` hardening: that control excludes ambient operator credentials, while this header is a peer-scoped credential minted for exactly this vetted fetch target. Validation: unit tests cover the ws/http key equivalence and a credential round-trip through the inbound validator (including root derivation past `.git` inside identifiers). The integration test runs two private services end to end: the mirroring instance only serves the state event after NIP-42 on the WebSocket AND the NIP-98 credential on the git fetch both succeeded against the private source. cargo test --lib private:: and --test private_mode private_instance_syncs_from_private_peer_with_outbound_credentials pass. |
||
|
DanConwayDev
|
e3c3a73e6d |
feat(sync): make outbound NIP-42 authentication optional and terminal on refusal
Outbound sync answers NIP-42 challenges with the relay owner key on
every connection. When no owner key is available, the previous code
panicked at registration (`.expect`), and the retry machinery would
still have reserved a one-shot authentication retry that nothing could
ever fulfil.
Approach: `RelayConnection::new{,_with_database}` now take
`Option<Keys>` and only attach the SDK authenticator when present,
exposing `answers_auth_challenges()`. Without an authenticator an
auth-required CLOSED is terminal like any other CLOSED: the terminal
listener retires the subscription, the data lane releases its live
permit, and `handle_subscription_closed` skips the one-retry
reservation and goes straight to retirement plus the
AuthenticationRequired policy refusal (24h probe). `register_relay`
degrades gracefully to an unauthenticated connection with a warning
instead of panicking.
Correctness assumptions: rust-nostr only retains auth-refused
subscriptions for post-authentication resubscription when an
authenticator is configured, so every has_authenticator branch mirrors
an SDK behavior split; a reserved retry without an authenticator would
dangle until disconnect cleanup.
Test infrastructure: new AuthGatingRelay helper - a NIP-42 gate in
front of a backend relay that serves a plain NIP-11 document,
challenges every session, refuses queries pre-auth, marks negentropy
unsupported, and either bridges (Admit) or answers `restricted:`
(Restricted) after a valid AUTH, recording authenticated pubkeys and
REQ counts.
Validation: integration tests prove (1) a public instance
authenticates to a gated ordinary relay with its owner key and the
retained subscription is answered after AUTH (announcement reaches
purgatory through the gate, the instance's only event source), and
(2) a restricted refusal after successful authentication parks the
work - the gate's REQ count holds still for a full 2s observation
window. cargo test --lib (789 passed) and --test sync
sync::outbound_auth pass.
|
||
|
DanConwayDev
|
d36b1227cf |
feat(sync): keep GRASP-08 private services out of public sync
A public mirror gains nothing from dialing a GRASP-08 private service: the service will never admit it, and even attempting the connection performs an AUTH exchange with a relay that only wants members. Worse, retry machinery would hammer it indefinitely. Approach: public instances fetch the NIP-11 document before the WebSocket dial (`preflight_limit_hints`). When it advertises GRASP-08, the worker returns a new `ConnectAttemptOutcome::PrivateService` without dialing. The actor parks the relay in `private_service_relays` (warn once, stable message tests grep for), retires all sync state without the re-registration path, and both `register_relay` and `schedule_connect_relay` refuse parked targets thereafter. Private instances treat GRASP-08 peers as ordinary sync targets and skip the probe. Session limit hints deliberately keep coming from the existing post-connect fetch: reusing the pre-dial probe for hints would remove the post-connect setup window whose stale-success handling (disconnect during NIP-11 setup) is separately guaranteed and tested. Correctness assumptions: the pre-dial NIP-11 probe is an outbound TCP connection, so `preflight_limit_hints` re-runs the resolved outbound target policy for event-directed URLs and skips the HTTP request entirely on rejection - `connect()` then fails with the same policy rejection through its own gate (tests/outbound_policy.rs stays green). The park set is in-memory by design: a service that stops being private becomes reachable again after a process restart at the latest. Deliberately excluded: private-instance behavior toward GRASP-08 peers (NIP-98 credentials on git fetches) lands separately. Test infrastructure: `wait_for_log_line` moved from outbound_policy.rs into the shared sync helpers; TestRelay gained a sync constructor with identity publication disabled so log assertions about the bootstrap connection are not confounded by user-index traffic. SetupDropRelay now answers pre-dial NIP-11 probes directly and only runs its drop-during-setup choreography for a fetch that arrives during a live WebSocket session; the naughty-list scheduling test accounts for the probe as a second accepted connection on the first attempt. Validation: new tests/sync/outbound_auth.rs proves the park warning appears exactly once and that, across a 2s observation window, the private bootstrap relay is never connected to and no NIP-42 authentication occurs. cargo test --test sync -- sync::outbound_auth sync::stale_connect_result sync::naughty_list_scheduling and --test outbound_policy pass. |
||
|
DanConwayDev
|
6ef105d95f |
test(sync): replace remaining fixed sleeps with observable waits
Sweep the fixed sleeps left in the sync suite after the tag_variations / live_sync hardening, following the same pattern: wait on the condition the test actually cares about with a bounded deadline, so healthy runs are no slower and only the failure bound widens to 30 seconds. - discovery, historic_sync, metrics, maintainer_reprocessing: replace "wait for discovery/sync" sleeps with bounded polls on the synced event, a metrics counter/gauge, or the syncing relay's log (rejected announcements are observable by their note ID), and widen the short verification deadlines that followed them. - purgatory_fetch, live_sync, historic_sync: replace fixed post-connect sleeps on raw clients with `connect().and_wait(timeout)`. - metrics: poll the metrics endpoint for readiness instead of sleeping through relay startup. TestClient::send_event is reworked to send through its single tracked relay so the SDK error kind survives: transient transport failures (disconnect, timeout, not-connected) are retried with the existing bounded backoff plus a reconnect, while a relay OK-false rejection now fails immediately instead of being retried, so tests cannot mask write-policy regressions. A transient client error under load previously killed adaptive_pagination's seeding despite the retry loop, because rejections and transport failures were indistinguishable at the pool level. Deliberate time-as-behavior sleeps stay and are now commented: the one-second rejected-hot-cache TTL expiries, created_at spacing before +1s-dated events, and fixed observation windows for events that must NOT appear. req_concurrency is deliberately untouched: its proper fix needs a relay observable for descendant-sweep completion (issue b052fa70). Validated under CI's hostile global git config (GIT_CONFIG_GLOBAL mirror of the workflow step): three consecutive green runs of `cargo test --locked --test sync`, plus `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, and a full `cargo test --locked` (its sync pass green a fourth time; one unrelated load flake in live_sync_regroups_after_filter_count_refusal passed on isolated rerun and in the repeat full run). |
||
|
DanConwayDev
|
7dc786827f |
feat(private-repos): require GRASP-08 advertisement for derived membership
Derived private membership previously admitted the NIP-11 owner of any relay referenced by an accepted announcement. A public relay's owner gains nothing legitimate from private membership - their relay enforces no confidentiality for the repositories it mirrors - so minting access for them needlessly widens the trust domain. Approach: parse the GRASP `supported_grasps` extension array from the raw NIP-11 body (the SDK type does not carry it) into a new `RelayLimitHints::grasp08` flag, thread it through the connect-attempt outcome, and only insert into `relay_owners` when the relay advertises "GRASP-08". Absent or malformed documents mean "not a private service". Operator-configured NGIT_PRIVATE_MEMBERS are unaffected. Correctness assumptions: `relay_owners` is the sole source of derived members (`effective_private_members`), so gating insertion gates the whole derivation; removing a stale entry on a non-advertising session keeps a relay that stops advertising GRASP-08 from retaining minted membership past its next reconnect. Test infrastructure: MockRelay can now serve a caller-provided NIP-11 document, `push_to_relay` gained an Authorization-header variant for pushes to private services, and TestRelay gained a persistent-LMDB private constructor. The integration test restarts the relay after promotion because the private NIP-42 gate also applies to the internal self-subscription, so locally published announcements only reach the sync manager through the startup database load; that pre-existing limitation is out of scope here. Validation: new unit tests for the supported_grasps parse; integration test proves a GRASP-08-advertising relay's owner is admitted while an otherwise identical non-advertising relay's owner stays restricted. cargo test --lib grasp08_flag and --test private_mode derived_membership_requires_grasp08_advertising_relay pass. |
||
|
DanConwayDev
|
b5a97e5df2 |
test(sync): wait on observable conditions with realistic deadlines
The tag_variations and live_sync suites waited for cross-relay discovery with a fixed one-second sleep before publishing events, and verified sync with five-second polls. On a loaded runner discovery can take longer than the sleep, and events published before the syncing relay subscribes are only recovered by later passes that miss the short verification window. CI hit this on this PR's first run (tag_variations uppercase-A), and different members of the same family fail locally on different runs under load. Discovery waits now use the existing wait_for_sync_connection helper (bounded poll of the syncing relay's metrics endpoint) instead of a sleep, the ordering test polls for each event instead of sleeping three seconds, and verification deadlines widen to 30 seconds. Bounded polls return as soon as the condition holds, so healthy runs are no slower; only the failure bound moves. Validated by running the sync suite repeatedly under a hostile git config on a heavily loaded machine: the previously flaky tests pass in every run. req_concurrency also flakes under such load at its phase-1 quiescence wait, but restructuring it needs an observable for descendant-sweep completion that the relay does not yet expose, so it is deliberately left unchanged (it has not flaked in CI). Formatting fixes carried earlier in this branch were dropped: master picked up equivalent rewraps via the relay-identity merge and is fmt-clean again. |
||
|
DanConwayDev
|
7334e04b88 |
feat(identity): publish relay owner events
The relay-owner key is intended to double as the service identity used by ngit-ci, but clients could only discover it through HTTP and owner-authored coordinator events without repository roots failed admission. Sign a minimal kind-0 NIP-05 bot profile at startup — per NIP-24 `name` is always set, here to the scheme-less public URL — alongside a single unmarked kind-10002 relay entry. An operator-customized profile is kept and never overwritten, and no identity event — locally stored or freshly generated — is published before the local database and at least one user-index relay have been successfully checked for that kind, so a database wiped and reseeded during an index outage can never displace a customized profile surviving on the indexes. Every send is preceded by a per-relay re-check: an identity found on an index relay is adopted locally, where replaceable-event semantics keep the newest copy, and is never overwritten, so publication only fills gaps on index relays that individually confirm they hold none; propagating a profile update onto an index that already has one is left to the operator's own client. A kind with no local copy is not even seeded until a reachable user-index relay confirms it holds no identity of that kind. Publication never blocks startup, retries transient failures with a capped backoff, and stops on terminal protocol rejections. With no user-index relays configured, missing kinds are seeded locally right away. In private mode (GRASP-08) identity events are seeded and served locally but never published, so a private relay does not advertise its existence. Trust valid owner-signed events only for kinds without a dedicated admission policy, such as ngit-ci coordinator advertisements that carry no repository root tag. Owner-signed NIP-34 announcements, state events, and PRs run the normal announcement validation, ref alignment, and purgatory git-data handling like any other author, and the relay's own kind 0/10002 identity is always accepted. The NIP-09/NIP-62 deletion gate still runs before any owner acceptance, so replaying a retracted owner event cannot undo its tombstone, and owner deletion/vanish requests keep their lifecycle handling. Because the event blacklist cannot block the owner key, rotating the key is the only remediation if it is compromised; this is documented. NGIT_DOMAIN is assumed to be the documented bare public authority: loopback authorities use ws and other hosts advertise wss, with bare IPv6 authorities bracketed. The test fixture reuses relay-owner keys across TestRelay::restart, and gains caller-provided keys, a private-mode member setup, and explicit user-index relay lists; MockRelay can start pre-seeded so a "recovering" index deterministically holds prior state. Identity retry intervals honor the existing NGIT_TEST fast-timer convention. No new configuration switches or ngit-ci changes are included. Includes rustfmt fixes for src/nostr/policy/announcement.rs and tests/private_mode.rs, which arrived on master unformatted and would otherwise fail the workspace format gate. Validated with rustfmt, strict workspace Clippy, the relay_identity and private_mode integration binaries, and the full workspace test suite. Individual sync/grasp06 integration tests fail intermittently under parallel full-suite load, each passing in isolation and on rerun; the same intermittent failures reproduce on origin/master without these changes. |
||
|
DanConwayDev
|
a3bfd023b2 |
feat(private-repos): gate announcement admission on membership
In GRASP-08 private mode, accepted kind-30617 announcements expand hosting and — via their referenced relays' NIP-11 owners — the effective member set itself. Admission never checked the announcement author, so a member could submit a third-party-signed announcement (or sync could import one from an operator-configured source) and mint membership for pubkeys no member ever chose. AnnouncementPolicy now rejects a repository announcement whose author is not a current effective member, using the shared PrivateAccess set (configured members plus admitted relay owners) threaded from server startup through create_relay and Nip34WritePolicy. The check runs at the top of AnnouncementPolicy::validate, the single choke point every announcement arrival path (direct publish, sync import, purgatory entry) funnels through, and uses the existing AnnouncementResult rejection machinery so operators observe these rejections like any whitelist rejection. Public-mode behavior is unchanged (the gate is None outside private mode) and no configuration was added: the gate is implied by NGIT_PRIVATE_MODE. Correctness assumptions: membership is evaluated against the live set at admission time via PrivateAccess::contains, not re-derived; state events (kind 30618) stay governed by GRASP-01 maintainer rules; and removal is non-retroactive — repositories admitted while their author was a member remain hosted until the operator curates them. Deliberately excluded: outbound authentication when syncing from other private services, which remains a follow-up in the design doc. Validation: cargo clippy --all-targets -D warnings clean; lib tests (783), private_mode (52, incl. new member/non-member admission integration test), nip34_announcements (60), repository_creation (47), and purgatory (55) suites all pass. |
||
|
DanConwayDev
|
c169658096 |
test(private-repos): broaden private-mode integration coverage
The GRASP-08 boundary previously had end-to-end coverage only for the HTTP repository-root credential. Exercise the remaining externally observable contracts through the subprocess fixture: - NIP-42 WebSocket lifecycle: challenge on connect, auth-required rejections for REQ/EVENT/garbage before authentication, member admission that bridges through to a served subscription, immediate restricted-close for valid non-member credentials, and the bounded three-attempt budget for invalid ones. - NIP-98 credential scope: expiry outside the 60-second window and subpath-scoped credentials both fail with the indistinguishable empty 401, while one repository-root credential is reusable across Smart HTTP endpoints. - Discovery: the NIP-11 document stays unauthenticated and advertises NIP-42/98 so clients can learn the requirement without credentials. Deliberately excluded: dynamic accepted-relay-owner admission still has only unit coverage; an end-to-end scenario needs a second announced relay fixture and is left as follow-up scope. Validation: cargo test --test private_mode (51 passed) and cargo clippy --all-targets -- -D warnings on the rebased branch. |
||
|
DanConwayDev
|
84096df0a1 |
test(private-repos): exercise the HTTP authentication boundary
The private-service implementation spans command-line configuration, HTTP routing, canonical repository identity, and credential validation. Unit tests of those pieces cannot prove that the production subprocess composes them into a fail-closed endpoint. Extend TestRelay with an explicit private-service constructor and exercise the real process over HTTP. The scenario verifies that missing and non-member credentials receive the same empty Nostr 401 challenge while a configured member crosses the authentication boundary. A nonexistent repository is intentional: authentication is the behavior under test, and successful authorization must remain distinguishable only by proceeding to ordinary Git routing. NIP-42 protocol framing remains covered by its focused state-machine tests; duplicating that protocol through an SDK client is excluded from this process-level HTTP regression. Validation: cargo fmt --check passed; cargo test --locked --test private_mode passed (45 tests including the new subprocess scenario). |
||
|
DanConwayDev
|
18eddeedb5 |
Merge #4e732958: feat(sync): discover participant mailboxes
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsyuueftzc68d5a3x5d0pecs66y0nacv4pc87kkcfw499adyl5t3ycxqgtta PR-Author: DanConwayDev's Agent nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0 CoverNote: ## What this adds Repository conversations can continue on relays used only by a non-root participant. For example, a reaction to an issue may be stored on one relay, while a reply to that reaction exists only on the reaction author's NIP-65 write relay. Root-author relay discovery and recursive reference queries cannot find that reply unless the participant's mailbox is also searched. This PR adds bounded, historic participant-mailbox coverage: 1. Preserve exact repository-root provenance while deriving the accepted recursive descendant frontier. 2. Treat authors of accepted replies, comments, reactions, zaps and other descendants as thread participants. 3. Accept kind `10002` relay lists only for repository owners, maintainers, root authors and those accepted participants. 4. Derive each participant's NIP-65 read, write and unmarked mailbox relays. 5. Query those mailboxes only for the repository roots and accepted descendant IDs associated with that participant. 6. Pass every returned event through the ordinary write policy, deletion/replacement rules and persistence pipeline. This discovers replies to reactions and other indirect descendants without fetching a participant's unrelated notes or trusting their relay list to bypass repository policy. ## Bounds and scheduling - The existing `NGIT_SYNC_RECURSIVE_DESCENDANT_LIMIT` bounds which indirect descendant IDs remain query roots. Each direct event that tags a repository root has its own bounded branch; the default is 500. - Once a branch reaches that bound, later indirect descendants cannot add more participant authors or query roots through that branch. - Exact root provenance prevents roots from one repository leaking into another participant query. - Participant mailbox coverage is historic only. It does not add permanent non-root participant subscriptions or expand the existing live-sync tier. - Filters use the existing byte-bounded grouping and paginated `RelayConnection::fetch_events` path, including request pacing, background priority, subscription-ledger capacity and EOSE/CLOSED handling. - At most one mailbox worker runs per relay, and at most one new due relay is started per maintenance pass. - Relays progress independently: there is no global mailbox lane, cross-relay success condition or shared completion counter. ## Restart and failure behavior - A probe starts only after both the WebSocket and the sync actor's connection lifecycle are ready. - Ready due relays are preferred, so old unreachable mailbox sources cannot starve connected work. - Completed mailbox workers are handled before new connection results, so a large startup connection queue cannot delay cursor progress or resource release. - Successful filters advance an in-memory, stable-sorted cursor immediately. Completing the last filter schedules the next historic rotation after 24 hours. - Failed filters release their relay worker and retry after five minutes without blocking other relays. - On restart, accepted roots, participants and kind `10002` ownership are reconstructed from LMDB. Filter cursors deliberately restart from the first current group; they are best-effort coverage, not durable exactly-once state. The implementation reuses the existing relay transport. It does not add a mailbox-specific protocol state machine, pending-batch purpose, CLOSE API, NIP-42 retry system, watchdog or connection-queue coordinator. ## Coverage and production evidence The audit motivating this change found 72 events visible through gitworkshop but absent from the pre-change production relay across the `gitworkshop` and `ngit` repositories. The persistent canary served 29 exact misses: six kind-1 notes, three reposts, ten reactions, six comments, two issues, one zap receipt and one repository-follow event. The exact candidate `de5fa6625aa5693afcde7335a572456680aef9df` activated on the isolated archive at 2026-08-14 19:23:49 UTC. Restart reconstruction found 4,291 accepted roots, 403 participant authors and 351 mailbox relays. During the recorded soak: - 63 mailbox filters started and all 63 reached terminal handling: 59 successes and four bounded failures. - Failures received the intended five-minute retry and did not block successful progress on other relays. - `relay.ngit.dev`, `nostr.land`, `nostr.mom`, `haven.danconwaydev.com/inbox` and other relays advanced independently. - `nostr.azzamo.net` completed all six filter groups; the final terminal reported `completed_cycle=true` and scheduled the next probe in 86,400 seconds. - The archive remained active with zero restarts and no process panic or fatal error. The temporary archive test override of `NGIT_SYNC_RECURSIVE_DESCENDANT_LIMIT=2` has been removed. Both the archive and public gitnostr.com have no override and therefore use the default of 500. Public gitnostr.com was not restarted or changed by the archive deployment. ## Validation and review state - Exact tested head: `de5fa6625aa5693afcde7335a572456680aef9df`. - One commit; 1,157 additions and 140 deletions across 11 files. - 767 library tests passed in the exact Nix release build locally and on the production host. - All three proactive Sync+ integration scenarios pass together, including a child found only through a participant reaction author's write mailbox. - Strict all-target Clippy, formatting, diff checks, Nix flake evaluation and a conflict-free merge-tree check against current `master` passed. - One remote build attempt hit the pre-existing `test_entries_expired_during_downtime` timing flake; its source is unchanged by this PR, and the unchanged candidate passed all 767 tests on retry before activation. Recommended for merge. The archive demonstrates successful and failed terminal paths, per-relay independence, restart reconstruction, cursor advancement and a complete historic-to-24-hour-refresh rotation under real startup load. |
||
|
DanConwayDev
|
de5fa6625a |
feat(sync): probe participant NIP-65 mailboxes
Repository conversations can continue in the read or write relays advertised by authors of accepted replies, reactions, zaps, and other descendants. Root-author inbox discovery alone therefore leaves valid indirect descendants undiscovered. Carry exact accepted-root provenance through the existing bounded descendant frontier, admit identity events only for those participants, and derive read/write mailbox ownership from accepted kind 10002 events. Probe one byte-bounded historic filter at a time with the existing per-relay fetch_events pagination, pacing, ledger, timeout, policy, and persistence paths. Starts are paced globally, but progress and terminal state remain independent per relay. Require both an established socket and the sync actor committed lifecycle before starting a probe. Prefer lifecycle-active due relays while falling back to the existing oldest-due dial order, so hundreds of unavailable mailbox sources cannot starve already-ready work. Drain completed mailbox probes before accepting more connection results so a busy startup queue cannot delay cursor progress or resource release. Production canaries exposed these startup conditions without requiring cross-relay coordination. The recursive descendant limit bounds which indirect IDs remain query roots; direct root references remain complete. Mailbox filter cursors are intentionally best-effort in-memory state: restart reconstructs ownership from LMDB and safely begins historic coverage again. This does not add permanent participant live subscriptions or a cross-relay completion coordinator. Validated with cargo check, strict all-target Clippy, 767 library tests including lifecycle and ready-selection regressions, and the three proactive Sync+ integration scenarios, including a write-mailbox child that references only a participant reaction. |
||
|
DanConwayDev
|
7bca6e2482 |
feat(http): serve NIP-05 root identity
Relay operators already have a stable Nostr identity for NIP-11 and NIP-42, but clients cannot resolve the domain itself to that key. Serve the NIP-05 root identifier so _@domain identifies a root-mounted relay without another key source. Add an exact root-level /.well-known/nostr.json route for GET and HEAD, serialize the relay-owner public key as lowercase hex under names._, retain permissive CORS, and give this route precedence over NIP-11 content negotiation. Derive NIP-11 support from the relay request path: only / advertises NIP-05, while nested relay paths omit it. Keep the changelog, architecture, README, and relay-owner configuration descriptions synchronized. The path check assumes the HTTP request path is the relay's public mount path; a future explicit base-path setting should remain the source for this decision if proxy routing hides that path. Named identities and recommended relay-hint mappings remain outside this change. Validated with 760 library tests, the root/non-root NIP-05 server integration test, strict all-target Clippy, cargo fmt, git diff checks, and Nix module parsing. |
||
|
DanConwayDev
|
9360d419b3 |
feat(sync): bound recursive related frontiers
Repository conversations can continue through parent-only event and address references, but recursively following an unbounded social thread can pull unrelated activity into proactive sync. Extend the existing descendant rotation transitively while bounding each subtree rooted at an event that directly tags a repository root. Derive branch membership deterministically from the accepted local database on each reconciliation tick. A branch below the configurable frontier limit contributes its event IDs and coordinates to the existing live-or-REQ+EOSE machinery; a full branch contributes no further child-query seeds. Rebuild auxiliary live coverage when the frontier shrinks so stale child filters are actually closed. Because LMDB is the checkpoint, restart needs no new durable state and source relay loops remain independent. The default limit is 500 and direct root-tagging events do not consume it. The limit is intentionally soft: replies to requests already in flight can still be stored, but cannot extend a full branch. The existing eight-generation traversal bound remains. Exact receive-time admission and cross-relay coordination are deliberately excluded. Validated with 758 library tests, six focused frontier tests, a limit-2 historic/restart integration test, the serialized sync suite (95 passed, 1 ignored, 2 unrelated timing failures that passed isolated reruns), strict workspace Clippy, cargo fmt, and nix flake check --no-build. |
||
|
DanConwayDev
|
e84be42446 |
feat(sync): cover roots with missing relay lists
Some accepted root authors do not publish a discoverable NIP-65 kind 10002, leaving their conversations outside Sync+ even after a successful user-index lookup. Use a bounded operator-configured fallback relay set only after such a successful empty lookup. Feed fallback roots into the existing GRASP-02 target overlay, so historic, live, rotating, subscription-budget, and recovery behavior remain unified. A later accepted relay list removes the author from desired fallback coverage while shared live subscriptions drain naturally. Keep eligibility restricted to accepted root authors and preserve single-flight identity discovery. This does not expand to response authors, add another scheduler, or infer fallback need from failed queries. Validated with the end-to-end missing-list fallback and later-list replacement scenario, discovery/config unit tests, Nix module parsing, rustfmt, clippy with warnings denied, and diff checks. |
||
|
DanConwayDev
|
e4d51692f2 |
style(test): remove needless URL borrows
Apply the remaining Rust 1.96 clippy suggestions in live-sync tests. The metrics helper already accepts string references from TestRelay directly, so borrowing those references again is redundant. This is a mechanical test-only lint repair with no behavior change. Validated by the clean workspace clippy run in the preceding commit. |
||
|
DanConwayDev
|
2ceb81595e |
style: satisfy the workspace clippy gate
Apply current Rust 1.96 clippy repairs across recent sync code and scope test mutex guards before async cleanup. Keep the test-launch outcome unboxed because it is short-lived control flow and boxing every successful relay fixture would complicate the shared harness solely for enum layout. These changes are lint-oriented and do not alter production behavior. The targeted allowance documents the intentional test-helper tradeoff. Validated with cargo clippy --locked --workspace --all-targets -- -D warnings. |
||
|
DanConwayDev
|
f78390f7e0 |
test(sync): allow serialized bounded transient requests
The connection-wide ledger may serialize transient REQ rounds under full-suite load. As with negentropy, overlap is not a correctness guarantee; requiring a peak of two made the exact CI command fail despite complete delivery and zero proxy rejections. Retain the meaningful workload and safety assertions: all sampled events arrive, total requests exceed the configured bound, zero requests are rejected, and peak occupancy never exceeds the bound. Observed under the exact cargo test --locked run after every other sync scenario passed. |
||
|
DanConwayDev
|
d324a320a8 |
test(sync): allow serialized bounded negentropy rounds
The connection-wide request ledger and paced descendant rotation may serialize historic negentropy rounds. The integration scenario still required overlap, even though overlap is not a correctness guarantee and the newer scheduling model deliberately permits a peak of one. Keep the safety and workload assertions: all sampled events must arrive, the proxy must reject zero rounds, peak concurrency must stay within the production bound, and total rounds must exceed that bound. Unit coverage continues to exercise concurrent ledger admission directly. Validated by the focused scenario through every assertion except the removed overlap-only lower bound. |
||
|
DanConwayDev
|
9b49056a14 |
fix(sync): schedule auxiliary coverage for full repositories
Priority demotion moved q-tagged and other non-essential repository references out of core live filters, but descendant maintenance selected only relays that already had root events. A full repository with no accepted roots therefore never installed its auxiliary repository coverage, breaking bounded-REQ and adaptive regrouping delivery. Include relays with full repositories in the bounded maintenance frontier. Update the constrained-REQ regression to wait for the observable auxiliary generation before using a q-tagged event as its live-delivery probe. State-only repositories remain excluded, and maintenance still advances only one selected relay per tick. Validated with the bounded-REQ and filter-count-refusal integration tests. |
||
|
DanConwayDev
|
e3e0cfd3a5 |
test(sync): wait for full live coverage before metric delivery
A connected source socket and a fixed delay do not prove that purgatory promotion has installed full-repository filters. The live-event metric test could publish both patches during state-only coverage and correctly observe no live sync increments. Wait with a bounded deadline for the full-repository AddFilters generation to complete its core live extension before publishing. This keeps the metric assertion tied specifically to live delivery. Validated with the focused live sync event-count integration test. |
||
|
DanConwayDev
|
b3938c84dd |
test(sync): allow bounded descendant catch-up to complete
The pre-subscription Layer-3 scenario intentionally publishes a comment before its root has been confirmed. Confirmation can consume a historic batch and the descendant fallback advances on a five-second maintenance tick, so the former five-second deadline could expire before eligible work was scheduled. Retain the pre-existing-event recovery contract and extend only its bounded observation deadline. The test still requires the comment and its exact parent reference to be recovered. Validated with the focused Layer-3 live-sync integration test. |