laantungir/ngit-grasp
1
0
Fork 0
mirror of https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git synced 2026-09-14 04:45:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,192 Commits 6 Branches 13 Tags
master
Commit Graph
68 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
DanConwayDev d71d888019 Merge #9698b1f9: test: eliminate shared fixture races and timing-depend… 
test: eliminate shared fixture races and timing-dependent assertions

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsfdx93lyfyyuvw79qg8qe4eec4gjxkqasu3glrrsljhdgle3hw5fqtx4mfz

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Parallel tests could lose released relay ports, leave connections alive after fixture shutdown, block Git children on full pipes, or infer readiness from scheduler-dependent sleeps. This revision extends the existing relay-port PR into one consolidated reliability proposal for Grasp.

- Retain SDK relay listeners through startup and exercise 32 concurrent fixtures.
- Transfer reserved loopback sockets into Grasp subprocesses through a private Unix test protocol, retain the address across restart, and keep offline/recovering endpoints continuously owned.
- Own HTTP, WebSocket upgrade and backend-forwarding tasks across the mock relay, auth gate, Git servers and proxy fixtures. Explicit shutdown cancels and joins requests; Git subprocess I/O drains concurrently.
- Wait for event visibility and actual connected-state metrics; correct metrics URL construction and stopped-source bookkeeping.
- Replace expiry scheduling margins with explicit timestamps and streaming sleeps/frame assumptions with observable gates and byte-level assertions.

Tests remain enabled and parallel. Normal server binding and service configuration are unchanged; inherited listener support requires NGIT_TEST=1 and validates a listening loopback socket. The private capability probe lets the companion ngit harness retain compatibility with older releases.

Targeted Linux validation in the repository development shell: 65 relay-connection tests, 33 rejected-index tests, 132 purgatory tests, all 3 streaming tests, 7 shared Git-server tests, and relay-identity/recovery tests passed. Final fixture_lifecycle passed 57 tests; the moved subprocess-restart regression passed separately. Scoped Clippy for fixture_lifecycle and relay_identity passed with warnings denied. A smoke test queried this Grasp binary using the modified ngit harness. Formatting and whitespace checks pass.

These checks are scoped, not a claim of full workspace, nixpkgs sandbox or Darwin validation. Full host builds remain outstanding for the new changes. External Buzz service fixtures are outside this PR. The separate ngit reliability proposal addresses the equivalent harness and Blossom timing issues.

AI assistance: implemented and reviewed with Codex (GPT-6).
 2026-09-12 15:53:41 +01:00
DanConwayDev 3a6de13568 Merge #edc95f2b: test(sync): simulate downtime in checkpoint integratio… 
test(sync): simulate downtime in checkpoint integration test

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqswmj2l9wqv8gnmety86n59j6j2vnmdr4rvg3nhlsnadudp4v0nuggz2q9ml

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

The checkpoint integration test can fail on a loaded builder because its 75 ms sleep leaves only 25 ms before the cold index expires. The nixpkgs 3.0.2 build observed `cold_expired == 1` where the test expected zero.

Simulate two minutes of downtime by editing the temporary checkpoint's `saved_at` field, with hot/cold retention periods of one minute and one week. Keep every existing assertion and leave production behavior unchanged. This extends the approach already applied to the equivalent unit tests in commit 14170f202f to the integration test that retained the sleep.

Validation: independent review, Rust formatting, whitespace checks and application to the released 3.0.2 source passed. Full Rust test execution remains pending in CI and the host nixpkgs build. The proposed nixpkgs package carries the matching test-only patch until a release includes the correction.

AI assistance: Codex (GPT-6) prepared the correction and PR, with an independent agent review. Publication uses dcagent.
 2026-09-12 14:50:03 +01:00
DanConwayDev 49d94d8a05 Merge #54658b4c: ci: publish release tags to crates.io from Nostr CI 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs9gevtf3cnx2t8m5h7cxyzcda7ygcf9pld6kdkcq4l90sfcr2m45qvcet5s

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Crates.io publication is a manual release step even though Nostr CI already validates tagged commits. Publish after the existing lint, formatting and test steps succeed, using a repository-scoped CARGO_REGISTRY_TOKEN secret only in the publication step.

Require a push event on a v-prefixed tag and an exact manifest-version match. Use Cargo package verification and --locked against the explicit crates-io registry. Only ngit-grasp is selected; grasp-audit remains unpublished.

This assumes the coordinator has the scoped token configured and the tagged commit contains this workflow. It does not change archive/NIP-82 publication, stable promotion, documentation sync or registry versions; no live crate was uploaded during development.

Validated workflow syntax with actionlint, matching/mismatched tag and missing-secret cases with an intercepted publish command, Cargo packaging without upload, cargo fmt --all -- --check, and git diff --cached --check.
 2026-09-12 07:57:02 +01:00
DanConwayDev 4b2d7511e1 Merge #a52fa551: ci(release): publish NIP-82 releases from tags 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs22ta92yrt4cmtklv9k5luw6t576d3z4nd0tfqud3s6swdcfrcxjgrmf5tu

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Motivation: make tagged ngit-grasp builds discoverable as complete NIP-82 releases without retaining the legacy zsp publication path.

Approach: commit the application and asset manifest, install ngit through setup-ngit v3, and publish with the established dcdev nbunksec after reproducible archive packaging.

Correctness: derive version and commit from the exact tag, enforce the expected application author and strict metadata, replicate to the pinned Primal, Ditto, and Haven Blossom servers, and publish to repository, catalog, and release relays.

Security: place the CI nbunksec in a mode-restricted temporary file and remove it on exit; no signing material is stored in the repository or process arguments.

Excluded: v3.0.1 predates this workflow and will be backfilled manually from its exact tag; automatic publication begins with the next genuine version tag.

Validation: actionlint accepted the workflow, and ngit parsed the manifest before stopping at the expected dcdev publisher guard without uploading or signing.
 2026-09-03 13:23:24 +01:00
DanConwayDev 6864148989 Merge #6661b113: fix(sync): bounded historical repository mailbox sync 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsxvcd3z0zplr59xgkceza6vsh63etm6995e788ps2hvj5v3kt7s0c3lnsw4

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Repository roots and root-only descendants can exist on an accepted owner or maintainer's NIP-65 mailboxes without appearing on a declared repository relay. Root-participant provenance alone cannot discover that first root or query a known root against an unrelated maintainer mailbox.

This proposal associates accepted public Full repositories with their owners and declared maintainers and probes their bounded, sanitized read/unmarked inboxes with repository-scoped historical mailbox work: exact repository coordinates plus every locally known root, on the existing paced, byte-bounded fetch path with independent per-relay progress. This coverage is deliberately historical-only: owning or maintaining a repository never adds an author's inbox relays to ordinary persistent live repository targets. Root authors retain the pre-existing live coverage on their own bounded inboxes, and participant-only mailboxes remain history-only with exact root provenance.

Every peer-advertised NIP-65 relay URL first passes single-URL target hygiene (static outbound-target checks, temporary .onion rejection, nostr-watch's 52-word generated-path rule, 64-hex path segments, hostname-repeated-in-path, and a 2,048-byte wire-length ceiling), is canonicalized with the shared relay-key semantics, and is deduplicated before at most four relays per author and purpose are selected in published tag order. Canonical selection keys now match the connection map, fixing trailing-root-slash mailbox relays that previously could never start their history probe.

Correctness depends on repository admission remaining authoritative and on every fetched event continuing through the normal write policy, persistence pipeline, and global NIP-09/NIP-62 tombstones. Private instances continue to withhold repository coordinates from public mailbox relays.

Deliberately excluded are permanent owner/maintainer live subscriptions, shuffled or weighted relay rotation, NIP-66 ranking, host-level identity merging, whole-list kind 10002 rejection, and any new operator setting.

Validated with nix develop -c cargo test (44 binaries, 2,876 tests, 0 failures), cargo fmt, and cargo clippy --all-targets with a clean report.
 2026-08-19 22:04:11 +01:00
DanConwayDev 681239c4b5 Merge #b6704561: feat(storage): retire migration backups family by fami… 
feat(storage): retire migration backups family by family

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqstvuz9v8ewz6hfkpwssajjl9sjsxk50y39tlq30x75f9f8qy6wx8g94xt55

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Version 3 automatically migrates complete per-owner and `/prs/` repositories into thin views backed by one local object family per repository identifier. This PR retires each legacy family backup as soon as that family is proven healthy, so migration overhead is bounded by the family in flight instead of retaining a second full copy of the Git data indefinitely.

## Retirement safety

- Every view must be correctly wired to its family and match the active migration journal snapshot.
- The family must contain every Git-readable backup object, all packs must be indexed and pass `git verify-pack`, and the normal steady-state family integrity inspection must be healthy. This last check verifies readable/hash-correct objects, refs, alternates, packs, and shallow state before retirement.
- A durable `retired` journal state is fsynced before deletion. Backup-directory removals and parent pruning are fsynced before the journal is removed, so restart recovery cannot lose track of a resurrected directory entry.
- Unindexed packs are quarantined by SHA-256 content beneath `.grasp/migration/unindexed-packs/`. Identical retries converge; different payloads cannot overwrite one another even when their original filenames match.
- Any active-migration gate error fails closed. A family that is structurally migrated but unhealthy keeps its backup, starts the service, logs an `ERROR`, and enters the ordinary asynchronous integrity-repair path.

## Shallow compatibility and repair

A server-side depth-one fallback existed only in untagged development revisions from 2026-01-05 (`623cae5`) through its 2026-01-12 fix (`f25eea8`); no tagged v1 or v2 release shipped it. It ran only when pending state or PR objects had not arrived through the normal push path and the server fetched them from another listed Git server.

V3 preserves any resulting `shallow` marker on the thin view, so shallow clones and the current tree remain as available as before migration, and keeps the legacy backup. The normal integrity worker automatically requests the full closure from accepted clone URLs. Successful repair removes the marker immediately and the next launch retires the backup; failed repair keeps both marker and backup and logs the affected identifier. No separate operator step is required for upgrade.

## Upgrade boundary

The v2-to-v3 Git-data migration is one-way. V2 does not coordinate writes through shared families and must not run against migrated thin views. Rollback means restoring the pre-upgrade Git and relay-data snapshot together, not only replacing the binary.

Installations already migrated by the earlier candidate are supported: manually removed verified backups leave completed journals that are compacted safely, while remaining backups are rechecked and retired family by family. Unverifiable stale backups are retained with a warning.

GC and unreachable-object pruning remain deliberately excluded to preserve delete-state rollback material. Optional S3 storage is also separate from this local-storage PR.

## Validation

Regressions cover corrupt loose family objects, repeated same-name unindexed packs with different content, durable interrupted retirement, active and stale journal states, manually removed backups, real depth-one migration, automatic closure recovery, marker retention on failed repair, and full client cloning after repair.

`cargo fmt --all -- --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test --workspace`, and `git diff --check` all pass on Rust 1.96.
 2026-08-19 09:23:58 +01:00
DanConwayDev d73f2a3276 Merge #4584cec7: Route Git traffic through identifier families 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsytpxwcu3rtwvqntrk64dnekh6xf9czgv8sedtd8rd95fsugc53xcwp9h3l

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Completes the local identifier-family storage model after the prerequisite storage-primitives PR was merged.

- Routes owner and `/prs/` reads, pushes, and proactive fetches through a shared `(object format, identifier)` object family.
- Lets related repositories satisfy reachable SHA wants and advertises retained family base refs, avoiding repeat uploads of objects already stored by the server.
- Migrates legacy repositories deterministically on launch while retaining rollback backups and preserving incomplete refs and readable objects.
- Adds one permanent family integrity/healing engine for packs, object connectivity, view alternates, and ref targets. It fetches exact missing OIDs from clone URLs in accepted announcements through the existing hardened outbound path, rechecks the family, and logs unresolved damage at `ERROR`.
- Runs that engine asynchronously after migration and exposes `ngit-grasp integrity-check --identifier <id> [--repair]` through a durable live-process request queue.

Migration does not get a separate recovery subsystem: it performs the structural conversion, then hands the resulting family to the ordinary steady-state checker. Unindexed legacy packs remain in the rollback backup. Garbage collection, legacy backup archaeology, and S3 storage remain out of scope.

Testing on gitnostr.com: the already-installed storage version makes structural migration a no-op, but the startup integrity pass still runs unconditionally, so this is a valid test of the permanent steady-state path. To prove remote self-healing, use a sacrificial identifier whose accepted announcement lists a second Git server containing the same reachable object; snapshot its family and views, move one verified loose object into quarantine, invoke `integrity-check --repair` or restart, and verify the repair log, restored object, `git fsck`, and a fresh clone. This does not re-test the first legacy-to-family transition; that transition should remain covered by the migration fixtures or a disposable pre-migration data copy. Do not remove the production migration marker to force a rerun.
 2026-08-18 15:26:21 +01:00
DanConwayDev ba25b3a463 Merge #b6c5d12b: feat(http): support path-mounted GRASP services 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqstd3w39dut9hu325jnwfcvqa9r6y47t064hvenss0q30grw3pup7gz6ukp2

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Allow operators to share an authority by configuring a normalized public base path. Route HTTP, WebSocket, Git, and GRASP-06 traffic beneath that prefix and carry the path through service validation, generated relay identity, self-sync, NIP-11 icons, and landing links.

Treat NIP-05 as a domain-root capability: path-mounted services neither expose the well-known document nor advertise NIP-05 in NIP-11, and generated owner profiles omit the nip05 field. Exact relay URL matching keeps sibling mounts distinct while clone URLs may descend below the configured prefix.

This assumes reverse proxies preserve the configured public prefix. Proxy-side prefix stripping remains deliberately out of scope.

Validation:
- nix develop -c cargo test
- nix develop -c cargo clippy --all-targets -- -D warnings
- cargo fmt --all -- --check
- git diff --check
- nix-instantiate --parse nix/module.nix
 2026-08-18 12:58:17 +01:00
DanConwayDev 27f9eaaf98 Merge #c195deba: Add local identifier-family storage primitives 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsvr9w7ht8z9l8rw4glgpnu2xdh8ngtjwp8gvm4xpgwql6507vn9tqzylvuk

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Stacked on nostr:nevent1qqsw3zk0fdhqeummycnmmpzls0zwjzxmkfgnqgqnsje2hhdvvsgewkspz3mhxue69uhhyetvv9ujumn8d96zuer9wc07x838.

Adds family paths and locks, thin repository views backed by Git alternates, anonymous negotiation bases, and retained refs. Local storage remains the default and deduplicates objects across every repository view sharing an identifier.
 2026-08-17 15:55:44 +01:00
DanConwayDev 927c38f059 Merge #e88acf4b: Design identifier-family Git object storage 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsw3zk0fdhqeummycnmmpzls0zwjzxmkfgnqgqnsje2hhdvvsgewksq2g90s

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Defines identifier-scoped object families shared by owner repositories and /prs/ views.

The design accepts allowReachableSHA1InWant for related repositories, keeps local storage as the default deduplicated backend, makes S3 opt-in, and deliberately excludes garbage collection so deletion rollback remains recoverable.
 2026-08-17 12:35:51 +01:00
DanConwayDev ce6e29ed21 Merge #31a50344: ci: cache Nix and Rust build inputs 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsrrfgrgj2lr8flyyquefg59jfgqze3h4h57e4f9keq32s8py6v3ts3ak2sv

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Reuse Nix store paths, Cargo sources, and Rust target artifacts across ngit-ci runs. Keys track the flake, toolchain, and Cargo dependency inputs while prefix restores preserve useful overlap after updates.

Cache restore and save remain non-fatal optimizations, and the existing formatting, lint, and test commands are unchanged.

Validated by listing the workflow with act and checking the resulting diff.
 2026-08-17 12:01:55 +01:00
DanConwayDev e672255f69 Merge #3a2bb755: test(sync): isolate NEG proxy source traffic 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsr52ah25k3ypn380yqut6rypqmhyw5pux2hgv9w9kwrvgfur3kpngdqltr5

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

The NEG concurrency proxy records counters across all client connections. Once live self-subscription became reliable, the permissive archive source discovered the proxy URL in its own announcement and ran negentropy through the same measurement point, making the test evidence include scenery traffic.

Reuse the production-outbound-policy archive source fixture already used by REQ. Its loopback event-directed targets are rejected while the syncing relay bootstrap remains operator-configured and allowed, preserving end-to-end coverage and attributing proxy counts only to the relay under test.

Correctness assumes this integration topology remains loopback-only. Deliberately excluded: production sync behavior and the REQ scenario are unchanged.

Validation: nix develop -c cargo fmt --check; cargo test --test sync sync::neg_concurrency::startup_historic_sync_stays_within_relay_neg_concurrency_limit; cargo test --test sync sync::req_concurrency::startup_historic_sync_stays_within_relay_req_concurrency_limit.
 2026-08-17 08:27:24 +01:00
DanConwayDev b50883d9f0 Merge #2f5ad1ca: fix(logging): align operational severity with actionab… 
fix(logging): align operational severity with actionability

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsz7kk3e29ceup4fzccjxq48r6vxu4f3jd54anfurevch0vp6lfmvq9fke8q

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Production evidence showed the default info stream was dominated by dependency chatter and peer-controlled detail: gitnostr emitted 1,008 per-repository discovery records in a 3,501-line window, the US relay emitted 232 in 972 lines, and the German relay emitted 2,288 discoveries plus 2,236 warnings in 11,472 lines. Rejected historical announcements dominated the warnings, while all three services were intentionally running NGIT_LOG_LEVEL=info rather than debug.

This PR scopes bare levels to ngit-grasp with dependencies held at warn, while preserving explicit EnvFilter expressions. It moves individual repository, event, filter, WebSocket, HTTP disconnect, and missing-object probe diagnostics to debug; retains aggregate sync outcomes and capability fallback at info; retains transient cooldowns at warn; and keeps internal database, policy, service, and subprocess failures at error. Unsupported NIP-77 is reported once per connection rather than once per filter.

Configuration docs, the NixOS module, the example environment, monitoring guidance, and changelog are updated together. Tests that use rejection diagnostics as an ordering barrier explicitly opt into application debug logging. A scheduler-sensitive checkpoint-expiry test encountered during the package build now simulates downtime through persisted state instead of a fixed sleep; production behavior is unchanged.

Validation:
- cargo fmt --check and git diff --check
- focused logging, HTTP, Git, self-subscriber, capability-gate, downtime, and maintainer-reprocessing tests
- all 795 library tests passed in the development suite
- all nine maintainer-reprocessing tests passed sequentially after opting their fixtures into debug
- the 2,500-event REQ concurrency stress test passed in isolation after an earlier host-contention timeout
- nix build .#ngit-grasp --no-link passed, including the pinned-toolchain 795-test package gate
 2026-08-17 07:44:28 +01:00
DanConwayDev 2a2cabd78d Merge #21fd077e: fix(private-repos): restore self-subscription in priva… 
fix(private-repos): restore self-subscription in private mode

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqszrlg80ct2jn7ednnam89yfsggeradp6wl27ncxsmwuqrd8746c5qll4q5e

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

The SelfSubscriber feeds the sync manager's repository index from the
service's own accepted events. It dialled our own public WebSocket
endpoint with an unauthenticated client, which a private instance's
NIP-42 gate correctly refused: the gate runs in the HTTP layer, ahead of
LocalRelay, and cannot distinguish our own process from any other
anonymous dialler. Announcements accepted at runtime therefore never
reached the sync index until a restart rebuilt it from the database,
stalling proactive sync and the dynamic membership derived from accepted
relay owners — worst on exactly the private-to-private mirroring GRASP-08
exists to enable.

Attach the subscriber to the embedded relay in-process instead. A custom
WebSocketTransport hands LocalRelay one end of an in-memory duplex pair
and keeps the other, so the client gets an ordinary relay session with
the same framing, subscription handling, and post-save broadcast, minus
the listener, the auth gate, and the network round trip. This replaces
the loopback dial in both modes: the public-mode feed is identical in
content and strictly more reliable, and it removes a self-directed
reconnect loop.

Chosen over attaching the relay owner key as a NIP-42 authenticator plus
adding the owner pubkey to the effective member set. That alternative
works, but widens the member set and the authenticated surface to solve a
problem that is not authentication: there is no remote party here. The
in-process route needs no key, no membership entry, and no configuration,
and nothing reaches the subscriber that the relay did not already accept
and persist.

Correctness assumptions: LocalRelay applies no NIP-42 or query policy of
its own — private-mode access control lives entirely in the HTTP layer —
so an in-process session is exactly a local session, not a bypassed
remote one. Both ends speak raw WebSocket framing over the duplex with no
HTTP upgrade, matching take_connection's Role::Server. The session
consumes one connection permit, as the loopback dial did.

The GRASP-08 regression test no longer restarts the relay over persistent
LMDB: it publishes an announcement at runtime and asserts sync
connections to both referenced relays, which is only possible if the live
feed reached the index. Verified to fail against the previous
implementation (60s deadline, no connections) and pass with this one.

req_concurrency's source relay now applies the production outbound target
policy. Its scenario lists a proxy URL in the announcement, and with a
reliable live feed the source discovers that URL — a distinct host:port
that happens to front itself — as an event-directed sync target and opens
its own REQ traffic through the proxy, contending for a budget the test
means to measure for the syncing relay alone. That behavior is
pre-existing and was already reachable after a restart; only its timing
changed. The policy keeps the source scenery without weakening the
assertion.

Deliberately excluded: neg_concurrency shares that topology but passes
unchanged, so its fixture is left alone; the duplicate NIP-11 fetch
between the pre-dial preflight probe and the post-connect hint fetch is
untouched.

Validation: cargo clippy --all-targets -D warnings; cargo test --lib (793
passed); cargo test --test private_mode --test sync --test
outbound_policy --test purgatory_sync (255 passed, including the full
sync suite under parallel load). req_concurrency's startup-burst test
passed 4/4 isolated runs after the fixture change.
 2026-08-17 04:48:47 +01:00
DanConwayDev 8b69971593 Merge #0a6a0d75: feat(private-repos): authenticate outbound sync betwee… 
feat(private-repos): authenticate outbound sync between GRASP-08 services

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsq56sdwh0cccm69uqln5ar58hf38xssqwpzs9fz87kfdtlv3x45qqr462cs

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Implements the outbound half of GRASP-08 private-service support: how an instance behaves as a sync client toward relays that demand authentication or advertise GRASP-08.

Policy: (1) every sync connection answers NIP-42 AUTH challenges with the relay owner key when available, on public and private instances alike; after successful authentication the refused subscription is retried once, and without an owner key authentication is skipped and auth-demanding subscriptions park immediately. (2) A restricted: rejection after valid authentication is terminal and parks subscription work via the existing policy-refusal machinery, so there are no retry storms. (3) A relay whose NIP-11 supported_grasps advertises GRASP-08 is recognised with a pre-dial NIP-11 fetch that re-runs the outbound target policy, keeping the SSRF gate intact: a public instance parks such a relay without ever opening the WebSocket (no AUTH exchange, nothing to launder), while a private instance treats it as a peer and attaches the GRASP-08 repository-root NIP-98 credential (60-second validity, minted fresh per git subprocess) to purgatory Git fetches from that host, both signed with the relay owner key. (4) Relays with missing or unreadable NIP-11 remain ordinary sync targets. (5) Membership tightening: the NIP-11 owner of a relay referenced by an accepted announcement is only admitted as a derived member when that relay also advertises GRASP-08; configured NGIT_PRIVATE_MEMBERS are unaffected.

NIP-42 here is identification, not confidentiality: our pubkey is already published via NIP-11 and NIP-05, and a private instance authenticating outbound discloses its identity consistently with the public-discovery stance of GRASP-08. Also adds disclosure guidance for private services: security contact via NIP-11 contact and the NIP-05 root identity, reports as NIP-17 encrypted DMs on the public mailbox relays of maintainers, collaboration access by adding the reporter to NGIT_PRIVATE_MEMBERS, and deliberately no non-member submission lane.

Validation: cargo fmt --check, cargo clippy --all-targets -D warnings, cargo test --lib (792 passed), and the private_mode, sync, outbound_policy, purgatory_sync, nip11_document and relay_identity integration suites all pass.
 2026-08-15 19:50:34 +01:00
DanConwayDev d248394bcf Merge #2568f4fc: ci: skip docs-only pushes with paths-ignore 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsz2685lna6yk59uk6u0elwaejhjw3gj32fdrvnh8kep2tu78l34eqr838a3

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Every push currently runs the full Rust CI pipeline, including pushes
that only touch documentation, markdown, or gitignored-except-README
session notes under work/. Add paths-ignore for docs/**, **/*.md, and
work/** to the push trigger so those runs are skipped, while a push
touching both code and docs still runs (paths-ignore only skips when
every changed file matches).

Per ngit-ci's filter semantics (ngit-ci docs/writing-workflows.md):
paths filters are supported for push triggers with GitHub glob syntax;
tag pushes always run regardless of path filters; and pull_request path
filters are not evaluated yet, so the pull_request trigger is left
unfiltered deliberately.

Validated by parsing the workflow YAML; no job content changes.
 2026-08-15 19:45:31 +01:00
DanConwayDev 5865ed4a47 Merge #f7cecf4c: test(sync): replace remaining fixed sleeps with observ… 
test(sync): replace remaining fixed sleeps with observable waits

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs00nk0f3vjr5l784yf6lcc7zultsy8asa82znqk6dq0mpk2m5kpfsl6hg0r

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Sweep the fixed sleeps left in the sync suite after the tag_variations /
live_sync hardening, following the same pattern: wait on the condition
the test actually cares about with a bounded deadline, so healthy runs
are no slower and only the failure bound widens to 30 seconds.

- discovery, historic_sync, metrics, maintainer_reprocessing: replace
  "wait for discovery/sync" sleeps with bounded polls on the synced
  event, a metrics counter/gauge, or the syncing relay's log (rejected
  announcements are observable by their note ID), and widen the
  short verification deadlines that followed them.
- purgatory_fetch, live_sync, historic_sync: replace fixed post-connect
  sleeps on raw clients with `connect().and_wait(timeout)`.
- metrics: poll the metrics endpoint for readiness instead of sleeping
  through relay startup.

TestClient::send_event is reworked to send through its single tracked
relay so the SDK error kind survives: transient transport failures
(disconnect, timeout, not-connected) are retried with the existing
bounded backoff plus a reconnect, while a relay OK-false rejection now
fails immediately instead of being retried, so tests cannot mask
write-policy regressions. A transient client error under load
previously killed adaptive_pagination's seeding despite the retry loop,
because rejections and transport failures were indistinguishable at the
pool level.

Deliberate time-as-behavior sleeps stay and are now commented: the
one-second rejected-hot-cache TTL expiries, created_at spacing before
+1s-dated events, and fixed observation windows for events that must
NOT appear. req_concurrency is deliberately untouched: its proper fix
needs a relay observable for descendant-sweep completion (issue
b052fa70).

Validated under CI's hostile global git config (GIT_CONFIG_GLOBAL
mirror of the workflow step): three consecutive green runs of
`cargo test --locked --test sync`, plus `cargo fmt --check`,
`cargo clippy --workspace --all-targets -- -D warnings`, and a full
`cargo test --locked` (its sync pass green a fourth time; one unrelated
load flake in live_sync_regroups_after_filter_count_refusal passed on
isolated rerun and in the repeat full run).
 2026-08-15 19:43:53 +01:00
DanConwayDev 22abda22a8 Merge #74151bb5: style: apply rustfmt to private-repos code 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs8g9gmk5erhxf2m28lh96lrcnqh2p995jwzgn9z6wn2xk7dgz68jg56dksy

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

The GRASP-08 service-auth PR branched before the pull_request CI
trigger existed, so no PR run checked it, and the membership-gating PR
was merged with a failing run. Both landed unformatted code, so every
master push now fails at cargo fmt --all -- --check before clippy or
the test suites execute.

Mechanical cargo fmt --all output only; no behavior change. Any clippy
or test failures the fmt gate was hiding are addressed separately.
 2026-08-15 13:54:40 +01:00
DanConwayDev 4be9408aed Merge #bd5dda7a: feat(identity): publish relay owner events 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqst6hw60gzs9qcxzr95h9wd76n3tctq9khdgpw2gxd75glsstmfvjgkn9xwt

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

This PR makes the relay-owner key discoverable and usable as the ngit-ci service identity. On startup the relay signs a minimal kind-0 NIP-05 bot profile (scheme-less public URL as name, `_@domain`, `bot: true`) and a kind-10002 list naming this relay as its sole read/write relay, then publishes them to the configured user-index relays in the background with a capped retry backoff.

Publication is deliberately conservative. No identity event — locally stored or freshly generated — is sent before the local database and at least one user-index relay have been successfully checked for that kind, and every send is preceded by a per-relay re-check. An identity found on an index relay is adopted locally (replaceable semantics keep the newest copy) and never overwritten, so the relay only fills gaps on index relays that confirm they hold none: a database wipe followed by a boot during an index outage can no longer displace a customized profile surviving on the indexes, and pushing profile updates onto an index that already has one is left to the operator's own client. A kind with no local copy is not even seeded until a reachable index confirms it holds no identity of that kind. Remote outages and local rejections never block startup. In private mode (GRASP-08) the identity events are seeded and served locally but never published, so a private relay does not advertise its existence.

The write policy trusts owner-signed events only for kinds without a dedicated admission policy, such as ngit-ci coordinator advertisements that carry no repository root tag. Owner-signed NIP-34 announcements, state events, and PRs run the normal validation, ref alignment, and purgatory handling like any other author; the relay's own kind 0/10002 identity is always accepted; and the NIP-09/NIP-62 deletion gate runs before any owner acceptance, so a replayed retracted owner event cannot undo its tombstone. Because the event blacklist cannot block the owner key, rotating it is the only remediation if it is compromised.

Integration coverage includes local seeding and indexing, restart behavior with an operator-customized profile, adoption from a user index after a database wipe, deferral while no index is reachable, non-displacement on a recovering index that already holds an identity, private-mode suppression, and owner-signed state events taking the normal admission path. Validated with rustfmt, strict workspace Clippy, and the full workspace test suite.
 2026-08-15 12:30:24 +01:00
DanConwayDev e173c58e67 Merge #9af177ea: feat(private-repos): gate announcement admission on me… 
feat(private-repos): gate announcement admission on membership

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsf4uthaf2cr5hes246tckccufhntfj3uc5su95t5l7z5p9g6s3jmq059w44

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

In GRASP-08 private mode, accepted kind-30617 announcements expand hosting and - via their referenced relays NIP-11 owners - the effective member set itself. Announcement admission never checked the author, so a member could submit a third-party-signed announcement (or sync could import one from an operator-configured source) and thereby mint membership for pubkeys no member ever chose.

This PR closes that amplification loop: in private mode a repository announcement is only admitted when its author (event pubkey) is a current effective member (configured members plus admitted relay owners, via the shared PrivateAccess set) evaluated at admission time. The gate sits at the single choke point all arrival paths funnel through - direct publish, sync import, purgatory entry - and rejections use the existing announcement rejection machinery. Public mode is unchanged, no new configuration is added, state events (30618) keep GRASP-01 maintainer rules, and removal is non-retroactive: repositories admitted while their author was a member stay hosted until the operator curates them. Also documents the rationale in the GRASP-08 design doc and architecture doc.

Deliberately excluded: outbound authentication when syncing from other private services, which remains a separate follow-up.

Validation: cargo clippy --all-targets -D warnings clean; lib tests (783), private_mode (52, incl. a new member/non-member admission integration test), nip34_announcements (60), repository_creation (47), and purgatory (55) suites all pass.
 2026-08-15 11:15:32 +01:00
DanConwayDev 2a213baee9 Merge #abd26762: test: survive hostile git config and non-reaping PID 1… 
test: survive hostile git config and non-reaping PID 1 in CI

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs2h5n8vgaysdav2nwh35f3rs3vvm6kjc0slv4d87jz56cepzusavc4xc29f

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Fixes the 7 lib-test failures seen in CI on master and on the pull_request-trigger PR.

Five tests built git fixtures with bare git commands, so the hostile pre-commit hook CI installs via core.hooksPath broke fixture commits silently. Fixtures now use the hermetic grasp_audit::git_command() and assert success on every step.

Two process-group tests probed descendant death with kill(pid, 0), which still succeeds for an unreaped zombie; act containers have no reaping PID 1, so the orphaned descendant stayed a zombie and the probes timed out. The probe now also treats an unreaped zombie (state Z in /proc/<pid>/stat) as terminated.

Both failure modes were reproduced and verified fixed locally under the workflow hostile git config plus a non-reaping subreaper wrapper. fmt, clippy -D warnings, the full workspace suite and grasp-audit all pass.

Stacked on nostr:nevent1qqsg7maf57s5v7u5cnvrvg8n3dl8dluremlmtqxwn9ex3rvmtrrjaqcpz3mhxue69uhhyetvv9ujumn8d96zuer9wc45dzcp so the pull_request trigger is present and CI runs against this PR.
 2026-08-15 10:01:16 +01:00
DanConwayDev 602f38878c Merge #3fa7d8ad: feat(private-repos): add GRASP-08 service authenticati… 
feat(private-repos): add GRASP-08 service authentication

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsrlf7c45djx8qdst0jg764n8n6wfrf05gffncjfnym3ceguy6k3pgezg30q

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Implements the mergeable single-service GRASP-08 authentication boundary on the existing canonical repository paths. GRASP-08 does not require another path component: Git remains at /<npub>/<identifier>.git, while service-fleet routing is deliberately left for a future proposal.

Private mode fails closed. WebSocket access authenticates with NIP-42 before traffic reaches the relay, and Smart HTTP uses the reusable repository-scoped GRASP-08 form of NIP-98. Missing, malformed, expired, or non-member credentials receive the same empty 401 response. The canonical public origin is operator-controlled, paths remain contained under the Git root, and ordinary GRASP-01 push authorization still applies after service authentication.

Membership combines explicitly configured pubkeys with the NIP-11 owners of relays referenced by accepted repository announcements. It reuses the existing once-per-connection NIP-11 fetch and five-second maintenance pass, so the feature adds no polling, subscriptions, relay connections, or background task. Owners learned only through unaccepted purgatory state cannot grant access.

The three atomic commits separate the authentication boundary, dynamic accepted-relay membership, and subprocess integration coverage. Configuration is synchronized across source, reference docs, NixOS module, and example environment; architecture documentation records the implemented single-service scope.

Validation:
- cargo test --locked --lib: 759 passed;
- private service subprocess scenario: 45 passed, including missing/non-member rejection and member admission;
- Smart HTTP streaming: 3 passed;
- NIP-01/WebSocket compliance: 52 passed;
- cargo clippy --all-targets -- -D warnings: passed;
- nix build .#ngit-grasp: passed.

Resource assessment: inactive private mode has no runtime path change. Active private mode adds one shared membership set, one generation watch, per-session NIP-42 bridging, and membership reconciliation on the existing maintenance cadence. Its relay-owner discovery reuses NIP-11 state already fetched for sync limits.

Recommendation: ready to merge as the GRASP-08 single-service foundation. Multi-service fleet orchestration and encrypted kind-10318 client discovery remain explicit follow-up scope.
 2026-08-15 07:53:03 +01:00
DanConwayDev ae22bcc1f2 Merge #8f6fa9a7: ci: run Rust CI on pull_request events 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsg7maf57s5v7u5cnvrvg8n3dl8dluremlmtqxwn9ex3rvmtrrjaqcm30vm0

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

The workflow only fired on push, so Nostr PRs (kinds 1618/1619) received
no CI runs until their commits landed on a branch. ngit-ci's
pull_request trigger fires on PR and PR-update events, giving
contributors feedback before merge.

The job itself is unchanged: the same checks now also run against the
PR head commit. No branch filters are added, matching the existing
unfiltered push trigger.
 2026-08-15 07:44:13 +01:00
DanConwayDev b4f554ad98 Merge #edf4290b: fix: disable ANSI log colors when stdout is not a term… 
fix: disable ANSI log colors when stdout is not a terminal

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqswmapfpw7etaxy5zhl5ltwlkhrycuhehjzdqztmupze0nxnype4vcuqwe07

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Motivation: four sync integration tests failed deterministically in
some environments: proactive_sync_plus::
missing_relay_list_uses_bounded_fallback_coverage,
maintainer_reprocessing::
unresolved_repositories_share_one_dependency_poll_per_relay,
metrics::test_live_sync_event_count, and reconnect_backoff::
flapping_relay_handshakes_do_not_reset_exponential_backoff. Each waits
for a plain `field=value` substring (e.g. `consecutive_failures=1`) in
the relay subprocess log written by the TestRelay fixture. The tracing
fmt layer colorizes by default, so the redirected log contained
`consecutive_failures^[[0m^[[2m=^[[0m1`, which never matches.
tracing-subscriber honours NO_COLOR, so the same tests passed or failed
depending on the invoking environment: runs with NO_COLOR set produced
clean logs, interactive-launched runs produced ANSI logs and
deterministic failures.

Approach: initialize the fmt layer with
`.with_ansi(std::io::stdout().is_terminal())`. Colors are emitted only
when a human is watching a terminal; redirected output (test fixtures,
journald, pipelines) stays plain. This is a logging hygiene fix in the
binary, not a test accommodation: the defect was writing terminal
control sequences to a non-terminal stream.

Correctness assumptions: the relay binary is the only process whose
logs the fixtures scrape, and tests always redirect its stdout to a
file, so is_terminal() is deterministically false there and the log
format no longer varies with the parent environment.

Excluded scope: several sync tests remain flaky under full-suite
parallel load but pass repeatedly in isolation
(req_concurrency::startup_historic_sync_stays_within_relay_req_concurrency_limit,
live_sync::live_sync_regroups_after_filter_count_refusal,
tag_variations::test_layer3_sync_with_lowercase_e_tag). They assert on
metrics or proxy behaviour, not log text; their bounded deadlines are
exceeded when the whole suite competes for CPU. Left undiagnosed rather
than papered over with wider timeouts.

Validation: the four log-scraping tests pass individually and in full
runs after the fix; cargo fmt makes no changes; cargo clippy
--workspace --all-targets -- -D warnings is clean; full cargo test is
green except the pre-existing load flakes above, each of which passes
3/3 when run in isolation.
 2026-08-15 07:24:46 +01:00
DanConwayDev 18eddeedb5 Merge #4e732958: feat(sync): discover participant mailboxes 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsyuueftzc68d5a3x5d0pecs66y0nacv4pc87kkcfw499adyl5t3ycxqgtta

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

## What this adds

Repository conversations can continue on relays used only by a non-root participant. For example, a reaction to an issue may be stored on one relay, while a reply to that reaction exists only on the reaction author's NIP-65 write relay. Root-author relay discovery and recursive reference queries cannot find that reply unless the participant's mailbox is also searched.

This PR adds bounded, historic participant-mailbox coverage:

1. Preserve exact repository-root provenance while deriving the accepted recursive descendant frontier.
2. Treat authors of accepted replies, comments, reactions, zaps and other descendants as thread participants.
3. Accept kind `10002` relay lists only for repository owners, maintainers, root authors and those accepted participants.
4. Derive each participant's NIP-65 read, write and unmarked mailbox relays.
5. Query those mailboxes only for the repository roots and accepted descendant IDs associated with that participant.
6. Pass every returned event through the ordinary write policy, deletion/replacement rules and persistence pipeline.

This discovers replies to reactions and other indirect descendants without fetching a participant's unrelated notes or trusting their relay list to bypass repository policy.

## Bounds and scheduling

- The existing `NGIT_SYNC_RECURSIVE_DESCENDANT_LIMIT` bounds which indirect descendant IDs remain query roots. Each direct event that tags a repository root has its own bounded branch; the default is 500.
- Once a branch reaches that bound, later indirect descendants cannot add more participant authors or query roots through that branch.
- Exact root provenance prevents roots from one repository leaking into another participant query.
- Participant mailbox coverage is historic only. It does not add permanent non-root participant subscriptions or expand the existing live-sync tier.
- Filters use the existing byte-bounded grouping and paginated `RelayConnection::fetch_events` path, including request pacing, background priority, subscription-ledger capacity and EOSE/CLOSED handling.
- At most one mailbox worker runs per relay, and at most one new due relay is started per maintenance pass.
- Relays progress independently: there is no global mailbox lane, cross-relay success condition or shared completion counter.

## Restart and failure behavior

- A probe starts only after both the WebSocket and the sync actor's connection lifecycle are ready.
- Ready due relays are preferred, so old unreachable mailbox sources cannot starve connected work.
- Completed mailbox workers are handled before new connection results, so a large startup connection queue cannot delay cursor progress or resource release.
- Successful filters advance an in-memory, stable-sorted cursor immediately. Completing the last filter schedules the next historic rotation after 24 hours.
- Failed filters release their relay worker and retry after five minutes without blocking other relays.
- On restart, accepted roots, participants and kind `10002` ownership are reconstructed from LMDB. Filter cursors deliberately restart from the first current group; they are best-effort coverage, not durable exactly-once state.

The implementation reuses the existing relay transport. It does not add a mailbox-specific protocol state machine, pending-batch purpose, CLOSE API, NIP-42 retry system, watchdog or connection-queue coordinator.

## Coverage and production evidence

The audit motivating this change found 72 events visible through gitworkshop but absent from the pre-change production relay across the `gitworkshop` and `ngit` repositories. The persistent canary served 29 exact misses: six kind-1 notes, three reposts, ten reactions, six comments, two issues, one zap receipt and one repository-follow event.

The exact candidate `de5fa6625aa5693afcde7335a572456680aef9df` activated on the isolated archive at 2026-08-14 19:23:49 UTC. Restart reconstruction found 4,291 accepted roots, 403 participant authors and 351 mailbox relays.

During the recorded soak:

- 63 mailbox filters started and all 63 reached terminal handling: 59 successes and four bounded failures.
- Failures received the intended five-minute retry and did not block successful progress on other relays.
- `relay.ngit.dev`, `nostr.land`, `nostr.mom`, `haven.danconwaydev.com/inbox` and other relays advanced independently.
- `nostr.azzamo.net` completed all six filter groups; the final terminal reported `completed_cycle=true` and scheduled the next probe in 86,400 seconds.
- The archive remained active with zero restarts and no process panic or fatal error.

The temporary archive test override of `NGIT_SYNC_RECURSIVE_DESCENDANT_LIMIT=2` has been removed. Both the archive and public gitnostr.com have no override and therefore use the default of 500. Public gitnostr.com was not restarted or changed by the archive deployment.

## Validation and review state

- Exact tested head: `de5fa6625aa5693afcde7335a572456680aef9df`.
- One commit; 1,157 additions and 140 deletions across 11 files.
- 767 library tests passed in the exact Nix release build locally and on the production host.
- All three proactive Sync+ integration scenarios pass together, including a child found only through a participant reaction author's write mailbox.
- Strict all-target Clippy, formatting, diff checks, Nix flake evaluation and a conflict-free merge-tree check against current `master` passed.
- One remote build attempt hit the pre-existing `test_entries_expired_during_downtime` timing flake; its source is unchanged by this PR, and the unchanged candidate passed all 767 tests on retry before activation.

Recommended for merge. The archive demonstrates successful and failed terminal paths, per-relay independence, restart reconstruction, cursor advancement and a complete historic-to-24-hour-refresh rotation under real startup load.
 2026-08-14 20:56:54 +01:00
DanConwayDev 97bf3e89c2 Merge #cea1ad82: feat(http): serve NIP-05 root identity 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsvagddsgtyqp03tjctp3p405ajqq7zxphcrej3mnu4j4kc5rkyflqnxtu05

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Relay operators already have a stable Nostr identity for NIP-11 and NIP-42, but clients cannot resolve the domain itself to that key. This proposal serves the NIP-05 root identifier so `_@domain` identifies a root-mounted relay without another key source.

It adds an exact root-level `/.well-known/nostr.json` route for GET and HEAD, serializes the relay-owner public key as lowercase hex under `names._`, retains permissive CORS, and gives this route precedence over NIP-11 content negotiation. NIP-11 derives support from the relay request path: only `/` advertises NIP-05, while nested relay paths omit it.

The changelog, architecture, README, and relay-owner configuration descriptions are synchronized. The path check assumes the HTTP request path is the relay public mount path; a future explicit base-path setting should remain the source for this decision if proxy routing hides that path. Named identities and recommended relay-hint mappings remain outside this change.

Validated with 760 library tests, the root/non-root NIP-05 server integration test, strict all-target Clippy, cargo fmt, git diff checks, and Nix module parsing.
 2026-08-14 20:28:20 +01:00
DanConwayDev 014875cb9f Merge #26412457: feat(sync): recurse through related event frontiers 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqszvsfy2lca5ujjnpn3h0kft403m5nnxqj6nsd68m394vlj9c8xjlc4hc37z

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

## Scope

Implements the recursive related-event frontier recommended by nostr:nevent1qqsfdj4fegeq7zjpje7cht6cj9xcsw5a9h2lcx2ju0f4uj80gsvvyhcpz3mhxue69uhhyetvv9ujumn8d96zuer9wc70zgv8.

PR1 and PR2 are merged. This is layer three; participant mailbox discovery remains the final stacked proposal in nostr:nevent1qqsyuueftzc68d5a3x5d0pecs66y0nacv4pc87kkcfw499adyl5t3ycpz3mhxue69uhhyetvv9ujumn8d96zuer9wca2tmqx.

Root-only and one-generation filters miss events which reference an immediate parent, including replies to reactions. This layer traverses accepted local `e`, `E`, and event-valued `q` references plus `a`, `A`, and address-valued `q` references. The resulting frontier feeds existing live coverage when it fits and the existing five-second REQ+EOSE historic rotation when it does not.

## Simplified bounded model

Let `R` be an accepted repository root and `D` an event which directly tags `R`.

- `D` owns one independent recursive subtree allowance, configured by `NGIT_SYNC_RECURSIVE_DESCENDANT_LIMIT` / `--sync-recursive-descendant-limit` and defaulting to 500.
- `D` itself is direct/core coverage and does not consume the allowance. Every other event which directly tags `R` starts its own branch.
- Locally stored descendants are walked breadth-first in stable creation-time/event-ID order through event and address references.
- While fewer than 500 recursive descendants are known for a branch, its members can be child-query seeds.
- As soon as the branch reaches 500, none of its members remain in recursive child-query filters. Existing auxiliary live coverage is closed and rebuilt without that branch; historic filters omit it too.
- A recursively discovered event never creates another independent allowance. The existing eight-generation traversal remains an additional safety bound.
- Direct/core sync and unrelated direct-root branches continue normally after one branch fills.

This is intentionally a soft query-frontier limit, not an exact storage cap. Events returned by a request already in flight are still processed normally. A single response could therefore store more than the configured number of siblings; on the next reconciliation the branch is full and none of those events can extend recursive fan-out. The design bounds continued graph expansion without adding receive-path rejection or reservation machinery.

## Relay independence and restart

There is no new cross-relay coordinator, counter, reservation, teardown dependency, or completion dependency. Each source retains its own connection, subscriptions, historic cursor, EOSE handling, pacing, health, and backoff.

The only shared input is the relay's existing local database. Every connection derives its desired frontier from accepted LMDB events during normal reconciliation. An event arriving from one source can therefore make the locally reconstructed branch full for later reconciliation on every source, but no source waits for or coordinates with another.

LMDB is also the restart checkpoint. Startup derives the same bounded frontier from persisted events; a branch already full issues no further descendant queries and receives no fresh allowance. No additional durable cursor or budget state is stored.

## Configuration

The positive integer option is present on all four supported surfaces:

- CLI: `--sync-recursive-descendant-limit`
- environment: `NGIT_SYNC_RECURSIVE_DESCENDANT_LIMIT`
- NixOS: `syncRecursiveDescendantLimit`
- example environment and configuration reference

Default: 500. Small values such as 2 or 3 are supported for integration tests and fresh-archive canaries.

## Validation

- Exact head: `9360d419b3bffc5703a90aae330c315ed22459ee`
- Stable PR root: `26412457f1da725298671bbec95d5f1dd2733025a9c1ba3ee25ab3f22e0e697f`
- Final diff: 667 additions, 110 deletions (777 changed lines), down from the discarded 1.3k-addition exact-admission design.
- `nix develop -c cargo test --lib -q`: 758 passed.
- Six focused recursive-frontier unit tests: passed.
- Limit-2 historic/restart integration: passed in 42 seconds.
- Serialized sync suite: all descendant cases passed; overall 95 passed, 1 deliberately ignored, and 2 unrelated Layer-2 timing tests failed once then passed isolated reruns.
- `cargo clippy --workspace --all-targets --all-features -- -D warnings`: passed.
- `cargo fmt --check`, `git diff --check`, and `nix flake check --no-build`: passed.

The real-relay integration constructs `R <- D <- A <- B <- C` with a limit of 2. It observes `A` and `B`, proves `C` is not queried, restarts against the same LMDB, publishes another child of `B`, proves both excluded children remain absent, and proves an unmetered direct/core marker still syncs.

## Why the model changed

An aggregate 4,096 limit truncated unrelated direct sets. A later 32,768 recursive aggregate still allowed one popular direct child to consume the entire repository budget. The discarded exact per-branch design prevented overshoot but added roughly 680 runtime lines of receive-time admission, reservations, and cross-relay lifecycle handling.

This head keeps the useful per-direct-child boundary and deliberately sacrifices an exact stored-event guarantee for a much smaller, reconstructible query-frontier rule. The final PR is a single atomic commit and adds no inter-relay behavior.

## Coverage evidence

The audit found 17 genuine missing events at recursive depths 3–5 in the named `gitworkshop` and `ngit` repositories, demonstrating that the former one-generation rule was incomplete. A superseded unrestricted recursive canary recovered at least 230 saved events, showing material recursive yield; it is not presented as validation of the final tiny-limit boundary.

## Canary deployment

The combined PR3 + stacked PR4 candidate was deployed only to the isolated archive canary at 2026-08-14 16:09:48 UTC. PR3 is exact commit `9360d419b3bffc5703a90aae330c315ed22459ee`; the deployed binary is `/nix/store/wlhqpr0x18njpmj3khwigrzhwj9z48lq-ngit-grasp-2.1.2/bin/ngit-grasp`.

The canary uses persisted LMDB at `/persistent/grasp/gitnostr-com-archive-soft-cap-2`, `NGIT_SYNC_RECURSIVE_DESCENDANT_LIMIT=2`, and the normal real-relay archive inventory. At the 2026-08-14 16:13:08 UTC snapshot it was active with PID 1553732 and zero restarts. Since activation, logs contained five started descendant historic queries and six terminal descendant batch transitions, demonstrating startup reconstruction and live REQ+EOSE descendant operation. No application panic, OOM, failed assertion, or process crash was present; the only textual `fatal` match was an unrelated external Git remote TLS hostname error.

The public `gitnostr.com` service remained on its prior binary and PID with zero restarts. This is deliberately described as an early three-minute operational snapshot, not a completed long soak or proof that every naturally occurring branch saturated the limit.

## Review state

Merge-ready. The simplified implementation, deterministic limit-2 restart integration, full validation above, and early isolated real-relay canary support a merge recommendation. Extended canary saturation/restart evidence and production deployment at the safe default of 500 remain operational follow-up checks and are not claimed as completed here.
 2026-08-14 17:22:44 +01:00
DanConwayDev ffe93bd089 Merge #a26eee65: fix(sync): retry related policy orphans 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs2ymhwv5e62ckys4n94ggqu07tl8f83t0hlk3wun2d9yvzx9w299szfklkk

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Stacked on nostr:nevent1qqsqzt6cxx9aeaj7wq5tr3mdne5e32ztpcmgvhpksz2jdyk0m20fksspz3mhxue69uhhyetvv9ujumn8d96zuer9wctgxcg4 and implements the durable dependency-retry layer from nostr:nevent1qqsfdj4fegeq7zjpje7cht6cj9xcsw5a9h2lcx2ju0f4uj80gsvvyhcpz3mhxue69uhhyetvv9ujumn8d96zuer9wc70zgv8.\n\nRepository-related events can legitimately arrive before the accepted event or repository address that makes them admissible. This change retains those policy orphans in the existing crash-safe rejected-event checkpoint and retries them in both directions whenever a dependency is saved.\n\nThe retained queue is deliberately bounded: 1,024 events, 8 MiB of serialized event payloads, 128 KiB per event, seven-day expiry, and 512 attempts per triggered closure. Oldest-first eviction is deterministic. Only terminal processing outcomes remove an entry; still-dependent and persistence outcomes remain pending.\n\nLocal evidence:\n- cargo fmt --check: pass\n- cargo test --lib: 754 passed\n- cargo clippy --all-targets -- -D warnings: pass\n- restore, forward/backward dependency matching, count-bound eviction, and end-to-end child-before-parent recovery are covered\n\nArchive canary evidence for exact commit 08d38fde:\n- deployed as 2.1.2-08d38fde at 2026-08-13 23:42:37 UTC\n- active/running with zero restarts; production remained on the parent PR\n- 26,636 deliveries observed across relay.ngit.dev and gitnostr.com after five minutes: 26,587 duplicates, 32 purgatory, 16 saved, one tombstone, zero rejected outcomes, and zero persistence errors\n- the retained-state sampler exported related_dependency_events=0 and sampled pipeline queue depth remained zero\n- no candidate-local panic, fatal, or persistence failure; channel-closed lines at 23:42:36 belong to shutdown of the previous process\n\nProduction evidence:\n- promoted unchanged at 2026-08-13 23:51:56 UTC as 2.1.2-08d38fde\n- active/running with zero restarts; archive start time remained unchanged\n- first 6,450 live deliveries were all duplicates, with zero rejected/persistence outcomes and queue depth zero\n- no candidate-local panic, fatal, or persistence failure in the post-start journal\n\nReady for merge as the second layer of the stack. Recursive relay fetching and participant mailbox discovery remain deliberately separate follow-up layers.
 2026-08-14 06:57:58 +01:00
DanConwayDev d1b46a7cee Merge #012f5831: fix(sync): account for hydration outcomes 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsqzt6cxx9aeaj7wq5tr3mdne5e32ztpcmgvhpksz2jdyk0m20fkss777nwc

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

## Scope

Adds explicit Proactive Sync Plus hydration outcome accounting and keeps dependency-sensitive or transient exact-ID failures pending instead of silently counting delivery as persistence.

Tracks nostr:nevent1qqsfdj4fegeq7zjpje7cht6cj9xcsw5a9h2lcx2ju0f4uj80gsvvyhcpz3mhxue69uhhyetvv9ujumn8d96zuer9wc70zgv8

## Local validation

- cargo test --lib: 750 passed
- censored-relay incomplete exact-ID recovery scenario: passed
- sync metrics integration suite: 11 passed
- cargo clippy --all-targets -- -D warnings: passed
- exact Nix candidate package: full release-profile test suite passed on rerun; one untouched 100/200 ms TTL test was timing-flaky on the first run

## Archive canary

Deployed 2026-08-13 23:06:19 UTC as 2.1.2-39e1f692 from /nix/store/api7j4h6x7724cjm3qz78x100m52r1zb-ngit-grasp-2.1.2. The archive stayed active with zero restarts. A clean pipeline window recorded 2,350 deliveries, 2,322 duplicates, 24 purgatory, 4 saved, zero rejected, zero persistence errors, and queue depth zero.

The longer burn-in demonstrated the diagnostic value directly: relay.ngit.dev requested 900 recovery IDs and delivered none, while relay.poster.place requested 90, delivered 90, and saved all 90. The only error-text was expected third-party DNS/TLS/WebSocket noise.

## Production evidence

Promoted 2026-08-13 23:22:29 UTC as public and loopback NIP-11 version 2.1.2-39e1f692 from /nix/store/iq80zz3zi5h8l6q21dcfnj9b6sqvkmxa-ngit-grasp-2.1.2. The unit is active with zero restarts. Its first pipeline window recorded 1,039 deliveries, 1,037 duplicates, one purgatory event, one explicit invalid rejection, zero persistence errors, and queue depth zero. No candidate-local panic, fatal, or persistence failure appeared after startup.

## Review state

Ready for review and merge. This is the base of the repository-coverage stack; later proposals will depend on it and cite this proposal.
 2026-08-14 06:54:34 +01:00
DanConwayDev 79f1d3b993 Merge #be6422ae: security: bound peer-controlled transient state 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqstuepz4mknkrlxznjptkwkeu4c2up5ya23tun8ke7rlr7qdptzdsc2ae0fd

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Issue nostr:nevent1qqsv09nf4dy084sk8vyl5yyg5v6e3ujfrxmuc93zmrc68nqeayup62cpz3mhxue69uhhyetvv9ujumn8d96zuer9wca4ndyx follows the auth-required OOM incident by auditing every transient collection whose lifetime can be influenced by a peer.

This stack fixes four concrete retention surfaces:
- EOSE/CLOSED actor notifications now have fixed backpressure while the independent terminal lane still releases permits;
- auth retry markers accept only subscription IDs owned by the current session ledger;
- deferred consolidation processes its deduplicated actor-owned state directly instead of accumulating self-wakeups;
- connection and NIP-65 worker result channels encode their existing producer bounds.

It also exports fixed-cardinality aggregate gauges for important long-lived sync state and adds the architecture inventory recording producer, cleanup owner, cardinality class, terminal behavior, and operational signal for each subsystem. Normal retained event/database growth remains out of scope.

Validation:
- `cargo test --locked --lib`: 749 passed;
- serialized `cargo test --locked --test sync -- --test-threads=1`: 96 passed, one deliberately ignored;
- formatting and warnings-as-errors clippy passed throughout the atomic commits;
- gitnostr.com is running exact revised tip `70dec5ac295dbedd340a9be5a8db873f4b4040fb`;
- production startup exercised 173 queued canonical relay attempts while execution remained under the existing eight-worker semaphore; the corrected gauge distinguishes this externally bounded queue from simultaneous dials;
- retained state remained small under startup traffic: four pending batches/subscriptions, zero auth retries and deferred consolidations, ten temporary dependency relays, 86 purgatory dependency attempts, and eight descendant rotations;
- zero service restarts; memory settled around 1.21 GiB with a 1.22 GiB peak;
- no lifecycle-channel saturation, forged-auth growth, panic, or proposal-related application error was observed. SDK error logs were malformed third-party events, unsupported NEG responses, an ordinary local reset, and an expected private-relay auth rejection.

Recommendation: ready to merge.
 2026-08-13 17:13:27 +01:00
DanConwayDev 3f621f3cbc Merge #3d857827: fix(http): account trusted proxy client addresses 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsrmptcyasq39jujn9qhhjekftqyy3j9hkedjzkq29sawm5tkqmw4qtgy9zx

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Reverse-proxied production deployments currently attribute every WebSocket connection to the proxy peer, collapsing per-IP limits, abuse metrics, and logs onto localhost. Blindly trusting forwarding headers would let direct clients spoof the same controls.

Add an opt-in CIDR trust boundary and resolve X-Forwarded-For, Forwarded, or X-Real-IP only when the TCP peer is trusted. Walk proxy chains from the peer inward, stop at the first untrusted hop, and fall back to the peer on malformed input. Feed the resolved address consistently into rust-nostr connection policy, connection metrics, and lifecycle logs.

Expose the setting across CLI/environment, NixOS, examples, and reference documentation. Multi-hop correctness assumes every trusted proxy appends or overwrites the forwarding chain and the backend is unreachable from untrusted networks. HTTP Git request accounting and PROXY protocol support remain out of scope.

Validation:
- nine address-resolution cases and three configuration cases passed;
- `cargo fmt --check` passed;
- `cargo clippy --locked --lib -- -D warnings` passed;
- deployed first as exact commit `91b43bc157e459101f8011dba7f99eebb5a5e1b3`, then retained beneath the peer-state audit tip;
- production logs retain `peer_ip=127.0.0.1` while reporting distinct forwarded IPv4 and IPv6 client addresses, demonstrating that connection accounting reaches the actual client;
- more than 35 minutes of stacked production operation recorded zero service restarts and stable memory.

Recommendation: ready to merge after the bounded-metrics proposal.
 2026-08-13 17:03:10 +01:00
DanConwayDev 9509d77bb0 Merge #b6f9e93b: fix(metrics): bound naughty-list cardinality 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqstd70f8wl7lrgsgp9pph060kpsjjudxzl6fzpjcd4lyhrl46ep9jc8gjfd4

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Production exposed one Prometheus series for every quarantined target and raw failure reason. Both labels are remotely influenced, and twelve-hour retention lets arbitrary peers create unbounded time-series churn while also copying their text into the metrics surface.

Remove the per-target metric and retain the existing three-category aggregate unchanged. Regression coverage repeatedly rotates arbitrary relay URLs and reasons, proving the exported series remain fixed and peer text never reaches Prometheus output; the changelog calls out the compatibility impact for dashboard operators.

Validation:
- focused cardinality regression passed;
- `cargo fmt --check` passed;
- `cargo clippy --locked --lib -- -D warnings` passed;
- deployed first as exact commit `a9a819801f23eaac16a1a748855a74a6d0053371`, then retained beneath two successive stacked production tips;
- production exports only the fixed-category aggregate: the removed `ngit_sync_naughty_relay_info` series remains absent;
- more than 40 minutes of stacked production operation recorded zero service restarts and no proposal-related warning.

Recommendation: ready to merge before its dependent proposals.
 2026-08-13 16:59:56 +01:00
DanConwayDev 47e9c9a818 Merge #023b8804: feat(sync): cover roots with missing relay lists 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsqywugq3lzs4t8ddnzd80tw6hznnfdwzqj3femj9aq2yw4kg9pdmcax44v6

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

## Problem

Accepted root authors without a discoverable NIP-65 kind `10002` leave their conversations outside Sync+ even after a user-index relay has answered successfully.

## Approach

After a successful empty index result, map only that author’s accepted roots onto a bounded operator-configured fallback relay set. The overlay uses the existing GRASP-02 historic, live, rotating, subscription-budget, and recovery machinery. A later accepted relay list removes the author from desired fallback coverage while shared subscriptions drain naturally. Failed index queries do not trigger fallback, and response-author expansion remains excluded.

## Validation before deployment

- End-to-end scenario passes: empty index lookup activates one fallback root and recovers its reply; a later kind `10002` moves coverage to its declared inbox and removes fallback desire.
- Discovery unit tests: 11/11 passed.
- Configuration fallback test passed.
- NixOS module parse passed.
- Workspace clippy with warnings denied, rustfmt, and diff checks passed.
- The broad test run passed all library suites and 95/96 Sync integration scenarios; the lone restart-burst proxy scenario passed standalone in 76.82 seconds and retains its documented load-sensitive baseline signature.

## Archive production validation

Exact PR commit `e84be424462278516faa7c4dd95d403b58c0a9dd` ran on the disposable archive burn-in from 2026-08-12 17:50:24 UTC.

- The service remained active with zero restarts; memory settled at 1.30 GiB peak.
- Successful index batches exercised the intended missing-list case: fallback coverage grew from 30 to 151 accepted root authors, then stabilized while ordinary NIP-65 discovery continued.
- The overlay added roughly 900–950 accepted roots to each of the four configured fallback relays. Existing filter consolidation used one additional live group where capacity remained and reused partial groups for later increments.
- None of the four fallback relays emitted a rate-limit refusal. Existing unrelated low-capacity relays continued through the already-bounded cooldown/retry path.
- No panic, OOM, killed process, or service restart occurred.

## Recommendation

Ready to merge. The production burn-in demonstrates the previously uncovered population exists at meaningful scale, is covered through the existing bounded scheduler, and does not introduce a new failure or resource trend.
 2026-08-12 22:47:29 +01:00
DanConwayDev 6475e8e25f Merge #2cb6b021: feat(sync): make Sync+ operator-controllable 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqszed4syy49sax7fdld4y25l4svrltxkd7gh5h9dh9cwkns058f3rcezp77e

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

## Outcome

Add a default-on `NGIT_SYNC_PLUS_ENABLED` opt-out across CLI/environment, NixOS, examples, and reference documentation. The switch gates only GRASP-03 mailbox discovery; ordinary GRASP-02 proactive sync is unchanged. NIP-11 advertises GRASP-03 only while the overlay is enabled.

This proposal is stacked on nostr:nevent1qqsrhrv4efjjrwgkct52xd0qa0rz24s4jmns6gznl9wv45a8xn8f9dgpz3mhxue69uhhyetvv9ujumn8d96zuer9wcn2g594 so it inherits the restored CI baseline.

## Validation

- Sync+ configuration tests: 2/2 passed.
- NIP-11 tests: 12/12 passed.
- `cargo clippy --locked --workspace --all-targets -- -D warnings`: passed on the combined stack.
- `cargo fmt --all -- --check`: passed.
- NixOS module parse and `git diff --check`: passed.
 2026-08-12 18:10:02 +01:00
DanConwayDev c38b7b8f24 Merge #3b8d95ca: fix: restore CI and auxiliary sync coverage 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsrhrv4efjjrwgkct52xd0qa0rz24s4jmns6gznl9wv45a8xn8f9dgzrwsj3

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

## Outcome

Restore all CI gates and repair the auxiliary live-coverage regression exposed while doing so. Full repositories without accepted roots once again enter bounded auxiliary maintenance, restoring `q`-tagged delivery after priority demotion.

## Validation

- `cargo test --locked`: passed; sync 95 passed, 0 failed, 1 ignored, and every other unit, integration, and doc-test target passed.
- `cargo clippy --locked --workspace --all-targets -- -D warnings`: passed.
- `cargo fmt --all -- --check`: passed.
- `git diff --check`: passed.

The commit sequence keeps mechanical formatting, stale test contracts, observable readiness, the runtime coverage fix, and lint cleanup independently reviewable.
 2026-08-12 18:08:04 +01:00
DanConwayDev 5dfa954265 Merge #7e850008: feat(sync): discover conversations through NIP-65 mail… 
feat(sync): discover conversations through NIP-65 mailboxes

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs8apgqpp9pay0yr20l3ev24n39x65penwld92f0a595jpqk3te4rcld7rej

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

# Recover accepted conversations through NIP-65 inbox relays

## Production problem

GRASP follows repository relay ownership and accepted root references, but a
participant may publish replies or updates only to inbox relays advertised in
their NIP-65 relay list. Without bounded identity and inbox discovery, an
accepted conversation can remain incomplete even though its author published
enough information for another client to find it.

This deliberately narrow proposal builds on the merged bounded repository-fetch
work from nostr:nevent1qqsxe24pvp4pqxuzvpvdrdpgx3j2ug9yqtl6ny8z8mjw5k7djmludxspz3mhxue69uhhyetvv9ujumn8d96zuer9wcz87vs2.

## Approach

- Limit identity discovery to owners and declared maintainers of accepted Full
  announcements plus authors of accepted roots. StateOnly/purgatory repositories
  and unrelated retained authors remain inert.
- Query configured user-index relays for accepted authors missing kind 10002,
  requesting kind 0 alongside it in batches of at most 100. Once a list exists,
  follow its write/unmarked outboxes additively for newer replacements.
- Admit, retain, broadcast, and serve identity events through the normal policy
  path. Only a successfully accepted/stored relay list may steer sync.
- Rebuild inbox ownership from retained kind 10002 events before network refresh,
  so restart does not make users depend on the index source remaining online.
- Feed inbox roots into the existing tiered descendant live, historic, and
  rotating REQ+EOSE machinery. There is no recursive author expansion or
  parallel data plane.
- Keep discovery globally single-flight, ledger- and pacer-accounted, and require
  immediate transient capacity before starting it. Successful discovery refreshes
  after 24 hours; incomplete discovery retries after five minutes.
- Apply replacement/removal to desired work immediately but allow physical live
  coverage to retire at natural EOSE/CLOSED/disconnect or ordinary consolidation.
  Runtime accepted-root deletion remains restart-reconciled because the
  self-subscriber has no bounded deletion signal.
- Treat a relay reached only for NIP-65 discovery as a discovery-only connection:
  it shares ordinary authorization, pacing, and subscription-ledger safety, but
  does not start repository/announcement sync merely because it served identity
  discovery. One such source is dialled per maintenance pass, drained promptly,
  and promoted cleanly if it later becomes an actual repository source.

The older broad proactive-sync proposal
(nostr:nevent1qqszdapt0kvrhzgskxyqqu7aaudlt3s0cjj5v2t9fnky9v4cws2gahgw9lqz4)
also proposed recursive scopes, response-author expansion, fallback relay
configuration, and separate orchestration. Those pieces remain deliberately
excluded. Simple is doing real work here.

## Reviewable commits

- `1129d03` records the minimal design and exclusions.
- `831eac2` derives accepted-root inbox overlays as pure indexed state.
- `ced9706` adds bounded NIP-65 discovery through the existing sync pipeline.
- `c8e49f2` bounds discovery, replacement, promotion, retry, and retirement.
- `07fe32` makes identity admission and retention atomic and restart-safe.
- `2538f90` exposes the three-relay user-index default through all four config
  surfaces.
- `6fa84d5` changes replacement/removal to natural live-coverage retirement.
- `870e125` expands discovery to accepted owners/maintainers and convergent
  NIP-65 outboxes.
- `3917238` isolates integration sources so the end-to-end claim is meaningful.
- `2f859b0` fixes a production-found lifecycle error by isolating discovery-only
  connections from ordinary fresh-start sync.

## Validation

- `cargo test --lib`: 729 passed on the proposal before the final lifecycle
  refinement; `cargo check --lib` passes on exact tip `2f859b0`.
- The focused proactive-sync integration passes on exact tip in 18.88 seconds.
  It covers identity admission, index discovery, outbox convergence, inbox-only
  reply recovery, StateOnly-to-Full promotion, natural retirement, and restart
  recovery with the discovery source offline. It also proves an advertised
  outbox used only for discovery never receives ordinary fresh-start sync.
- The exact-tip full `--test sync --test-threads=1` run completed with 84 passed,
  11 failed, and 1 ignored. A representative failure
  (`test_live_sync_layer3_events`) reproduces unchanged on exact merged parent
  `ff2280d`; the old live-sync/full-file failures are not hidden or expanded into
  this proposal.
- The release Nix build completed and its library tests passed during the remote
  deployment build.

## Paired production burn-in

Exact candidate `2f859b0` and exact merged parent `ff2280d` started from separate
empty datasets at 2026-08-12 11:10:45 UTC. Both follow the same three original
GRASP upstreams (`gitnostr.com`, `relay.ngit.dev`, and `git.shakespeare.diy`), run
in the same background slice, have `MemoryHigh=8G` and `MemoryMax=12G`, and are
scraped by Prometheus. Public gitnostr.com remained on its released package.

The feature deliberately yielded to initial historic work. At 11:32:07 it
activated its first accepted-author batch (43 authors, 4 inbox relays). Continued
bounded rounds followed the accepted inventory as historic sync discovered it;
by the 14:08 checkpoint the candidate retained 430 distinct kind-10002 authors,
472 profiles, and covered 338 inbox relays. The control intentionally retained
no identity events because it has no Sync+ discovery path.

A direct-ID attribution check avoided treating asynchronous total event counts
as a feature delta. At 14:08 the candidate and control retained 26,066 and
26,384 unique events respectively, but their independent historic syncs had
accepted different roots and those totals are not directly comparable. The
candidate had 166 accepted-root descendants absent from the control. All 166
were queried directly from each of the three original upstreams in 10-ID REQs:
17 successful EOSE rounds per upstream, with zero failed rounds. **37 remained
absent from all three original upstreams**: kinds 1 (5), 6 (1), 7 (8), 1111
(11), 1630 (1), 1632 (5), 1985 (2), 9735 (3), and 39701 (1).

This is a strict point-in-time lower bound, not the total feature gain. It counts
only descendants retained by the candidate, absent from the independently
progressing control, and repeatedly unavailable by direct ID from the original
source set. Events that Sync+ found earlier but the control later acquired are
deliberately not claimed.

Cold sync briefly filled the candidate's 1,000-event data lane around 11:34,
with an 85.2-second worst queue delay while large new source batches arrived.
Further finite discovery waves at 12:03 and 12:08 expanded coverage to 144 and
then 198 inbox relays and produced another backlog (39.5-second worst queue delay,
depth 256). This was material pressure, not hidden from the recommendation. It
recovered without restart or failure: by 12:19 recent reports showed current
depths mostly 0-31 and the most recent busy relay window was down to 4.34 seconds.
The control also experienced cold-burn pressure on the shared constrained host.
This evidence therefore claims bounded recovery after the full discovery
frontier, not that extra coverage is free or improves baseline throughput.

At the final 2-hour-57-minute checkpoint both archives had zero restarts and no panic,
OOM, hard-memory-limit, or repository-corruption signature. Both exercised the
8 GiB soft boundary and correctly paused background Git fan-out; neither recorded
a `MemoryMax` event. Candidate current/peak memory was 8.59/8.59 GB versus the
control's 8.28/8.59 GB. Expected hostile-relay connection, signature, timeout,
and rate-limit noise occurred in both instances. The three Prometheus targets,
public gitnostr.com, caddy, and Prometheus remained healthy throughout.

## Deliberately shortened convergence measurement

We stopped waiting for ecosystem-wide cold-sync convergence after 2 hours 57
minutes. Productive work was still flowing: in the preceding 15 minutes the
candidate completed historic work on 26 relays and processed 3,000 events. Its
eligible-author upper bound was 884, comprising stored root authors plus stored
announcement owners/maintainers without externally reproducing the in-process
accepted-Full filter; 430 had a retained relay list and at most 454 remained
without one. Dead/unreachable relays are not treated as unfinished useful work:
they back off to one low-cost probe per 24 hours.

This cuts short the previously estimated remaining convergence wait by roughly
30 minutes to 2.5 hours. It does not cut short the stability soak: nearly three
hours exercised multiple discovery waves, soft-memory pressure, historic
pagination, and recovery. Global convergence is an unsuitable merge boundary
because newly accepted repository inventory can legitimately extend the frontier
while the service runs.

## Recommendation

**Ready to merge at exact tip `2f859b0`.** The focused integration proves the
bounded lifecycle, the paired production burn proves real incremental recovery,
and the constrained archive survived the expanding frontier without restart,
OOM, or hard-memory failure. Remaining uncertainty affects the eventual size of
the benefit—not correctness or operational safety—and does not justify bloating
this proposal or holding it open indefinitely.
 2026-08-12 15:16:47 +01:00
DanConwayDev ff2280d9d5 Merge #87ff640e: fix(sync): bound peer-closed subscription recovery 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsg0lmype9p7dgaesvuadserhevmfu4we88herl6n2lgw37vc73urggazghv

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Production gitnostr.com on v2.1.2 (`bb38e6c`) accumulated 132,902 desired subscriptions for `nostr.wine` against a two-subscription baseline. Its RSS rose from roughly 4 GiB to 53 GiB before host-wide OOM pressure. Logs tied the growth to about 50 `auth-required` CLOSED responses per second.

rust-nostr deliberately retains an `auth-required` subscription when an authenticator exists so it can answer the NIP-42 challenge and retry the same subscription ID. Our live-coverage actor independently generated replacement IDs, did not classify this refusal, and therefore left each rejected generation in the SDK registry.

The first `auth-required` CLOSED now preserves the same SDK subscription and ledger slot for that NIP-42 retry. If the same subscription is refused again, ngit-grasp retires it from both the SDK registry and its ledger and applies the existing 24-hour policy pause. Every other peer CLOSED is terminally unregistered because SyncManager owns deliberate recovery. An otherwise unclassified live CLOSED enters the existing 65-second cooldown before one repair attempt instead of rebuilding immediately. The connection remains open throughout.

This deliberately does not disable authentication or change rust-nostr globally. Correctness assumes a peer CLOSED is terminal except for rust-nostr's one same-ID NIP-42 retry, and that SyncManager remains the sole owner of replacement generations.

Validation:

- `cargo test --lib`: 711 passed, 0 failed.
- Deployed exact tip `80086ceb9922ad2418215139bf95cde10030049b` to gitnostr.com and relay.ngit.dev.
- gitnostr.com restarted from roughly 53 GiB RSS to about 1.5 GiB cgroup memory; relay.ngit.dev restarted to about 1.4 GiB. Both retain `MemoryHigh=32G` and `MemoryMax=48G` containment.
- Both services remained active with zero restarts. gitnostr.com had no warning-level entries in the initial observation; relay.ngit.dev's warnings were bounded policy refusals rather than recovery churn.
- relay.ngit.dev exercised the complete NIP-42 path in production: first `nostr.wine` refusals retained their same subscription IDs for authentication, a repeated refusal retired that ID and entered the 24-hour policy pause, and the former roughly 50-per-second replacement storm did not return.
- relay.ngit.dev recorded zero repetitions of its prior `search filter is required` CLOSED storm.
- Prometheus now scrapes cgroup CPU/memory metrics every 15 seconds on both hosts with 30-day, 2 GiB-bounded local retention, so renewed growth is historically queryable rather than dependent on point-in-time inspection.

The archive candidate/control instances were disabled during incident recovery so they cannot compete with gitnostr.com while this exact tip is observed.
 2026-08-12 11:27:38 +01:00
DanConwayDev 9a9b3a0c58 Merge #6caaa160: Bound duplicate repository fetches and hedge slow sour… 
Bound duplicate repository fetches and hedge slow sources

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsxe24pvp4pqxuzvpvdrdpgx3j2ug9yqtl6ny8z8mjw5k7djmludxsxzmeud

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

# Bound duplicate repository fetches and hedge slow Git sources

## Production problem

The archive burn-in observed several simultaneous `git fetch`/`index-pack`
pipelines writing into the same large bare repository. Existing controls were
keyed by remote domain, protecting each upstream server from our request rate,
and by aggregate cgroup pressure, protecting the host after pressure appears.
Neither prevented independent identifier/domain workers duplicating expensive
work against one local object database.

## Approach

- Reserve per-domain fetch-pass capacity atomically and release it through an
  owned cancellation-safe permit. `ls-remote` remains inside this protection.
- Coordinate work by resolved local repository path. Duplicate callers wait
  for the active wave and then re-check missing OIDs.
- Give one preferred source a 30-second head start. If it remains incomplete,
  permit exactly one source on another domain to hedge; never start a third.
- Make concurrent object ingestion free of `FETCH_HEAD` and automatic
  maintenance side effects, and serialize promotion of fetched OIDs.
- Stream bounded Git stdout/stderr and expose aggregate activity, duration,
  outcome and byte metrics. Five minutes without either stream producing a
  byte terminates that command's private process group; there is no total
  duration limit, so an active large transfer may run indefinitely.
- On inactivity, send the whole process group SIGTERM, retain its unreaped
  leader for a ten-second grace period, then SIGKILL before reaping. A guard
  applies the same group cleanup if the owning Rust future is cancelled.
- After an unproductive primary/hedge pair has fully ended, try the remaining
  sources sequentially. A third fetch never overlaps the bounded pair.

The two attempts deliberately do not produce a learned source ranking. They
write into the same object database, so either command can observe an object
installed by the other and completion order cannot attribute the object to a
source. Truthful ranking would require isolated per-attempt object stores and
explicit winner promotion, which is excluded here in favour of the smaller,
correct hedge.

The 30-second hedge delay is not a timeout. The separate five-minute inactivity
limit measures transport silence rather than elapsed runtime; every output byte
resets it. Timing remains operational evidence only and never ranks sources.

## Commits

- `b9ed6e8` atomically reserves remote-domain capacity.
- `3903ffe` serializes work per object database and rechecks waiting demand.
- `665af5b` removes shared fetch-head and maintenance side effects.
- `e1eee18` adds exactly one delayed distinct-domain hedge without scoring.
- `0d7c77d` introduces bounded concurrent stream draining and operational
  metrics without changing completion policy.
- `25c535f` adds inactivity recovery, cancellation-safe process-group cleanup,
  and remaining-source fallback.

## Validation

- Full library suite: 709 passed.
- Deterministic scenarios cover duplicate-demand single-flight, independent
  repositories, delayed hedge victory, two slow sources with no third attempt,
  and same-domain alternatives excluded from hedging.
- Domain admission contention and cancellation tests pass.
- Stream capture remains bounded while readers continue draining/counting.
- Two real concurrent object-only Git fetches into one bare repository leave
  `FETCH_HEAD` absent, all requested objects readable, and `git fsck` clean.
- Tests cover a productive delayed hedge after an empty primary and prove a
  shared-object observation cannot become source preference.
- An active-output fixture survives a short test inactivity threshold. Silent
  TERM-resistant process groups are forcibly reaped; cancellation also kills
  descendants before follow-up work proceeds; an unproductive pair reaches a
  productive third source without three-way concurrency.

## Archive validation

The earlier `dfa9afc` burn-in demonstrated the hedge and foreground health but
also exposed that its persisted rankings were not attributable in a shared
object database. That recommendation was withdrawn.

Exact corrected tip `890321b867e8cf4765b1470b65c27b0ea4682526` ran on the
constrained disposable archive from 2026-08-10 21:57:38 UTC. In the first
9 minutes 47 seconds it:

- completed 325 repository Git passes and promoted 146 queued repositories;
- exercised 97 successful hedge advertisements and 87 successful hedge batch
  fetches, so the delayed alternative path was active under real load;
- recorded zero restarts, panics/OOMs, Git corruption signatures, or failed
  service health;
- reached 3.50 GiB peak memory and settled to 19 tasks while the existing
  resource-pressure gate paused new fan-out ten times rather than disrupting
  foreground work; and
- emitted one 60-second activity warning for a primary `relay.ngit.dev` batch
  that was still producing progress (`quiet_secs=0`, 20,835 stderr bytes).
  It was observed rather than killed, as intended for a potentially large
  legitimate fetch.

That run remains useful threshold evidence: the only 60-second warning had
`quiet_secs=0`, so the legitimate large fetch would not have approached the new
five-minute inactivity boundary. It predates process-group recovery, however.

Exact corrected tip `25c535f96aa9969543116a5c96a77aa5971130a6` then ran as
the control in the paired constrained archive from 2026-08-12 01:53:19 UTC. By
03:11 UTC it had completed 1,785 Git passes and fetched 2,985 requested OIDs.
The real workload exercised both sides of the hedge: 50 hedge batch fetches and
6 hedge residual fetches completed successfully, alongside 289 primary batch
and 168 primary residual successes. All subprocess gauges returned to zero.

Across that run the exact tip had zero service restarts, surviving zombie Git
processes, panic/OOM, or repository-corruption signatures. Memory peaked at
4.76 GB and returned to 1.42 GB while the service remained responsive; its
`/metrics` endpoint answered in 15.6 ms at the settled comparison. No
five-minute inactivity termination was needed in this workload. Earlier
threshold evidence had already shown a legitimate 60-second fetch continuing
to produce output, and the final implementation correctly leaves such active
work unbounded by total duration.

The burn also exposed one pre-existing synchronous local-OID-copy path outside
this proposal's observed network-fetch/process-group wrapper. A single zombie
was temporarily pinned behind that caller during an earlier refinement run,
did not accumulate, and cleared on the deliberate restart. It is not caused or
masked here and remains excluded from this bounded network-fetch change.

This exact-tip constrained production evidence, together with the deterministic
process-group and fallback tests, supports merging this proposal.
 2026-08-12 07:17:47 +01:00
DanConwayDev 5c1fbb2fea Merge #75950a9a: Protect foreground service from cold-sync Git fan-out 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs8t9g2nggd9rsmshw2pumpefnw0rpavl2zpwjnsh7sy0sv9fvdnkgasgyjf

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

# Protect foreground service from cold-sync Git fan-out

## Production problem

A clean archive burn-in was run under `CPUQuota=200%`, `MemoryHigh=12G`, and
`MemoryMax=16G` from 2026-08-10 15:14:20 UTC. Per-domain purgatory throttles
were individually respected, but a cold sync spanning many domains created
roughly 149 service tasks, dominated by Git subprocess pipelines.

Against the same ngit-grasp process during that workload:

- idle NIP-11 latency of about 0.3 ms rose repeatedly to 1-5+ seconds;
- several NIP-11 requests timed out after five seconds;
- WebSocket connection/query probes intermittently failed;
- two-second Prometheus scrapes timed out; and
- cgroup CPU throttling accumulated continuously.

This demonstrates actual foreground starvation. The existing per-domain limit
protects each remote server, but does not bound aggregate work across many
different domains.

## Approach

Place complete event-directed background Git passes behind a small
resource-pressure gate:

- The first one-second observation window admits two passes per effective CPU,
  discovered from Linux cgroup v2 `cpu.max` or available processor count.
- Each healthy observation window grows the allowance by 20% or one pass,
  whichever is greater, so there is no permanent aggregate limit on an
  unconstrained machine.
- New cgroup CPU throttling or memory-high pressure reduces new admission to
  one pass while already-running work drains naturally.
- Sustained pressure is logged at most once per minute, keeping the operational
  signal without flooding logs as healthy and throttled samples alternate.
- Existing per-domain concurrency and request-rate protections remain intact.
- Foreground Git pushes do not use this outbound recovery path.

This deliberately avoids configuration, a permanent concurrency ceiling, and
a general-purpose priority scheduler.

### Why ramp before pressure exists

The original implementation had no aggregate limit: its per-domain limits
could collectively launch work across hundreds of independent domains. A
fixed aggregate ceiling would solve that burst but needlessly constrain hosts
with ample resources. Waiting for pressure before applying any restraint also
proved too late: cgroup CPU and memory signals are retrospective, so roughly
149 tasks had already escaped before the kernel could report their cost, and
they had to drain before foreground service recovered.

The initial allowance is therefore a one-second discovery probe rather than a
lasting limit. Healthy observations increase it without a ceiling; pressure
only pauses new admission while existing work drains. Growth is deliberately
20% or one pass rather than strict doubling: `4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10
-> 12 ...` gives each preceding step time to appear in the next pressure
sample, while an unconstrained host continues opening capacity. For
best-effort background recovery, a few seconds of slower capacity discovery is
preferable to risking foreground relay availability.

## Validation before production comparison

- `nix develop -c cargo test --lib`: 692 passed before the final isolated
  ramp-growth refinement.
- Exact-tip throttle tests: 14 passed, including gradual unbounded growth.
- Throttle-focused tests cover the initial probe, CPU-quota parsing, pressure
  detection, and existing per-domain behavior.
- Targeted rustfmt and `git diff --check` pass.

## Production validation

The strict-doubling predecessor `7c16ff9` first completed a clean, roughly
ten-minute burn-in under the same two-CPU, 12/16-GiB constraints. All 23
sampled NIP-11 requests and all Prometheus scrapes succeeded; task fan-out
stayed at 24-55 rather than the baseline's roughly 149; memory peaked at 5.23
GiB; and the service completed 344 Git passes while genuinely CPU-throttled.
There were no restarts, panics, allocation failures, or OOM events.

Exact final revision `434e43fc6240be24957bf876bedf6db10c919ed5` replaces doubling with
20%/+1 growth. It was deployed only to the disposable archive, whose dataset
was deleted again before a clean burn-in from 2026-08-10 19:36:55 UTC.
Through 19:40:54 UTC:

- all 10 sampled NIP-11 requests succeeded in 0.0003-0.37 s and all sampled
  Prometheus scrapes succeeded in 0.012-0.60 s;
- WebSocket REQ/EOSE completed successfully in 1.87 s;
- service task fan-out stayed at 12-34 and memory peaked at 3.11 GiB;
- 1,799 of 2,410 CPU periods were throttled, exercising the pressure path;
- the process nevertheless completed 64 Git passes and fetched 645 requested
  OIDs, reaching 2,320 repositories and a 1.8 GiB archive dataset; and
- there were zero memory-pressure/OOM events, restarts, panics, or allocation
  failures. Four pressure warnings over four minutes confirmed the
  one-per-minute bound.

The candidate therefore preserves foreground relay operation under the same
cold-sync pressure that reproduced starvation, while continuing to make
substantial sync progress. This exact revision is recommended for merge.
 2026-08-10 20:08:07 +01:00
DanConwayDev 5bad62375b Merge #347f6831: sync: tier live coverage under connection pressure 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsrglmgx8q2u69zk6zppglyvn9q45cfv5juy2yv7khq0a23xlpl5hg54dczz

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Reference coverage now has ordered capacity tiers instead of a binary core/descendant split. Announcements, state, canonical repository `a`, and canonical root `e` coverage remain essential; root `E`, core `q`/`A`, descendant `e`/`a`, and descendant `q` are admitted as complete priority tiers when the shared NIP-11-aware ledger has room. Descendant `E`/`A` and every tier below the cutoff use the existing cursor-overlapped REQ+EOSE rotation.

The four commits are deliberately reviewable in isolation:

1. Name and test the ordered filter vocabulary without changing wire behaviour.
2. Apply the complete-tier live cutoff and paced packed fallback through the existing ledger and five-second scheduler.
3. Union core and descendant reference values before historic grouping, reducing overlapping delivery without changing coverage.
4. Preserve the complete auxiliary frontier when any admitted live subscription receives CLOSED, avoiding a reconciliation-window gap.

## Validation

- `nix develop -c cargo test --lib`: 686 passed before the final focused lifecycle assertion; the final tip's focused lifecycle test also passes.
- All three descendant live/fallback/addressable integration scenarios pass together.
- All three adaptive-pagination scenarios pass.
- `nix build .#ngit-grasp` passes; the final small lifecycle commit was then built successfully by the archive's remote Nix deployment.
- A full `--test sync` run retained timing failures that reproduce on master under the already-merged per-relay query pacer. The branch-specific descendant failures exposed by that run were fixed by rotating one relay-compatible packed REQ rather than singleton filters.

## Archive production evidence

Exact proposal tip `27f31e50ed86ce83f12bd247c4bb7be8b8cbf2b0` was deployed to the disposable archive burn-in instance at 10:13:51 UTC on 2026-08-10 and observed through 10:41:48 UTC. The service remained active with the original PID and zero restarts. Memory was 611 MiB current / 613 MiB peak at the final trend sample while archive hydration was still active.

The archive exercised both sides of the policy across real relay sizes and advertised capacities:

- `git.shakespeare.diy` (fallback budget 20): 4,587 event IDs plus 5 coordinates; 36 filters stayed live in 14 subscriptions and 11 filters became 5 packed rotating REQs.
- `nos.lol` (advertised budget 20): 2,636 event IDs plus 12 coordinates; 24 filters stayed live in 8 subscriptions and 7 filters became 3 packed rotating REQs.
- `relay.primal.net` (advertised budget 20): 850 event IDs plus 3 coordinates; 16 filters stayed live in 5 subscriptions and 3 filters became one rotating REQ.
- `relay.kcbitcoiners.com` (advertised budget 2,000): ordinary auxiliary tiers fit live; only the deliberately historic-only tier rotated.

Across the window, 17 distinct relays exercised persistent tier admission and 17 exercised fallback rotation. The service started 457 packed fallback batches and confirmed 472 descendant-purpose batches. Five early confirmations inherited a broader batch failure flag but each corresponding cursor reached terminal success and later cycles continued; no auxiliary live subscription received CLOSED. There were zero query-rate-limit responses, zero subscription-limit responses, zero panics/OOM signatures, and no unexplained auxiliary-coverage retirement (the sole retirement followed a real frontier change and was reinstalled).

The observed watchdogs were two pre-existing empty historic pages from `relay.nostrcheck.me`; neither involved the new descendant rotation. Four policy refusals came from the access-restricted `relay.getalby.com/v1` and followed the existing 24-hour deferral path. External authentication failures and peer resets remained isolated to their established recovery paths.

## Recommendation

Ready to merge. The archive demonstrated live and rotating coverage concurrently across fallback and advertised capacities from 20 to 2,000, with packed multi-filter rotation completing repeated full cursor laps on the largest relays. The remaining remote failures are pre-existing peer policy, authentication, transport, and historic-page recovery cases rather than regressions in this proposal.
 2026-08-10 15:44:29 +01:00
DanConwayDev d318226849 Merge #ba2d248c: feat(sync): follow addressable descendant references 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqst5tfy3s2fsrz8psulwwgs72klm9d3jyxhwnxzkjdmdjpf979jfyqhs0akd

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Repository descendants can refer to a direct thread member by its NIP-01 address rather than its event ID. The existing non-recursive frontier retained only event IDs, so `a`/`A` tags and coordinate-valued `q` tags could fall outside both live and historic coverage.

This represents the frontier as event IDs plus coordinates derived from valid replaceable and addressable direct members. Both sets use the existing live admission, five-second fallback rotation, and ordinary historic sync paths. Malformed addressable events without `d` do not contribute a coordinate.

The one-generation boundary and existing connection-budget policy are preserved. Recursive discovery and connection sharding remain deliberately excluded.

## Validation

- `cargo test --lib`: 682 passed.
- All three descendant integration scenarios pass together, covering retained live mode, constrained event-ID fallback, and constrained `a`/`A`/coordinate-`q` fallback. The constrained fixtures advertise four subscriptions so auxiliary live coverage cannot legitimately fit.
- The final proposal commit is `533cb6279d2197d624e2d6b622e8abba86ecc56d`.

## Archive production evidence

The runtime change soaked on the disposable archive instance from 08:38:00 to 08:57:33 UTC on 2026-08-10. Later amendments changed only the test capacity and commit prose; the runtime tree was unchanged. The archive pin now names the final proposal commit, running from `/nix/store/nb0vw0im8xp4x61xzhdymh88rfn8wwmy-ngit-grasp-2.1.2` since 09:02:44 UTC.

Real addressable frontiers were retained within the shared subscription ledger:

- `nos.lol`: 2,620 event IDs plus 11 coordinates, 21 filters in 8 subscriptions.
- `relay.primal.net`: 840 event IDs plus 2 coordinates, 9 filters in 3 subscriptions.
- `nostr.bitcoiner.social`: 166 event IDs plus 7 coordinates, 6 filters in 1 subscription.
- `ngit.danconwaydev.com`: 2,547 event IDs plus 34 coordinates, 21 filters in 8 subscriptions.
- `relay.mostr.pub`: 143 event IDs plus 1 coordinate, 6 filters in 1 subscription.

Across the evidence window there were zero query-rate-limit signatures, zero watchdog signatures, zero relay-disconnect logs, and zero panic/OOM signatures. The service did not restart. Existing unrelated remote-relay hydration and policy failures remained bounded by their established recovery paths.

The constrained fallback was proven deterministically by the integration scenario; the archive window exercised live coordinate coverage but did not naturally reach a coordinate-bearing constrained relay before the final provenance redeploy.

## Recommendation

Ready to merge. The implementation extends the existing bounded mechanisms rather than adding another scheduler, subscription class, or connection.
 2026-08-10 10:05:15 +01:00
DanConwayDev e61bee6f58 Merge #599cc928: sync: minimise live subscription churn 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs9n8xf9rnczwhv6u6yagtp95u8y45fd2pahtthkfensw5nnaekx0sls8puq

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Built on the descendant-sync commits merged by #2b377e20 and implements nostr:nevent1qqswhdur5f7z8zwy726za854qhahnt9k5ytru2h6zqyysrzdskwzk9cpz3mhxue69uhhyetvv9ujumn8d96zuer9wcykq3rj. The proposal and current master share exact descendant tip `9f3c943`, so it can merge without rebasing or duplicating commits.

Repository discovery is batched every five seconds. Previously each batch added live subscriptions until a coarse threshold rebuilt the complete live set, needlessly closing healthy core and descendant subscriptions.

This proposal is split for review:

1. `186d9a8` makes repository filter chunking deterministic, giving stable grouping independent of HashMap insertion order.
2. `9a1fa8d` uses the real filter-count and serialized-byte grouping result to select useful consolidation work. Full core and descendant groups remain open. If repacking the complete mutable tail releases slots, it does so; if byte limits mean the tail cannot shrink, it replaces only the smallest useful subset needed to reduce additional slot usage. Failed CLOSE or replacement-open operations restore the exact retired tail. Capacity pressure retains the existing deferred full-regroup backstop while historic recovery continues.

Validation:

- `cargo check --lib`
- `cargo test --lib`: 680 passed
- tests cover stable full/auxiliary IDs against a real embedded relay, deterministic grouping, count and byte constraints, replacement-open rollback, and partial-CLOSE rollback
- 13 small partial groups plus one new filter become 2 replacements, releasing 11 slots
- 17 byte-bound groups plus one small filter replace only 1 group, avoiding a new slot without rebuilding the other 16
- a byte-full group which cannot help is left untouched

Disposable archive production refinement:

- the count-only version exposed the exact byte-bound case: a one-filter update on relay.ngit.dev retired 17 groups and reopened 17, releasing no slots
- exact final tip `9a1fa8d`, under the equivalent roughly 50,000-ID / 167-chunk startup workload, preserved 17 groups and replaced exactly 1 for the same one-filter update, with zero additional slots
- the final-tip service remained active with zero restarts, zero candidate-specific rate-limit, ledger-overrun, rollback, or panic errors, and about 349 MB peak during the comparison window
- the public gitnostr.com service remained on released v2.1.2 throughout

Earlier archive evidence also established that descendant permanent coverage, multi-subscription splitting, constrained fallback rotation, and historic completion continue correctly beneath this change.

Recommendation: ready to merge.
 2026-08-10 08:29:28 +01:00
DanConwayDev b9eebb2e25 Merge #2b377e20: feat(sync): schedule direct-member descendant history 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqszkdm7yq5lkte4ycw72r3esglyp5z2g70cdqm0gv9zkfgyrttk2kq0su8wr

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Closes #727b0992 with the capacity-aware hybrid design agreed in review.

Repository collaboration events may reference only their immediate parent. Existing repository/root filters can therefore miss replies to a discovered reply when those events omit repository and root tags.

The proposal is intentionally split into four reviewable commits:

- `d203f9d` refactors the existing historic REQ path around one shared queued submission helper, without changing scheduling.
- `5f84b8b` includes known direct-member descendants in ordinary historic coverage.
- `4dced40` retains separately consolidated descendant live coverage when core live coverage, a two-slot control reserve, and at least one transient historic slot all fit the relay per-connection budget.
- `9f3c943` otherwise rotates descendant filters through the same transient ledger, starting at most one queued or active descendant request per relay, on a five-second cadence.

Fallback requests use REQ+EOSE, advance their cursor only after successful EOSE, and overlap the prior successful upper bound by 15 minutes. They do not skip a filter merely because the transient slot is busy: the request waits in the existing queue. Permanent coverage is paired with the complete historic baseline so events created before installation are not missed.

The design deliberately does not add request-class priority, recursive descendant traversal, durable cursors, multi-connection sharding, new configuration, or minimum-churn core consolidation. The latter is tracked separately at nostr:nevent1qqswhdur5f7z8zwy726za854qhahnt9k5ytru2h6zqyysrzdskwzk9cpz3mhxue69uhhyetvv9ujumn8d96zuer9wcykq3rj.

Local validation:

- `cargo check --lib`: passed
- `cargo test --lib`: 673 passed
- forced-fallback integration at advertised capacity 5: passed
- permanent-live integration: passed
- 600-member unit scenario confirms descendant filters split across multiple live subscriptions
- full `cargo test --test sync`: 83 passed, 10 failed, 1 ignored; failures retain established load-sensitive signatures while both new descendant scenarios pass under whole-file load

Disposable archive production evidence:

- exact tip `9f3c943` and the behavior-identical pre-amend executable both started with zero restarts and no candidate-specific rate-limit, panic, or ledger-overrun errors
- dependency sync queued 2,293 IDs in 8 chunks and 34,868 IDs in 117 chunks without starving later work
- permanent descendant coverage was installed across real relays; examples include 24 filters split over 12 subscriptions on relay.damus.io, 9 over 3 on grasp.budabit.club, and 6 over 2 on git.nostrhub.io
- the naturally constrained git.shakespeare.diy session had a 20-slot fallback budget; 27 descendant filters could not fit alongside 1,279 repositories and 961 roots, so filters 0, 1, and 2 rotated sequentially and completed successfully without skipping the busy transient queue
- the refined exact tip remained stable with bounded memory (about 517 MB peak during its observed window)
- the stacked follow-up executable retains the same descendant behavior and is now exercising a 49,253-ID dependency batch on the same archive dataset without restart or rate-limit failures

Recommendation: ready to merge.
 2026-08-10 08:27:15 +01:00
DanConwayDev 7b5b81ff14 Merge #a44ff1d9: fix(sync): adapt query starts after rate-limit exhaust… 
fix(sync): adapt query starts after rate-limit exhaustion

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs2gnl3m9fv4zvge4mfc0rzfgpgfe7l4zemrr62crwsvfjcdp5u9uq9y99ms

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Production showed a roughly 67-second oscillator against relay.ngit.dev: simultaneous subscriptions stayed bounded, but query turnover exhausted a separate 120-query-per-minute allowance after every cooldown. That allowance is a newly enabled rust-nostr 0.45 LocalRelay default, not an ecosystem floor: the preserved cross-relay survey in commit 9723ff4 found no other audited implementation enabling an equivalent query-specific per-connection default. A finite limit remains useful as one DoS layer, but a small per-connection bucket is weak protection on its own because connections can be multiplied.

Production iteration also showed that rust-nostr charges multiple SDK-managed NEG-MSG continuation frames inside one admitted NIP-77 round. This revision therefore paces all application-visible starts only after an explicit refusal and uses paced REQs for the rest of that session, because internal NEG-MSG frames cannot honor the gate. It intentionally does not proactively pace startup history: doing that safely requires request-class priority so live coverage is not delayed and is a separate follow-up.

Validation: 657 library tests, 89 sync integration tests (1 ignored), and nix build .#ngit-grasp. Production soak of the behaviorally identical pre-documentation tip: one expected startup refusal; immediate 600 ms pacing and session REQ fallback; zero further refusals across more than four former failure periods; four successful ReqEose batches advancing 13 state-only repositories; NRestarts=0. Final commit 86d19b7 changes documentation and comments only relative to that soaked runtime.
 2026-08-08 09:52:24 +01:00
DanConwayDev 50033e0157 Merge #97556042: chore(sync): diagnose missing transient terminals 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsfw4tqgfrz874nyp8ae9h08dsrj0zc44w7p5qcgc5xgx7m5epu4rctceh6k

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Production at `58ef8682` repeatedly held transient subscriptions until the 120-second watchdog across independent relays and request classes. Diagnostic tip `b55d2de` was deployed to gitnostr.com at 2026-08-07 14:52:34 UTC against a directly comparable startup workload.

The diagnostics established that expired subscriptions were already absent from rust-nostr's active subscription map, so the relay terminal had arrived but ngit-grasp had not released its local ledger ownership. An initial 4,096-message queue removed watchdogs from a 36,054-ID relay.ngit.dev burst, but production falsified queue sizing as a general solution: the five-per-cycle signature moved to ngit.danconwaydev.com and then git.shakespeare.diy, including four empty responses on each peer. Empty responses exposed a second ordering hole, while the shifting high-volume peer showed that any finite EVENT queue can remain congested under sustained pages.

The final proposal fixes both lifecycle dependencies by construction:

- register a second rust-nostr broadcast receiver synchronously as a terminal-control lane; it consumes EOSE/CLOSED and connection terminal status independently of the processor-facing EVENT queue;
- pre-generate every transient subscription ID and register its generation-scoped permit before sending the REQ, pass that ID to rust-nostr, and roll registration back on subscribe failure.

The terminal listener is the sole transient-release path during a connected session. The processor listener still delivers ordered EVENTs, pagination terminals, and live CLOSED restoration. EOSE still enqueues CLOSE before returning the slot; CLOSED and connection teardown remain definitive boundaries. The EVENT queue stays at its original bounded 1,000 messages—correctness no longer depends on buying lifecycle headroom with memory.

Subscription concurrency, filter contents, pagination, watchdog timing, retry behaviour, live lifecycle, and the configuration surface are unchanged. The bounded diagnostic fields remain useful for attributing any residual watchdog.

Regression coverage deliberately blocks a capacity-one EVENT lane behind 1,200 LocalRelay events and requires all transient permits to release within three seconds. A second scenario opens 25 immediate empty queries and requires every pre-registered permit to release within the same bounded deadline. The architecture document records the implemented control/data separation and registration ordering.

Pre-deployment validation of final tip `04b93c9`:

- all 648 library tests pass;
- both focused lifecycle regressions pass;
- standalone `startup_historic_sync_stays_within_relay_req_concurrency_limit` passes with zero rejected REQs;
- changed-source rustfmt and `git diff --check` pass.

Production verification of exact tip `04b93c9a1df9d28fc6af96b131d0ebc5646552ca`:

- remotely built and activated on gitnostr.com at 2026-08-07 15:37:54 UTC;
- immediately reproduced a larger comparable workload: relay.ngit.dev had 994 missing IDs/4 chunks and 38,518 IDs/129 chunks;
- through 15:45:27 (7m33s, more than three complete 120-second recurrence windows), there were zero watchdogs with `sdk_subscription_active=false`, including zero at relay.ngit.dev, ngit.danconwaydev.com, and git.shakespeare.diy;
- four watchdogs did fire correctly for genuinely active relay-side subscriptions: three historic pages at relay.cyberguy.fyi and one at nostr.sebastix.dev, all `sdk_subscription_active=true`;
- the target relay continued useful progress: its generic announcement batch completed and later REQ/EOSE batches had confirmed 86 state-only repositories by 15:40:45 despite the pre-existing startup rate-limit CLOSED burst;
- service remained active with zero restarts, zero panics, zero budget-exhaustion/concurrency-limit messages, and 1.24 GiB memory at the final sample.

This exact tip is recommended for merge. Production distinguishes the repaired local-accounting defect from the watchdog's intended recovery of peers that genuinely leave subscriptions open.
 2026-08-07 16:54:56 +01:00
DanConwayDev 32813b92f1 Merge #a36a8dc2: chore(sync): classify transient watchdog recovery 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs2x65dcgs4h5flnnjyj945khwrq6dt3tlnet5t9tut4wtfyc2ej5ggjhke8

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Production at `57bc76ea` forced exactly three `relay.ngit.dev` transient subscriptions through the 120-second watchdog every two minutes: 30 CLOSEs in the first 20 minutes. Subscription accounting remained within bounds, but the held-permit state and warning retained only random subscription IDs, so the stalled lifecycle could not be distinguished without guessing.

Require every auto-close REQ caller to provide one of six bounded request classes, retain that class with its per-session permit, and expose it in watchdog logs and a class/outcome Prometheus counter. Pagination distinguishes ordinary continuation from NIP-11 hint verification using its existing state; exact-ID purgatory polling remains correctly outside these labels because it uses the separately timed `fetch_events` path.

This is observability only: timeout, retry, pagination, request admission, filter contents, and slot-release behaviour are unchanged. Relay URLs and subscription IDs are deliberately excluded from metric labels, and no configuration surface is added.

Validated by the 646-test library suite, including fixed-label uniqueness and hint-verification classification tests, plus rustfmt and `git diff --check`.

Production verification on gitnostr.com deployed from exact PR commit `58ef8682f3da6447f8fd52560f6148fe3ba4e6a9` from 2026-08-07 13:49:09 UTC:

- `ngit_build_info` reports commit `58ef8682f3da6447f8fd52560f6148fe3ba4e6a9`.
- The service remained active with zero restarts and no error-level journal entries.
- Nine watchdog recoveries across three complete 120-second cycles were all labelled `request_class="negentropy_hydration"`, all targeted `wss://relay.ngit.dev`, and all successfully sent CLOSE; the metric recorded the same class/outcome.
- Startup negentropy advertised 995 and 40,071 missing event IDs, split into 4 and 134 hydration chunks. The three-at-a-time cadence is therefore queued hydration work advancing as three stalled relay-side subscriptions are closed, not unbounded retry or pagination traffic.

The change has produced the intended low-cardinality diagnosis without changing request behaviour. It is ready to merge. The remaining missing-EOSE behaviour should be reassessed after the still-old `relay.ngit.dev` is upgraded, rather than speculatively changing the client in this PR.
 2026-08-07 15:05:03 +01:00
DanConwayDev 64f33dcda7 Merge #843ee3ad: fix(build): publish the deployed source revision 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsgg0hr45szmdeum8mahjkvzvfvmfndpgtayxtl9jhqsfajtyhys6gc8japx

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Production currently exports ngit_build_info{commit="unknown"}, so every stabilisation cycle must reconstruct the running revision from the nixos-fi1 pin, Nix store path, and activation time. Two independent gaps cause this: metrics reads GIT_HASH while build.rs emits GIT_COMMIT_SHORT, and filtered Nix source archives intentionally contain no .git directory.

This change gives all build surfaces one trustworthy revision source. Nix package and NixOS-module outputs inject the locked flake self.rev; ordinary Cargo builds retain a Git HEAD fallback. Metrics publish the full 40-character revision, while NIP-11 and the landing page retain an eight-character display form. Direct imports of nix/module.nix remain supported and report unknown unless a revision is explicitly supplied.

Validation: 21 metrics-related library tests passed; nix flake check --no-build passed for packages and both exported modules; clean-flake evaluation equalled the committed revision; nix build .#ngit-grasp passed; and the resulting binary contains both the exact full revision and expected short display revision. Production verification will confirm the metric, NIP-11 document, and landing page agree with this PR tip.
 2026-08-07 14:02:12 +01:00
DanConwayDev 87d7732910 Merge #86376f5d: fix(relay): preserve bursty client sessions 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsgvdm0t5fnvz2q5zf2p6myctssejzp8t6h5cq9fz84qrkv8u784sc3dgkww

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

## Production problem

rust-nostr 0.45 introduced a connection-wide 300-frame-per-minute bucket and closes the WebSocket when it is exhausted. gitnostr.com recorded 5,020 `too many client messages` disconnects between 2026-08-06 10:00 and 2026-08-07 07:55 UTC. This predates and is independent of the subscription-budget ledger.

Correlating relay timestamps with Caddy's real-client access log showed two populations:

- a rapid source sent more than 300 frames and roughly 1.2 MiB in about 1.5 seconds, which should remain bounded;
- legitimate browser sessions from `gitworkshop.dev`, `gittr.space`, `armada.buzz`, and a localhost development origin also crossed the ceiling.

## Approach

Raise only the fixed connection-wide allowance to 6,000 messages/minute (100/second). All activity for a relay is multiplexed through one gitworkshop WebSocket: independent loaders, pagination, reconnects, publishes, AUTH, REQs, and CLOSEs. Batching occurs within individual loaders rather than across the whole connection.

An initial 1,200/minute production candidate was deliberately rejected after it disconnected a legitimate localhost browser session sustaining roughly 44–45 frames/second. The final allowance provides about 2.2x sustained headroom over that observed workload, plus the token bucket's initial burst capacity, while retaining a finite catch-all for malformed or non-operation frames.

The tighter independent quotas remain unchanged: 60 EVENT writes, 120 queries, and 30 authentication events per minute. No new configuration option is added because this non-standard limit cannot be discovered by clients and ngit sync cannot adapt to it.

## Correctness evidence

- A 1,201-frame burst keeps the WebSocket usable and completes a subsequent REQ/EOSE exchange.
- A rapid 6,001-frame burst is still disconnected.
- `nix develop -c cargo test --test relay_message_rate`: 46 passed.
- `nix develop -c cargo test --lib`: 643 passed on the initial candidate; the final change only raises the tested constant and the final targeted suite passed 46/46.
- `nix build .#ngit-grasp`: passed on the initial candidate; deployment built the exact final package.

## Production validation

The exact final tip `a6b5e3a9f66f8bb6e4cc3f23ef62c9be1a9b37f2` was deployed as `/nix/store/bp02cmfviwmf03px52xbjqs8rk4a01ih-ngit-grasp-2.0.0` at 2026-08-07 09:43:58 UTC.

Through 09:57:33 UTC it admitted 874 WebSocket connections with:

- zero `too many client messages` closures;
- zero concurrent-REQ rejections, connection-permit rejections, or subscription-budget failures;
- zero restarts and zero panics;
- 18 expected transient-REQ watchdog CLOSE recoveries;
- approximately 995.9 MB peak memory.

This exact-tip production evidence, together with the failed lower-limit experiment and regression scenarios, supports merge.

## Excluded scope

Per-IP admission fairness and changing rust-nostr's upstream default are separate concerns. A 6,000/minute upstream default is worth proposing as a pragmatic compatibility floor, but this PR does not alter upstream policy.
 2026-08-07 10:59:28 +01:00
DanConwayDev f15b49c39e Merge #f1ed9f57: fix(sync): unify per-connection subscription budgets 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs0rmvl2u9pmpfct2zaxqc2p84n0jcsnxjrkhdnjxp93u4a00a0l0gstttec

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Root cause and scope: the standalone sixth-REQ failure was partly a test-proxy accounting defect—the relay had sent CLOSED, but the proxy retained dead subscription IDs. Commit `8d1aaeb` fixes that harness without making a production claim. Commit `5653aaa` then replaces separate NEG/REQ limits with one per-session ledger derived from NIP-11 `max_subscriptions`, falling back to 20 and reserving two slots. It accounts persistent live groups, NEG, historic/fallback/pagination/verification/retry REQs, and exact-ID purgatory polls. Live L1+L2+L3 coverage is transactionally replaced/restored; stale-session waiters are generation-rejected; live CLOSED triggers complete-set recovery. Multi-connection sharding remains out of scope.

Timeout correction: production disproved the initial whole-session teardown approach. It caused avoidable reconnect and actor-backlog delays when relay.ngit.dev omitted EOSE/CLOSED. Commit `15369ca` instead sends CLOSE for only the timed-out subscription and releases only its generation-scoped slot after the SDK accepts CLOSE; a failed send retains the slot until ordinary connection teardown. This matches the accepted accounting boundary and removes 49 net lines from the ledger tip.

Acceptance-fixture correction: an initial fifth whole-file run reported one rejection at a recorded peak of five, but the proxy did not retain the rejected ID. Review found no subscription-opening bypass and found that the proxy checked capacity before recognizing same-ID REQ replacement, while also treating EOSE as closure. Commit `2bdc11d` makes the fixture follow NIP-01: replacement does not add occupancy, EOSE retains it, and only CLOSE/CLOSED releases it; future rejections log the exact frame and active IDs. This is test-only and does not claim the uninstrumented rejection was conclusively a replacement.

Review follow-up: `add9c66` corrects the architecture document to describe the implemented per-subscription watchdog CLOSE rather than the abandoned session teardown. `f5081a3` re-derives sync work after deferred live consolidation, preventing quiet relays from stalling until an unrelated trigger. `cb0af7b` admits transient work without waiting while holding only its class or ledger permit, and explicitly rolls back earlier live permits on a session-generation error. CLOSED-burst coalescing remains excluded: generation-only deduplication could suppress a legitimate CLOSED from replacement coverage and needs a separate lifecycle design.

Validation: `cargo test --lib` passed 638/638; the standalone req-concurrency scenario passed under the stricter relay-lifetime proxy; `nix build .#ngit-grasp` passed. After the review fixes, five consecutive full `--test sync` runs passed in five attempts, including five req-concurrency passes and zero recorded proxy rejections. The final `e53c576` commit adds only the unreleased changelog entry, so it does not invalidate runtime evidence for parent `cb0af7b`.

Production evidence (gitnostr.com, reviewed runtime `cb0af7b`, deployed 2026-08-06 23:21:21 UTC from nixos-fi1 `e0eb753`): across the exact 3h00m06s window ending 2026-08-07 02:21:27 UTC, 82 relay sessions configured ledgers and 125 stalled transient requests were recovered with individual CLOSEs. There were zero subscription-limit rejections, zero budget-exhaustion warnings, zero relay.ngit.dev disconnects, zero panics, and zero service restarts. Memory ended at 1.24 GiB. Ordinary sync and purgatory maintenance continued throughout. Together with the earlier 32-minute `15369ca` soak, this demonstrates bounded slot recovery without the reconnect train caused by the abandoned whole-session candidates.

Reviewability: the runtime/test mechanism remains 1,491 additions/229 deletions across four files relative to master, plus a six-line changelog entry. In the largest file, `relay_connection.rs`, runtime ledger/lifecycle logic and focused invariant tests are deliberately colocated. The size reflects unifying every subscription consumer plus transactional live coverage and stale-session safety; the final timeout and admission corrections simplify the operational failure modes. A file-only extraction would move lines without reducing the correctness surface, so no cosmetic refactor is included.

Part of `nostr:nevent1qqspxxvcaqj96zt8sdj520h9nl4rtq3873m6za6keg2gm2xn6dgp55cpz3mhxue69uhhyetvv9ujumn8d96zuer9wc85zw09`.
 2026-08-07 06:44:46 +01:00
DanConwayDev 6cccebb93b Merge #6b9488f1: fix(sync): adapt pagination to relay page size 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsxh9yg79mm2wkr7qu50s8hucq6wufr2aandvr2yvy3epg6zhxeleccjw758

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Production risk: historic REQ filters omit limit, while live Ditto-shaped relays can return only 100 events per filter and advertise only a much larger max_limit. The static threshold of 200 therefore treats a full 100-event page as exhausted and silently loses older history.

Commit 1 counts every raw matching delivery before deduplication, rejected-event suppression, purgatory routing, or write policy, and derives the cursor from that same raw stream. Trace verification confirmed repeat deliveries reach ProcessResult::Duplicate.

Commit 2 learns the largest raw page per relay connection session, combines it only with NIP-11 default_limit, and uses max(90, floor(0.9 * estimated_cap)). NIP-11 is fetched per session; max_limit never raises omitted-limit thresholds. A suspiciously short hinted page gets one cursor verification page, and any unseen event discards the hint for that session.

Validation: cargo test --lib passed 623/623; three adaptive integration cases passed (Ditto-shaped 100/no-default/1000-max with 320 issues, honest default_limit, and lying-high default_limit); cargo check --workspace --all-targets passed.

Known caveat: the required standalone req-concurrency test still fails on its documented pre-existing transient proxy limit defect and oldest-issue assertion. Under this diff the same too many concurrent REQs signature was noisier (five notices across two relay processes versus the documented baseline single rejection) because the lower learned threshold exposes more pagination pages when NIP-11 is unavailable. This PR does not mask or expand into that separate concurrency defect.
 2026-08-06 13:25:52 +01:00
DanConwayDev e53d2f30ec Merge #2960a755: fix(sync): batch purgatory dependency polls by relay 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqszjc98249z0hz7zpf3gqmqyhxru0leqep5dvp77x8zur8z2msxmeg6adac7

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Production on gitnostr.com recorded four unresolved repositories producing 193 exact-ID relay fetch log entries and 482 requested event copies during the 2026-08-06 06:26:59-06:42:35 UTC window. The 30-second recovery cadence is intentional so authorization and state events are found promptly when they appear, but each purgatory repository currently spawns its own request to every hinted relay and competes with ordinary synchronization for per-connection query-rate budget.

This supersedes the closed standing-subscription proposal nostr:nevent1qqsfp9n5hq3q6x4hkr8pcv62ut5qjdvthlqgzpjlqmmx2cgd97h6zygpz3mhxue69uhhyetvv9ujumn8d96zuer9wc55cclu. That approach avoided repeated query creation but consumed scarce concurrent subscription slots and required substantially more lifecycle machinery.

Build one maintenance-round plan keyed by canonical connected relay, union all due cold dependency IDs from every selected purgatory announcement, and issue bounded 100-ID queries per relay. Preserve the existing 30-second cadence, parallel relay I/O, exact-ID retry accounting, authorization ordering, and early exit when dependencies resolve. Include repository_count and requested_count in the existing fetch log so production batching is directly observable.

The 100-ID chunk assumes an exact-ID filter can return at most one match per requested ID and stays within the lowest audited default result cap. This deliberately does not add live subscriptions, alter purgatory retention, add exponential backoff, or change general historic-sync requests.

Validated with cargo check --workspace --all-targets, all nine maintainer-reprocessing integration scenarios, and 47 sync unit tests. The new scenario indexes two cold invitations for separate repositories behind one relay and observes one recovery query with repository_count=2 and requested_count=2.

Production verification on exact PR commit 6dfe0a7e25: during the 2026-08-06 10:10:54-10:20:54 UTC soak, 31 batched relay queries represented 93 repository-relay polls, a 66.7% reduction against the direct unbatched counterfactual. Queries requested 206 IDs and recovered 4 events; real batches reached five repositories and eleven IDs. There were zero batched-refetch failures, zero panics, and zero service restarts.
 2026-08-06 11:23:02 +01:00
DanConwayDev 0a73d68da4 Merge #3509c127: fix(sync): suppress naughty relay connection attempts 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsr2zwpylk5cw0654n7hhrqgdd0jhc4pyyr8fmwf0lccwmks4f2fdqumxzdg

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Persistent DNS, TLS, and protocol failures were recorded in the relay naughty list, but reconnect eligibility and every scheduling path ignored that state. Production consequently kept announcing and launching retries after normal health backoff.

Consult the existing tracker both when deciding reconnect eligibility and at the shared connection-scheduling boundary before lifecycle mutation or worker-capacity reservation. This preserves the required first dial, suppresses all later scheduling paths while the entry is live, and restores eligibility through the existing 12-hour expiry behavior.

The scenario regression uses a deterministic broken WebSocket endpoint. It proves that the classification dial is the only connection accepted across multiple reconnect ticks and that no reconnect intent is emitted after the endpoint becomes naughty. This deliberately leaves error classification, expiry duration, Git-domain throttling, and transient relay backoff unchanged.

## Validation

- Final PR commit: `ff388ce184430869c0e5e6fecadb4bf96aec951c`.
- Full workspace suite passed: 617 library tests plus every integration and doc-test target.
- Final sync target passed: 79 passed, 1 pre-existing ignored.
- Focused health tests passed: 16 passed.
- `git diff --check` passed.

## Production verification

Deployed unmerged PR commit `ff388ce184430869c0e5e6fecadb4bf96aec951c` to gitnostr.com via nixos-fi1 commit `f030cfc987b128356ff856c45d45984f040c1ffc`. The exact flake pin, live ExecStart path `/nix/store/4clqvqy04lmz1q3fq6964r020x985kaf-ngit-grasp-2.0.0/bin/ngit-grasp`, and Nix registration time 2026-08-05 14:55:05 UTC independently identify the deployed revision. Activation began at 14:55:20 UTC.

Observed production window: 2026-08-05 14:55:20–15:10:37 UTC (15m17s). Fourteen relays organically entered the naughty list. Every one had zero subsequent reconnect-intent lines, zero repeated event-directed target rejections, and zero repeated naughty-failure lines. During the same window, 110 legitimate reconnect attempts continued for non-naughty degraded relays, demonstrating that the reconnect scheduler remained active rather than globally stalled.

No regression signal appeared during the soak: `NRestarts=0`; no error-priority journal entries, panics, scheduler failures, or stale-attempt results; memory peaked at approximately 1.26 GB.

This production result, together with the deterministic scenario regression and green test suite, is sufficient for merge.

Tracked under nostr:nevent1qqspxxvcaqj96zt8sdj520h9nl4rtq3873m6za6keg2gm2xn6dgp55cpz3mhxue69uhhyetvv9ujumn8d96zuer9wc85zw09
 2026-08-05 16:16:55 +01:00
DanConwayDev 1c586d269c Merge #cc893bf3: feat(grasp-audit): emit full audit JSON reports 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsvezfm7w48p7z2sk05xs48pnjx7fddys0apezs9yvg7zrp9l7f9rsmathdx

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Expose the merged stable audit result model through audit --json using one schema-versioned run envelope. Completed audits include tool/run metadata, pinned specification revisions, aggregate counts, and per-test outcomes. Parse, setup, and runtime failures emit the same envelope with an error outcome.

Human output and audit ordering remain intact. Exit status is 0 for pass, 1 for completed test failures, and 2 for invocation or runtime errors.
 2026-08-05 14:42:35 +01:00
DanConwayDev 9ca59b428a Merge #d10d26ba: fix(sync): retain NIP-77 for small hydration residuals 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsdzrfxh2zzfnhr0dmhdkf3y9gjsc53v0yj9vfwt4vl3xd9rqfyn0sg5j4k7

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Retain NIP-77 when an otherwise successful exact-ID hydration leaves a small residual. Semantic REQ+EOSE fallback now requires a material batch (at least 20 advertised IDs), an initial delivery rate of at most 10%, and a zero-progress residual retry. Other residuals keep NIP-77 enabled and enter bounded missing-event recovery.

## Why

Production showed healthy or mostly healthy relays being marked incompatible solely because a small residual retry returned zero. `git.nostrhub.io` delivered 1,420/1,818 before withholding an expired residue; `nos.lol` delivered 115/118 and all six sampled residual IDs later became available by both NIP-77 and ordinary exact-ID REQ. Neither warrants semantic fallback. The threshold preserves the original compatibility path for the distinct case where roughly 90% of a material advertised inventory cannot be hydrated by ID but can be recovered semantically.

## Validation

- Unit boundaries cover the four production ratios plus the 20-event/10% policy edges.
- The censoring-proxy scenario exercises a partial first pass, zero-progress retry, degraded status, continued live sync, and bounded recovery; it proves the relay retains NIP-77 and does not enter semantic fallback.
- Full `nix develop -c cargo test` green: 616 library tests; all integration targets; sync 78 passed / 1 pre-existing ignored; doc tests green.

## Production verification

Deployed unmerged PR commit `a7da97e1054a5069fa67bd764389885de569fa7a` to gitnostr.com via nixos-fi1 commit `5845011`. The exact flake pin, systemd ExecStart path `/nix/store/jg5yi9pfq94xl52sgqk4das5cs5bnl8i-ngit-grasp-2.0.0/bin/ngit-grasp`, and Nix registration time 2026-08-05 12:59:38 UTC independently identify the deployed revision. Activation began at 13:01:19 UTC.

Observed production window: 13:01:19–13:16:21 UTC (15m02s). Organic startup sync exercised the small-residual classification: `relay.ngit.dev` delivered 976/978 advertised events, and the deployed code retained NIP-77 and created one exact-ID retry for the two residual IDs. No material-hydration incompatibility was recorded and no semantic REQ+EOSE switch occurred. The relay had not returned the retry EOSE by the end of the window, so terminal zero-progress recovery is supported by the end-to-end scenario rather than claimed as organic production evidence.

No regression signal during the window: `NRestarts=0`, 52 events synced, service remained active, and memory ended at approximately 1.26 GB, within the recent production envelope.

The investigation also confirmed two separate facts outside this PR: all six sampled `nos.lol` residual IDs subsequently became available through both NIP-77 and ordinary REQ; and the purgatory recomputation path can target gitnostr.com itself, which remains a separate self-exclusion fix.

This production exercise of the deployed residual classification, combined with the end-to-end terminal recovery scenario and full green suite, is sufficient for merge.
 2026-08-05 14:21:40 +01:00
DanConwayDev a2215b9b8c Merge #b69eac8d: feat(grasp-audit): harden discovered-server probes 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqstd84v3473q4ts5jjtcmlh7az34pgxrvv49g89l9vyumjyxk4n3lqwpmgfv

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Add opt-in network hardening for scheduled read-only probes of untrusted GRASP endpoints. Validate public wss/https destinations, pin WebSocket and HTTP connections to pre-resolved addresses, refuse redirects and cross-origin clone URLs, and bound DNS resolution and response bodies.

Hardening remains disabled by default so ws://localhost and other local development targets continue to work. Hardened probes cannot be combined with --create-repo.
 2026-08-05 13:03:41 +01:00
DanConwayDev 96190bef52 Merge #c9bce948: fix(sync): bound purgatory fetches for missing remote… 
fix(sync): bound purgatory fetches for missing remote tips

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsvn08ffq3yerehv7cw56vs99sguq0hp039q7kjlx3wm2jhmncdtkq5chje3

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Fetch needed OIDs advertised as remote ref tips in one safe batch, isolate residual missing-object requests, and memoize confirmed per-URL misses while the sorted advertisement fingerprint remains unchanged. This prevents repeated purgatory passes from re-requesting hundreds of known-missing tips.

## Validation

- Scenario regression fails on `master` with one failed upload-pack request carrying all ten wants.
- On this branch, the first pass batch-fetches both advertised tips, isolates each unavailable tip to a single-want request, and leaves the unsatisfied state in purgatory.
- A fresh replacement state event deterministically triggers a second pass; its unchanged advertisement adds zero failed upload-pack exchanges because the eight confirmed misses are memoized.
- Helper tests cover stable order-independent advertisement fingerprints, invalidation on ref changes, 30-minute lazy expiry, and the 1,024-URL cap.
- Full `nix develop -c cargo test` suite green: 615 library tests, all integration targets (sync: 78 passed / 1 ignored), and doc tests.

## Production verification

Deployed unmerged PR commit `479d3b451f65519b181b6c4fd065873b718c5359` to gitnostr.com via nixos-fi1 commit `a491bc7`. The pin, systemd ExecStart store path `/nix/store/mqsxa5v38nfgmy5880g808xyrblyfq3g-ngit-grasp-2.0.0`, and its Nix registration time (2026-08-05 10:46:27 UTC) independently confirm the deployed revision; activation began at 10:47:40 UTC.

Bounded soak: 2026-08-05 10:47:40–11:17:13 UTC (29m33s). The deployed fetch pipeline completed two organic advertised-tip passes in-window, and a third at 11:18:25: all three had `residual_attempted=0`, `residual_missing=0`, and fetched every needed advertised tip. Counters at 11:18 showed 3 passes / 3 advertised tips / 3 fetched. The production workload did not enter the remote-missing branch; that branch is covered by the end-to-end repeated-pass scenario above.

No regression signal during the soak: `NRestarts=0`, watchdog reclaims 0, 477 events synced, 105 relays tracked / 66 connected at the end of the window, and memory 1.34 GB (within the prior ~1.4 GB baseline). Serving-side `not our ref` errors continued from an external third-party crawler as expected and are not generated by gitnostr.com's outbound fetch pipeline.

This split of evidence—real production exercise of the exact deployed pipeline plus end-to-end scenario exercise of the memoized repeated-miss path—is sufficient for merge without manufacturing public production events.
 2026-08-05 12:26:54 +01:00
DanConwayDev af4e40db36 Merge #d46c9733: feat(grasp-audit): define stable audit results 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsdgmyhx0qyy0423kure57ppcu6rz4jhkefrsnztphn037pypa2wfqk8gzvp

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Add serializable per-spec and per-test report data without wiring a new CLI output mode. Pin suite metadata to the audited GRASP revisions, expose explicit outcomes and millisecond durations, and bound diagnostic text safely.

Use typed probe check names and shared GRASP-06 test identifiers so emitted IDs come from one source while preserving existing human and probe JSON output.
 2026-08-05 11:13:50 +01:00
DanConwayDev c0e7af7055 Merge #cd4d6f74: feat(grasp-audit): support explicit audit identities 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsv6nt0ws0mrn9utmr9atvgvcpp0qq4nk66mqujwlsvz83nd7trl4cmtj3nc

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Use the same identity inputs for probes and full audits: inline --nsec, --nsec-file, or GRASP_AUDIT_NSEC. Parse nsec and hex keys, retain generated-key fallback, and share the environment name with Git subprocess secret scrubbing.
 2026-08-05 11:10:17 +01:00
DanConwayDev 958aef2af4 Merge #7d13802c: fix(grasp-audit): keep signing secrets out of git 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs86yuq93gmfaa9323eymlnpjt7lqlc50nypkkms4yf30cgk5u6ljc7p9luw

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Strip both the audit identity and relay-owner secret variables at the existing hermetic git subprocess boundary. This prevents git hooks, credential helpers, remote helpers, and their descendants from inheriting signing keys.
 2026-08-05 10:46:34 +01:00
DanConwayDev 320f3fdae1 Merge #a8536225: fix(sync): bound concurrent negentropy rounds per rela… 
fix(sync): bound concurrent negentropy rounds per relay connection

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs2s5mzy4kddgl27xq9s02cf5evatgqkm63x9xqyeapfdrahu9wtfcdudjkr

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Startup historic sync opened one NIP-77 negentropy diff per filter with no bound: an unbounded join_all launched every diff in a batch simultaneously, so a large watched set (146 filters on the bootstrap relay) burst past relay per-connection subscription budgets. On strfry-family relays negentropy views share maxSubsPerConnection with ordinary subscriptions; nos.lol (budget 20) answered a gitnostr.com startup with 34 "too many concurrent NEG requests" rejections, 61 per-filter timeouts, and 21 failed fallback-subscription creations in two minutes (2026-08-04, observed during the cycle-2 soak of ecb6c8b6). The cycle-2 cooldown contains the damage; this PR removes the cause.

A per-connection semaphore (4 permits, shared across clones so concurrent batches to one relay share the bound) gates negentropy_sync_diff; the permit is held for the whole round including the timeout. Queued rounds re-check NIP-77 availability after acquiring and bail to the per-batch REQ+EOSE fallback without recording a failure, so rounds that waited through the start of a cooldown cannot escalate it. Four permits keeps the tightest commonly observed budget (20, shared with live subscriptions) mostly free; the constraint research and budget model are documented in docs/explanation/sync-scaling-constraints.md, added here.

Reproduction: a new scenario test drives two real relays end to end through a WebSocket proxy that enforces the strfry limit of 4 and delays NEG responses so rounds provably overlap; a 150-root-event repository produces a six-round historic batch. Unfixed: 3 rounds rejected (opened 6, peak pinned at the limit) 342200224 the production signature. Fixed: zero rejections, peak <= 4 with overlap retained, sync completes. Full test suite passes.

Deliberately excluded: bounding REQ+EOSE fallback subscriptions (permit lifetimes span EOSE handling; deferred to the budget-ledger work), byte-budgeted filter chunking (proposed as the next cycle), and any configuration surface for the bound.

Part of the sync-stabilisation tracker: nostr:nevent1qqspxxvcaqj96zt8sdj520h9nl4rtq3873m6za6keg2gm2xn6dgp55cpz3mhxue69uhhyetvv9ujumn8d96zuer9wc85zw09

## Production verification (gitnostr.com)

Deployed before merge, per the stabilisation cycle.

Commit `a0520c3d` (concurrency bound) deployed 2026-08-04 21:29:37 UTC; commit `dbc6f7e2` (byte-budgeted chunks, added after the bound verified) deployed 21:58:59 UTC. Deployed revisions confirmed via the nixos-fi1 pin plus ExecStart store-path registration times.

- The startup failure signature is gone across both startups: 0 "too many concurrent NEG requests" rejections (the previous revision drew 34 within two minutes of every startup), 0 rate-limit NOTICEs attributable to negentropy, and 0 failed REQ+EOSE fallback-subscription creations (previously 21).
- Byte-budgeted chunking reduced the largest startup batch from 153 filters to 61, and startup completed with 0 failed live, historic, or fallback subscription creations.
- The bound composes correctly with the cycle-2 cooldown: queued rounds skipped into per-batch REQ+EOSE fallback when a cooldown started while they waited (observed on wss://relay.damus.io: 1 transient failure, 20 queued rounds skipped without escalating the backoff), and slow-relay timeouts escalated 60s -> 300s as designed.
- Steady state 22:15-22:33 UTC: zero rejections, NOTICEs, or cooldowns of any kind; 430 events synced since restart; 67/104 relays connected; no service restarts; stable memory (~1.2 GB).
- Remaining rejections were confined to one startup minute and are pre-existing REQ-path signatures outside this PR's scope, at or below historic rates ("too many concurrent REQs" from nos.lol: 6, versus 7739 over the prior three days; "too many values in ids" from relay.cyberguy.fyi: 19, from the unchanged 300-ID exact-ID fetch chunks). Both belong to the deferred subscription-ledger work documented in docs/explanation/sync-scaling-constraints.md.

Commit `680eaf2f` (maximised 32 KB / 96 KB byte budgets) deployed 2026-08-05 06:41:18 UTC; revision confirmed via the nixos-fi1 pin plus the ExecStart store-path registration time (2026-08-05 06:40 UTC).

- "too many concurrent NEG requests" stayed at 0 across startup and steady state.
- The largest startup batch halved again as predicted: 61 filters (16 KB chunks) -> 34 (32 KB chunks), with 0 failed live or historic subscription creations.
- No new relay pushback from the larger filters and REQs: startup-window signature counts are at or below the previous revision's startup (rate-limit NOTICEs 25 -> 16, transient negentropy failures 12 -> 13, ERROR lines 219 -> 175, fallback-subscription failures 23 -> 23). damus "blocked: too many query results" (2 occurrences) appeared at the same rate on the 16 KB revision overnight, is absorbed by the REQ+EOSE fallback, and is the per-query result cap recorded in the constraints doc as the eventual bound for the ledger work.
- Steady state to 06:58 UTC: 385 events synced since restart, 67 relays connected, no service restarts, stable memory (~1.25 GB).
- The pre-existing startup burst of "too many concurrent REQs" from nos.lol (6, all in one second) persists as expected; it is the target of the next commit on this PR (bounding historic/fallback REQ+EOSE concurrency).

Commit `e942199a` (per-connection transient REQ concurrency bound) deployed 2026-08-05 07:56:36 UTC; revision confirmed via the nixos-fi1 pin plus the ExecStart store-path registration time (2026-08-05 07:55:21 UTC).

- The target signature is gone: "too many concurrent REQs" from nos.lol went from 6 per startup burst to 0 across startup and a 45-minute soak (07:56-08:41 UTC).
- "too many concurrent NEG requests" stayed at 0, and the 30 s permit watchdog never fired (0 reclaims), so every permit was released through the normal EOSE/CLOSED path.
- No sign of permit starvation from the new bound: startup completed fully (67 relays connected, matching the previous revision), 451 events synced by 08:41 UTC (previous startup window: 385), the largest startup batch stayed at 34 filters, and steady-state sync activity (reconnection cycling, negentropy rounds) is normal.
- Remaining signature counts are at or below the previous revision's window: rate-limit NOTICEs 16 -> 10, transient negentropy failures 13 -> 12, fallback-subscription failures 23 -> 22, non-serving ERROR lines ~25 -> 28. damus "blocked: too many query results" (2) persists at its known rate and is absorbed by the REQ+EOSE fallback; relay.cyberguy.fyi "too many values in ids" did not appear in this window.
- Serving-side ERROR volume in the window is dominated by pre-existing signatures unrelated to the sync path this PR touches: an external client fetching objects one at a time ("git upload-pack: not our ref", 4,521 occurrences at a steady 150-270/min, signature present in the journal since 2026-07-29) and the known inbound-websocket "Connection reset without closing handshake" noise (435).
- Steady state to 08:41 UTC: no service restarts, stable memory (~1.4 GB).
 2026-08-05 09:46:52 +01:00
DanConwayDev 633224bc81 Merge #c1b384df: fix(sync): use bounded cooldown for transient negentro… 
fix(sync): use bounded cooldown for transient negentropy failures

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsvrvuyml2nqtsnpfjz0vrmrfnw52apdvggskpwpr82pqkt6aaq0dqy9k7hy

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

One transient negentropy diff error permanently disabled NIP-77 for the relay connection. On gitnostr.com (2026-08-04, PR commit 256a9912) a single client-side "channel lagged by 3" during the startup burst marked the bootstrap relay non-NIP-77 within seconds of startup; 20 relays were marked in the first minutes and reconciliation collapsed from 739 runs to 24 after 14:30 UTC. 93% of observed failures (timeout, channel lagged, blocked/rate-limit) carry no information about NIP-77 support, and the affected relays advertise NIP-77 in their NIP-11 documents.

This PR classifies diff failures: only explicit unsupported signals ("negentropy not supported", "server does not support our negentropy protocol version") permanently mark the connection; everything else applies an escalating per-relay cooldown (60s/5m/30m/2h) that a successful diff resets. Failures arriving while a cooldown is active come from diffs already in flight when it started and do not escalate the backoff. Per-batch REQ+EOSE fallback is unchanged, so sync progress never depends on the classification. The concurrent-abort error no longer fabricates a NOTICE-based detection.

## Production verification (gitnostr.com)

Deployed from this PR's commit `ecb6c8b68caa77ebb8b894145101af55124006c5` at 2026-08-04 19:24:49 UTC, before merge. Soak window 19:24-19:51 UTC, covering service startup — the window in which the failure always fired:

- The misclassification signature ("does not support NIP-77 (detected via NOTICE)") occurred 0 times; on the previous revision it fired 98 times on the bootstrap relay within seconds of startup.
- The bootstrap relay wss://relay.ngit.dev was neither marked unsupported nor placed in cooldown; 845 negentropy reconciliations completed, continuing past startup instead of collapsing fleet-wide to REQ+EOSE.
- Escalating cooldown observed end-to-end in production: wss://wheat.happytavern.co timed out at 19:26:22 → 60s cooldown → retried after expiry → timed out again at 19:31:39 → escalated to 300s.
- Genuine unsupported detection still works: wss://relay.primal.net sent NOTICE "bad msg: negentropy disabled" and was permanently marked.
- Startup NEG bursts absorbed: 13 transient-cooldown warnings total across 10 relays (previously hundreds of cascading failures); nos.lol's 34 concurrent NEG-request rejections produced a single 60s cooldown.
- No new WARN/ERROR signatures versus the previous revision.

Remaining observation for a future fix: the startup historic sync still opens enough concurrent NEG requests to draw rate-limit NOTICEs ("ERROR: too many concurrent NEG requests" from nos.lol, 34 at startup) and per-filter timeouts; the cooldown now contains the damage, but bounding sync concurrency at startup would remove the cause.

Part of the sync-stabilisation tracker: nostr:nevent1qqspxxvcaqj96zt8sdj520h9nl4rtq3873m6za6keg2gm2xn6dgp55cpz3mhxue69uhhyetvv9ujumn8d96zuer9wc85zw09
 2026-08-04 20:53:22 +01:00
DanConwayDev be71c2a580 Merge #e2dfa0e5: fix(sync): stop refetching structurally malformed even… 
fix(sync): stop refetching structurally malformed events every historic pass

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsw9haquk7w32zavgeyqevzxn2uwkmj8z0ufe8wsadn8tt0mxq245grk39n0

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Production evidence (2026-08-04, deployed b3ca0159): 494 "Event missing 'd' tag, cannot track in rejected index" warnings across 41 distinct event IDs in ~3h10m — an ~11-minute re-download/revalidate cycle per event. The rejected-events index is keyed by pubkey+identifier, so announcements and state events without a 'd' tag could never be tracked and were fetched and rejected again on every historic pass.

This adds an ID-keyed "unrecoverable" store to `RejectedEventsIndex` for rejected events that have no pubkey+identifier key and can never become valid. No repository identifier is invented; entries hold only the event kind and rejection time. Both existing skip paths consult it unchanged, since each already works on exact event IDs: `contains()` before live/REQ processing and `get_all_event_ids()` when excluding IDs from negentropy refetch.

Entries expire with the cold-index bound (default 7 days), are swept by the existing daily cleanup task, and persist across restarts in `rejected-events-cache.json` with a serde default so older cache files still restore. Direct live submissions never reach this sync-only path; their rejections remain logged by the write policy.

Deliberately excluded: no Prometheus gauge for the new store, and no recovery machinery — unrecoverable entries are terminal until expiry.

Tracked in nostr:nevent1qqsy928qmqsfk32xg8fjltxrqfp8xqg4pdtnxzyxh2xs89jlvfc9urgpz3mhxue69uhhyetvv9ujumn8d96zuer9wc0ttwts

## Testing and production verification

**Unit tests** (first commit, `256a9912`): ID tracking without an invented identifier; bounded expiry; save/restore roundtrip across restart; restore of pre-upgrade cache files lacking the new section; and refetch exclusion for a real d-tag-less kind-30617 event. Full `cargo test` suite, `cargo fmt`, and `cargo clippy --all-targets` pass.

**Production soak before merge**: gitnostr.com was deployed from unmerged PR commit `256a9912` (service restart 14:07:20 UTC; binary store path registered 14:05:55 UTC, verified via the nixos-fi1 pin). Results:

- Zero occurrences of the old warning signature after deploy.
- The new tracking path fired on exactly the expected population: 41 distinct event IDs — matching the 41 from the production evidence — each tracked on the first historic pass at 14:07–14:08.
- 72 redundant re-validations followed at 14:09 from the bounded exact-ID missing-event recovery path (batches formed before the IDs were tracked), then complete silence: no further occurrences for 35+ minutes, spanning 3+ of the old ~11-minute cycle periods. Service healthy, no restarts, no new error patterns.

**Second commit** (`9d717fe`) addresses the one residual observed in the soak: missing-event recovery now consults the rejected index before re-processing delivered events, matching the existing skip in the relay event loop. A tracked ID still counts as recovered, so recovery outcomes are unchanged. Validated with fmt and the sync unit-test suite; the redundant re-validation behaviour it removes was observed directly in production logs.
 2026-08-04 16:02:08 +01:00
DanConwayDev f90107c679 Merge #02142b44: fix(sync): preserve proactive sync under relay REQ lim… 
fix(sync): preserve proactive sync under relay REQ limits

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsqy9ptg3h6de3wypkmfyed2pgy5gamaxzsfjacrh7w8ewvlkzfq5cefuqpy

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Public relays cap active subscription IDs, while GRASP-02 previously opened separate live and historic REQs for each state, repository-tag, and root-event filter. Rejection of the tail removed live coverage and could leave historic work incomplete.

This three-commit series first adds multiple-filter NIP-01 REQs at the transport layer, then batches up to ten compatible proactive filters under each subscription ID while preserving independent event counts, oldest timestamps, and pagination frontiers. The existing 300-ID negentropy fetch chunks deliberately remain one filter per REQ so their message-size bound is preserved.

Grouped historic pagination explicitly assumes that relay result limits apply independently per filter, the effective per-filter cap is at least 75, and no additional total-result cap can starve filters within a grouped REQ. These assumptions are recorded in the implementation and design documentation.

Rate-limit recovery now keeps the first deadline during an active cooldown, starts a fresh cooldown when a post-deadline recovery attempt is rejected, and does not confuse WebSocket connection success with REQ acceptance.

The integration regression uses a relay capped at four active REQs and synchronizes on an observed q-tagged event rather than a fixed sleep. Formatting, warnings-as-errors Clippy, and the complete ngit-grasp test suite pass. General asynchronous NIP-01 CLOSED recovery is inherited behavior and is intentionally left for focused follow-up work.
 2026-08-01 15:03:21 +01:00
DanConwayDev e889ffa4a7 Merge #c63bcd0b: fix: prune stale purgatory states after promotion 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsvvw7dpwceddf093cgt8rhdgrudfsmw25u3296cqw526dggel234cr8039y

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

## Summary

After a repository state is successfully promoted from purgatory, remove only non-preferred states on the same author and identifier when the locally available Git objects cannot reconstruct them.

## Preserved behavior

- State promotion remains oldest-first so reconstructable predecessors enter relay history before replacements.
- Equal timestamps retain the lower event ID under NIP-01.
- Reconstructable predecessors and other maintainer coordinates remain available as rollback candidates.
- Lifecycle-rejected saves do not trigger cleanup.

## Deliberately out of scope

This revision does not change receive-pack locking, push authorization, deletion handling, persistent watermarks, queuing, or admission control. Transactional receive-pack and DoS-resistant admission are a separate follow-up project.

## Validation

- 570 library tests
- 42 purgatory-sync tests
- workspace clippy with warnings denied
- two rounds of independent review by three reviewers

Latest narrow revision:
nostr:nevent1qqs2zv3rdvpcnsgnjtr5tm29r6vspgx9vv2jhur87u4uqrqaqutgkmspz3mhxue69uhhyetvv9ujumn8d96zuer9wcpzqetunatx5f3844mpje54xc0pmq2tmdy7s7us7ndcrraer3vac9p2jvp827
 2026-07-30 11:55:11 +01:00
DanConwayDev 49497629f7 Merge #28ce7595: Keep relay owner secrets out of process arguments 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsz3nn4jh38w8aa5ty7fg5cuxncmers3gwgupw90nmw7nfvmytpl2cne0c5v

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Fixes NixOS deployments exposing the relay-owner nsec through a shell-expanded ExecStart command.

- removes the relay-owner secret from CLI parsing and loads it from the relay_owner_nsec systemd credential, NGIT_RELAY_OWNER_NSEC/.env, or the persistent fallback in that order
- rejects empty or invalid provisioned identities instead of silently generating a replacement, and creates fallback key files with private permissions
- changes the NixOS module to LoadCredential plus a direct binary ExecStart and documents the operator migration

Regression coverage verifies argv rejection, credential trimming and validation, credential precedence, environment handling, fallback behavior, and generated file permissions. The full 566-test library suite and binary target pass.

Compatibility: direct --relay-owner-nsec users must move the value to NGIT_RELAY_OWNER_NSEC or a credential/file. The existing NixOS relayOwnerNsecFile option remains supported, but its source file must exist when systemd starts the service. This PR does not deploy or rotate any production key.
 2026-07-27 15:28:42 +01:00
DanConwayDev 438080c191 Merge #dd459383: docs(nix): avoid forcing unfiltered local module sourc… 
docs(nix): avoid forcing unfiltered local module sources

nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsd63vns0w7hayc350m0uymv9ummy058j2kff8m706sfgvecwh6kegdckpha

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

CoverNote:

Documents resource-safe NixOS module validation for ngit-grasp.

Importing `nix/module.nix` from a working tree is lazy by itself. The expensive path begins when rendering an enabled service forces the module-built package through `ExecStart`; `buildRustPackage` then coerces `src = ../.` relative to that local module and can recursively hash or copy an unfiltered checkout. Git-backed flake imports avoid this because `../.` resolves inside the already filtered store source.

The guide now recommends a Git-backed module or non-forcing builder stub, evaluation without builds, dry-run derivation inspection, one shared package derivation across instances, and conservative initial build limits. It also records the separate risk of accidentally forcing two ngit-grasp packages with different Rust toolchains concurrently.
 2026-07-27 15:24:09 +01:00
DanConwayDev be18f6083c Merge #d6e07abe: Fix invitation recovery during busy relay startup 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqsddcr6h6cafdqp6qmcxh9cd8fxc6tr7nc6asx60krjxeuqx34zkjc9p0928

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

Fixes maintainer invitation acceptance that remains in purgatory while a production GRASP server starts and ingests a large bootstrap history.

The live reproduction showed two independent blockers: startup waited minutes for retention cleanup across more than 52,000 deletion requests, then a rate-limit notice and paginated EOSE formed a circular wait on the SyncManager lock. Fresh invitation announcements were accepted into purgatory, but the five-second recovery pass could not run.

This PR starts retention catch-up after the relay and sync workers are live, resumes the exact historic page outside the actor lock after rate-limit cooldown, and replaces repeated full-index action construction with lightweight dirty-relay recomputation.

Coverage verifies deferred retention cleanup, preserves generic pagination while the actor remains available, limits relay recomputation to the changed batch, and keeps the existing maintainer invitation/state integration scenarios passing.
 2026-07-27 14:20:27 +01:00
DanConwayDev e6a0451200 Merge #565b5f5c: Fix receive-pack completion readiness and timeouts 
nostr:nevent1qgsx2lyl2e4zvfadwcvkd9fkrcwczj7mf858hy85mwqclwgut8wpg2spz3mhxue69uhhyetvv9ujumn8d96zuer9wcq3yamnwvaz7tm8d96xummnw3ezucm0d5q3kamnwvaz7tmwva5hgtnyv9hxxmmwwashjer9wchxxmmdqqs9vk6ltnmqd7tmte7pt53svndcz8w7eh68mg5kt2tckjyv3fdkllq2hx629

PR-Author: DanConwayDev's Agent
nostr:npub1v47f74n2ycn66asev62nv8sas99akj0g0wg0fkup37u3ckwuzs4q7cwtp0

PR description:

## Context

Commit f4828c63 streamed Git smart HTTP responses to fix the large-push timeout reported in Nostr issue 497c8ae1554037142e366f9ba363ba898fc16e90198fded195b9a45adb6f38c7. That correctly exposes receive-pack progress while Git resolves deltas and checks connectivity, but it also exposed the terminal 0000 flush before GRASP finished process_newly_available_git_data.

Git clients can treat that flush as completion without waiting for HTTP EOF. As a result, nostr_push could return while the repository announcement and state were still in purgatory. This is causing intermittent setup failures in the ngit git_push_merge test harness when publish_repo is followed immediately by a clone or proposal push.

## Changes

- cb7e5ae0 retains only the terminal receive-pack flush while continuing to stream all preceding Git progress, then promotes and saves repository state before releasing completion.
- 8abd2f4f sends protocol-valid sideband band-2 progress every five seconds while unusually complex post-push promotion or cross-owner synchronization is still running.
- The keepalive sender is cancelled and joined before the terminal flush; non-sideband responses remain unchanged.
- Deterministic coverage blocks promotion, observes progress and keepalive traffic, proves the event is unavailable before promotion, and proves it is queryable before completion.
- The changelog describes both completion readiness and timeout hardening.

## Scenarios covered

1. Large packs spending tens of seconds in Git delta resolution or connectivity checks continue to receive the progress streaming introduced by f4828c63.
2. Immediate follow-up operations cannot observe successful push completion before GRASP repository state is queryable.
3. Long GRASP post-push finalization does not trip ngit or libgit2 per-recv I/O timeouts when side-band-64k is negotiated.

## Validation

- cargo fmt --all --check
- cargo test
- cargo clippy --all-targets -- -D warnings
- post-rebase git_response_streaming integration suite

Related ngit push-status issue: af160581998e25482d81a6aa933591a32977c0d882fc4910da65bec20cff8258.
 2026-07-26 02:30:02 +01:00