ngit-grasp/closed/820a-relay-ngit-dev-migration.md

# Migrate relay.ngit.dev from ngit-relay to ngit-grasp

**ID:** 820a

## Problem

relay.ngit.dev currently runs ngit-relay (reference implementation). We want to consolidate on ngit-grasp as the production implementation to:
- Reduce operational complexity (one codebase instead of two)
- Focus logging and observability improvements on a single project
- Leverage ngit-grasp's proven performance on ngit.danconwaydev.com

**Why this matters:** Running two different implementations increases maintenance burden and makes it harder to add features like comprehensive logging and performance monitoring. Consolidating on ngit-grasp enables:
- Unified logging and metrics infrastructure
- Single codebase for bug fixes and features
- Consistent behavior across all deployments
- Reduced testing and deployment complexity

## Plan

**High-Level Migration Approach:** Archive-based migration to ensure complete git data availability and prevent purgatory issues.

**Timeline:** 3-5 days total (from archive sync completion)

**Migration Phases:**

1. ✅ **Phase 1: Archive Sync** (1-2 days) - COMPLETED
   - Deploy archive instance syncing from relay.ngit.dev
   - Archive fetches all git data during sync
   - Monitor sync progress and completion

2. ❌ **Phase 2: Validation** (4-6 hours) - BLOCKED
   - Nostr events (kind 30617): 99.86% sync ✅
   - State events (kind 30618): Production 39.3%, Archive 52.4% ❌ **INCOMPLETE**
   - Production baseline established: 685 repos, 212 npubs, 95% git data match ✅
   - Archive sync analysis complete: 1,227 repos, 1,226 npubs ✅
   - **CRITICAL BLOCKERS IDENTIFIED:**
     - 22 npubs in production NOT in archive (data loss risk) ❌
     - 1,036 npubs in archive NOT in production (unexplained) ⚠️
     - Repo count mismatches in shared npubs (sync inconsistency) ⚠️
     - State event coverage <60% on both relays (expected >95%) ❌
   - **Required:** Full investigation and re-sync before proceeding to Phase 3
   - Next: Investigate missing npubs, audit extra npubs, reconcile data

3. **Phase 3: Test Instance** (1 day) - PENDING
   - Deploy test instance syncing from archive
   - Validate all repos accessible
   - Verify no purgatory issues

4. **Phase 4: Production Cutover** (2-4 hours) - PENDING
   - Stop ngit-relay, deploy ngit-grasp
   - Monitor 30-minute critical window
   - Rollback if issues detected

5. **Phase 5: Post-Migration Monitoring** (48 hours) - PENDING
   - Intensive monitoring (15min → hourly → 4h → 8h intervals)
   - Compare metrics to baseline
   - Remove ngit-relay container after 7 days if successful

**Detailed Implementation:** 
- [Migration Guide](../../docs/how-to/ngit-relay-migration-guide.md) - High-level migration phases and procedures
- [Migration Validation Guide](../../docs/how-to/migration-validation-guide.md) - Detailed validation procedures and troubleshooting

## Accelerated Migration Plan

**Timeline:** 3-5 days total (vs 3-5 weeks original)

### Day 0-3: Minimal Baseline Collection (2-3 days) - IN PROGRESS ⏳

**Goal:** Capture enough baseline data to detect major regressions

**Status:** Collection started 2026-01-17 23:30 UTC, extended to 2-3 days (until 2026-01-20/21)

**Minimum metrics needed:**
- [x] CPU usage patterns (1 full day cycle minimum) - COLLECTING ✅
- [x] Memory baseline (current RSS, growth rate) - COLLECTING ✅
- [x] Peak connection counts (identify daily peak hours) - COLLECTING ✅
- [x] Basic response time percentiles (p50, p95, p99) - Via connection stats ✅
- [x] Error rate baseline (if any) - Via systemd logs ✅

**Metrics Status (2026-01-18):**
- ✅ Critical metrics working: connections, CPU, memory, load average
- ❌ Container stats broken (cgroup access issues) - NOT CRITICAL
- ❌ FD counts broken (permission issues) - NOT CRITICAL
- Decision: Proceed with working metrics, sufficient for migration decision

**Quick setup:**
```bash
# On relay.ngit.dev VPS
# 1. Check current metrics via systemd/journalctl
systemctl status ngit-relay
journalctl -u ngit-relay --since "24 hours ago" | grep -i error

# 2. Capture system metrics snapshot
top -b -n 1 | head -20
free -h
ss -s  # socket statistics

# 3. If nginx metrics available, capture 24h sample
# Otherwise, rely on system metrics + manual testing
```

**Acceptance criteria:**
- [ ] 2-3 days of continuous operation data (IN PROGRESS - ~10.5h collected)
- [ ] Identified peak usage hours (collecting data)
- [ ] No critical errors in baseline period (monitoring)
- [ ] Documented current resource usage (ongoing)

### Day 3-4: Rapid Preparation (4-6 hours)

**Configuration review:**
- [ ] Map ngit-relay config → ngit-grasp config
- [ ] Prepare NixOS service definition
- [ ] Document rollback procedure (keep ngit-relay container)
- [ ] Pre-build ngit-grasp on VPS (cache dependencies)

**Quick prep checklist:**
```bash
# 1. Review current ngit-relay config
docker inspect ngit-relay | jq '.[0].Config.Env'

# 2. Create ngit-grasp service config (see template below)
# 3. Test build on VPS
nix build github:DanConwayDev/ngit-grasp

# 4. Verify data directory permissions
ls -la /persistent/ngit-relay/
```

**Rollback plan:**
- Keep ngit-relay container stopped but available
- Document exact `docker start` command
- 5-minute rollback window if issues detected

### Day 4-5: Migration Execution (2-4 hours)

**Execution window:** Off-peak hours (based on Day 0-1 data)

**Steps:**
1. [ ] Stop ngit-relay container (don't remove)
2. [ ] Deploy ngit-grasp via NixOS
3. [ ] Verify service starts and binds to port
4. [ ] Test WebSocket connection
5. [ ] Test git clone operation
6. [ ] Monitor for 30 minutes (critical window)

**Go/No-Go criteria (30 min checkpoint):**
- ✅ Service running and accepting connections
- ✅ No error spikes in logs
- ✅ Memory usage within 2x of baseline
- ✅ CPU usage reasonable (<80% sustained)
- ✅ Git operations functional

**If No-Go:** Rollback immediately
```bash
systemctl stop ngit-grasp-production
docker start ngit-relay
# Total rollback time: <5 minutes
```

### Day 5-7: Intensive Monitoring (48 hours)

**Critical monitoring period:**
- [ ] Hour 1: Check every 15 minutes
- [ ] Hours 2-6: Check every hour
- [ ] Hours 6-24: Check every 4 hours
- [ ] Hours 24-48: Check every 8 hours

**Monitor:**
- CPU/memory trends (compare to baseline)
- Connection counts (should be similar)
- Error logs (any new patterns?)
- Response times (p95, p99)
- Git operation success rate

**Success criteria (48h):**
- ✅ No critical errors
- ✅ Resource usage ≤ 2x baseline (acceptable for new impl)
- ✅ No user complaints
- ✅ Git operations working
- ✅ WebSocket connections stable

**If successful:** Remove ngit-relay container after 7 days
**If issues:** Rollback and investigate

## Progress

### 2026-01-23 [Session 16:00] - Issue Paused - Starting Fresh with v2
- Decision: Migration has become too complex with extensive investigation history
- Reason: Need simpler, more focused approach with lightweight how-to documentation
- Action: Pausing this issue and creating new v2 issue to start from scratch
- Context: Will reference existing scripts and learnings from this worktree
- Goal: Create small, easy-to-implement how-to document for VPS migration
- Status: ⏸️ **PAUSED** - Superseded by new v2 migration issue

### 2026-01-17 [Session 22:00]
- Created: Initial issue based on pivot from 573b timeout diagnosis
- Context: Phase 1 of 573b is complete (connection limits increased, rate limiting fixed)
- Rationale: Better to consolidate on ngit-grasp than implement duplicate features in ngit-relay
- Blocker: Must collect baseline metrics BEFORE migration for objective comparison
- Next: Deploy nginx metrics to relay.ngit.dev and begin baseline collection

### 2026-01-17 [Session 23:30]
- Added: Accelerated migration plan (3-5 days vs 3-5 weeks)
- Decision: User wants to move quickly after minimal baseline collection
- Approach: 48-72h baseline → rapid prep → execute → intensive monitoring
- Risk mitigation: Keep ngit-relay container for instant rollback
- Next: Begin minimal baseline collection (48-72h)

### 2026-01-17 [Session 23:45] - Baseline Collection Started
- Completed: Enhanced diagnostics deployed to all three relays
- Improvements:
  1. ✅ Fixed connection counting (port-based, not PID-based)
  2. ✅ Added container stats collection (CPU, memory, network I/O)
  3. ✅ Added connection state breakdown (ESTABLISHED, TIME_WAIT, etc.)
  4. ✅ Created comprehensive MONITORING.md documentation
- Baseline Collection Status:
  - **Started:** 2026-01-17 23:30 UTC
  - **Duration:** 36 hours (until 2026-01-19 11:30 UTC)
  - **Relays monitored:** gitnostr.com, relay.ngit.dev, ngit.danconwaydev.com
  - **Enhanced metrics:** Connection counts per service, container stats, state breakdown
  - **Purpose:** Establish baseline before relay.ngit.dev migration
- Timeline Update:
  - **Day 0 (2026-01-17 23:30):** Baseline collection started
  - **Day 0+10min (2026-01-17 23:40):** Verify data quality
  - **Day 1.5 (2026-01-19 11:30):** Review baseline data (36 hours)
  - **Day 2:** Rapid preparation (if baseline looks good)
  - **Day 3:** Migration execution
  - **Day 3-5:** Intensive monitoring
- Next Steps:
  1. Check baseline at 10-minute mark (verify collection working)
  2. Review baseline after 36 hours
  3. Make migration decision based on baseline data quality

### 2026-01-18 [Session 10:00] - Baseline Collection Status Update
- Completed: 2 iterations of diagnostics fixes to improve metric reliability
- **Critical Metrics Working:**
  - ✅ Connection counts per service (port-based detection working)
  - ✅ System metrics (load average, memory, CPU breakdown)
  - ✅ Top processes (identifying resource consumers)
  - ✅ Connection state breakdown (ESTABLISHED, TIME_WAIT, etc.)
  - ✅ Basic container visibility (containers appear in top processes)
- **Non-Critical Metrics Still Broken:**
  - ❌ Container stats (cgroup access issues, requires privileged systemd-run)
  - ❌ File descriptor counts per process (permission issues)
  - Note: Container resource usage still visible in top processes section
- **Decision: Proceed with Current Metrics**
  - Critical metrics (connections, CPU, memory) working reliably
  - Broken metrics are nice-to-have, not essential for migration decision
  - Have sufficient data to detect major regressions
  - Can proceed with migration planning based on working metrics
- **Baseline Collection Extended:**
  - Current duration: ~10.5 hours (started 2026-01-17 23:30 UTC)
  - Extended to: 2-3 more days (until 2026-01-20 or 2026-01-21)
  - Reason: Want more data to establish reliable daily patterns
  - Collection continuing with working metrics
- **Updated Timeline:**
  - **Day 0 (2026-01-17 23:30):** Baseline collection started
  - **Day 0.5 (2026-01-18 10:00):** Diagnostics fixes complete, extending collection
  - **Day 2-3 (2026-01-20/21):** Review baseline data (2-3 days total)
  - **Day 3-4:** Rapid preparation (if baseline shows stable patterns)
  - **Day 4-5:** Migration execution
  - **Day 5-7:** Intensive monitoring
- **Next Steps:**
  1. Continue baseline collection for 2-3 more days
  2. Review baseline data patterns after collection period
  3. Make migration decision based on baseline analysis
  4. Proceed with rapid preparation if baseline looks good

### 2026-01-19 [Session 11:35] - Baseline Analysis Complete - VPS UPGRADE REQUIRED ⚠️
- Completed: Comprehensive analysis of 36 hours of baseline metrics
- **CRITICAL DECISION: Migration BLOCKED until VPS upgrade**
- **Baseline Findings (2026-01-17 23:30 to 2026-01-19 11:35 UTC):**
  - **CPU:** Average 6.96 load (348% utilization), peak 11.83 (592%) - 🔴 SEVERELY OVERLOADED
  - **Memory:** 96.8% average usage, only 130MB available - 🔴 MAXED OUT
  - **Swap:** 2.0GB average (40% of 5GB), growing 0.75GB → 2.5GB - 🟡 HEAVILY USED
  - **Connections:** Port 8081: 28 avg/120 peak, Port 8083: 31 avg/126 peak - ✅ ONLY 3% OF CAPACITY
  - **Correlation:** Load vs connections = 0.210 (weak) - Confirms git operations are bottleneck, not connections
- **Peak Usage Patterns:**
  - Peak hours: 16:00-20:00 UTC (load 6.5-9.9, connections 865-977)
  - Off-peak hours: 00:00-12:00 UTC (load 6.3-7.4, connections 350-470)
  - **Critical insight:** Even off-peak shows HIGH load (6.3-7.4), indicating baseline overload independent of traffic
- **Migration Risk Assessment:**
  - **Current VPS (2-core, 4GB):** 🔴 **HIGH RISK** - Migration will add load spike to already overloaded system
  - **Outcome:** High probability of service degradation or outages during migration
  - **Recommendation:** ❌ **DO NOT MIGRATE** on current VPS
- **VPS Upgrade Requirements:**
  - **Minimum (Survival):** 4 cores, 8GB RAM (~$20-40/month) - 🟡 MEDIUM RISK
  - **Recommended (Healthy):** 6 cores, 8GB RAM (~$40-60/month) - 🟢 LOW RISK ✅
  - **Optimal (Future-proof):** 8 cores, 16GB RAM (~$60-100/month) - 🟢 LOW RISK
  - **Decision:** Upgrade to 6-core, 8GB minimum before migration
- **Updated Migration Timeline:**
  - **REVISED:** 5-7 days total (was 3-5 days)
  - **Day 0-1:** VPS upgrade + verification (NEW - CRITICAL)
  - **Day 2-3:** Rapid preparation
  - **Day 4-5:** Migration execution
  - **Day 5-7:** Intensive monitoring
- **Data Quality:**
  - ✅ 36 hours continuous collection, 2,160 data points
  - ✅ Clear patterns identified, no anomalies
  - ✅ **Sufficient for migration decision** - no additional collection needed
  - ✅ Findings are conclusive: VPS upgrade required
- **Next Steps:**
  1. **CRITICAL:** Upgrade VPS to 6-core, 8GB RAM (BLOCKER for migration)
  2. Verify upgrade impact (24-48 hours monitoring, expect load to drop to ~60-70%)
  3. Proceed with accelerated migration plan (Day 2-7)
  4. Monitor intensively during and after migration
- **Files Created:**
  - `/tmp/vps-baseline-report.md` - Comprehensive 36-hour analysis with migration scenarios
- **Status:** Baseline complete ✅, Migration BLOCKED ⚠️ pending VPS upgrade
- **Blocker:** VPS upgrade to 6-core, 8GB RAM required before migration can proceed

### 2026-01-20 [Session 16:00] - Worktree Created for Migration Work
- Started: Created dedicated worktree for migration work
- Context: Work has progressed in /tmp directory, now moving to proper worktree
- Next: Save relevant files from /tmp into this worktree and continue migration work

### 2026-01-20 [Session 18:30] - State Event Validation FAILED - Migration BLOCKED ❌
- Completed: State event (kind 30618) validation between relay.ngit.dev and archive
- **CRITICAL BLOCKER IDENTIFIED:**
  - State event sync: **42.4%** (expected >99%) ❌
  - Archive missing 338/587 repos from production (57.6% gap)
  - Git data verification failed: sample repos have empty refs directories
  - Purgatory suspected: repos synced but git data incomplete
- **Comparison to kind 30617:**
  - Announcements (30617): 99.86% sync ✅
  - State events (30618): 42.4% sync ❌
  - Indicates archive failed to fetch/maintain git data for majority of repos
- **Root cause analysis:**
  - Archive successfully synced event metadata
  - Archive failed to fetch git data from clone URLs
  - ngit-grasp correctly withholds state events when git data missing
  - Result: 57.6% of repos would lose main/master branch visibility after migration
- **Files created:**
  - `work/state-validation/STATE-VALIDATION-REPORT.md` - Comprehensive analysis
  - `work/state-validation/missing-from-archive.txt` - List of 338 missing repos
  - `work/state-validation/archive-metrics.txt` - Archive relay metrics
- **Migration status:** ❌ **BLOCKED** - Cannot proceed to Phase 3 with <50% state event coverage
- **Next actions required:**
  1. Investigate why archive git fetches failed for 338 repos
  2. Check archive sync logs for git fetch errors
  3. Determine if sync still in progress or permanently failed
  4. Fix git fetch issues and re-sync missing repos
  5. Re-validate state events once archive sync issues resolved
- **Blocker:** Archive sync incomplete/failed - must achieve >99% coverage before migration

### 2026-01-20 [Session 19:00] - Archive Sync Blocker ROOT CAUSE FOUND 🔍
- Completed: Investigation of archive sync failures for missing 338/587 repos (57.6%)
- **ROOT CAUSE IDENTIFIED:**
  - **Archive is attempting to clone from `https://relay.ngit.dev/...` URLs**
  - **relay.ngit.dev does NOT serve HTTP git clones** - only WebSocket relay service
  - Manual test confirmed: `git clone https://relay.ngit.dev/<npub>/<repo>.git` returns 404
  - Archive has been running 36+ hours but silently failing on relay.ngit.dev URLs
- **Clone URL Pattern:**
  ```
  ["clone","https://git.shakespeare.diy/.../repo.git","https://relay.ngit.dev/.../repo.git"]
  ```
  - Primary URLs (git.shakespeare.diy, github.com, etc.): ✅ Works
  - Fallback URLs (relay.ngit.dev): ❌ 404 Not Found
- **Archive Metrics (Port 7443):**
  - Service status: ✅ Running normally (19+ hours uptime)
  - Events synced: 3,445 from wss://relay.ngit.dev
  - Repositories tracked: 1,856 (matches 42% coverage)
  - Disk usage: 4.8GB (40-48% of expected 10-12GB)
  - **Confirms partial sync, not full sync**
- **Why Only 42% Coverage:**
  - Archive successfully fetches repos with working primary clone URLs
  - Archive FAILS for repos with only relay.ngit.dev clone URLs (338/587 = 57.6%)
  - Missing repos likely have relay.ngit.dev as primary or only clone URL
- **Configuration Analysis:**
  - Archive config is CORRECT: `NGIT_DOMAIN=archive.internal`
  - Sync source is CORRECT: `wss://relay.ngit.dev`
  - Archive is NOT filtering relay.ngit.dev URLs (domain is different)
- **Files Created:**
  - `work/state-validation/ARCHIVE-SYNC-INVESTIGATION.md` - Full investigation report
  - Documented service status, metrics, logs, root cause, and recommended fix
- **Recommended Fix: Option 1 (RECOMMENDED)**
  - Configure relay.ngit.dev to serve HTTP git clones
  - Add nginx/HTTP service pointing to `/persistent/grasp/production/git/`
  - Serve at `https://relay.ngit.dev/<npub>/<repo>.git/`
  - Makes relay.ngit.dev a complete git hosting service
  - No event republishing required
- **Alternative Fixes:**
  - Option 2: Update state events to remove non-working clone URLs (complex)
  - Option 3: Enhance archive retry logic (doesn't solve URL problem)
- **Next Steps:**
  1. Decide on fix approach (Option 1 recommended)
  2. Configure HTTP git service on relay.ngit.dev
  3. Restart archive to retry failed fetches
  4. Monitor archive metrics for coverage improvement
  5. Re-run state validation (expect >95% coverage after fix)
- **Migration Status:** ❌ **BLOCKED** - HTTP git service required on relay.ngit.dev
- **Success Criteria:** Archive coverage >95% (587/587 repos), git data size ~10-12GB
- **Issue Created:** 1ce6-purgatory-retry-mechanism.md - Documents purgatory retry problem for future fix

### 2026-01-20 [Session 16:00] - Git Data Analysis Complete - 70.8% Production Repos Fully Synced
- Completed: Comprehensive analysis of production git data vs state events
- **Key Findings:**
  - **177/250 repos (70.8%)** have perfect sync on production ✅
  - **62/250 repos (24.8%)** are empty/blank - legitimately missing git data
  - **6/250 repos (2.4%)** have partial matches - need careful review
  - **5/250 repos (2.0%)** have no matches - stale state events or force-pushed
- **Comparison to Archive:**
  - Production: 70.8% complete matches, 24.8% empty/blank
  - Archive: Only 13% repos with git data, 85% in purgatory
  - **Critical insight:** Archive sync is failing for repos that production already has
- **Implication:** Archive-based migration approach has fundamental issue
  - Archive can't sync repos that production already serves successfully
  - Need to identify which "Complete Match" repos are missing from archive
  - Review archive logs to understand why these repos aren't syncing
- **Next Steps:**
  1. Identify which 177 "complete match" repos are missing from archive
  2. Review archive logs to understand sync failures
  3. Mark 62 empty/blank repos as not needed (can discard)
  4. Careful review of 11 partial/no-match repos (architect analysis)
- **Files Created:**
  - `/tmp/git-state-analysis/category-complete-matches.txt` - 177 fully synced repos
  - `/tmp/git-state-analysis/category-empty-blank.txt` - 62 repos with no git data
  - `/tmp/git-state-analysis/category-partial-matches.txt` - 6 repos needing review
  - `/tmp/git-state-analysis/category-no-matches.txt` - 5 repos with wrong commit hashes
- **Migration Status:** Archive approach investigation ongoing ⚠️

### 2026-01-20 [Session 18:10] - COMPREHENSIVE ANALYSIS COMPLETE - Archive NOT Ready ❌
- Completed: Full production vs archive comparison with corrected analysis methodology
- **CRITICAL FINDING: Previous analysis had permission issues - actual results differ significantly**
- **Production Baseline (relay.ngit.dev - ngit-relay):**
  - Total repos: 685 repos (6.5GB)
  - NPub count: 212 unique users
  - State events: 633 total, 249 served (39.3% coverage)
  - Git data match: **95% of state events have matching git directories** ✅
  - Directory structure: `/persistent/relay-ngit-dev-ngit-relay/data/repos/<npub>/<repo>.git`
- **Archive Status (ngit-grasp-relay-ngit-archive):**
  - Total repos: 1,227 repos (5.4GB) - **79% MORE than production**
  - NPub count: 1,226 unique users - **5.8x MORE than production**
  - State events: 332 served (52.4% coverage) - **33% MORE than production**
  - Service: Active and syncing (3+ hours uptime)
  - Directory structure: `/persistent/grasp/sync-archive/git/<npub>/<repo>.git`
- **Migration Gap Analysis:**
  - **Shared npubs:** 190 (both production and archive)
  - **Missing from archive:** 22 npubs (in production but NOT in archive) ❌ **DATA LOSS RISK**
  - **Extra in archive:** 1,036 npubs (in archive but NOT in production) ⚠️ **UNEXPLAINED**
  - **Repo count mismatches:** Many shared npubs have different repo counts between prod/archive
- **State Event Coverage:**
  - Production: 249/633 (39.3%) - **60.7% gap**
  - Archive: 332/633 (52.4%) - **47.6% gap**
  - Neither relay serves complete state event coverage
- **Analysis Script Issues Discovered:**
  - Initial runs showed "100% no_git_dir" due to permission issues (script didn't use sudo)
  - Corrected analysis with sudo shows 95% git data match in production
  - Script expects specific directory structure that matches both production and archive
- **Migration Readiness Assessment:** ❌ **NOT READY**
  - **Blocker 1:** 22 npubs missing from archive (data loss risk)
  - **Blocker 2:** 1,036 unexplained extra npubs in archive (data quality concern)
  - **Blocker 3:** Repo count mismatches in shared npubs (sync inconsistency)
  - **Blocker 4:** Neither relay has complete state event coverage (<60%)
- **Required Actions Before Migration:**
  1. **Investigate 22 missing npubs** - Why aren't they in archive? Recent additions? Blocked?
  2. **Audit 1,036 extra npubs** - Where did they come from? Historical data? Test data?
  3. **Reconcile repo count mismatches** - Identify stale/missing repos in shared npubs
  4. **Full re-sync** - Ensure all 212 production npubs are in archive with correct data
  5. **Validate state event coverage** - Understand why only 39-52% coverage (expected >95%)
- **Files Created:**
  - Analysis reports from 4 parallel subagent investigations
  - Production baseline: 685 repos, 212 npubs, 95% git data match
  - Archive status: 1,227 repos, 1,226 npubs, active sync
  - Gap analysis: 22 missing, 1,036 extra, inconsistent repo counts
- **Next Steps:**
  1. Investigate the 22 missing npubs (check recent activity, sync logs)
  2. Audit archive git history to understand 1,036 extra npubs
  3. Create detailed sync plan to reconcile production → archive
  4. Re-validate after sync completion
  5. Only proceed to Phase 3 after achieving >95% coverage
- **Migration Status:** ❌ **BLOCKED** - Archive needs cleanup and full re-sync before migration

### 2026-01-21 [Session 10:00] - Fetch Failure Deep Dive - ROOT CAUSE CORRECTED
- Completed: Deep investigation of "not our ref" error and purgatory flow
- **Previous analysis was INCORRECT** - "not our ref" was a red herring
- **Actual Root Cause:** Negentropy sync failures preventing state event transfer
- **Key Findings:**
  - Archive service IS running correctly on port 7443 (NOT 7334)
  - "not our ref" error was from git.shakespeare.diy (stale data), NOT relay.ngit.dev
  - Purgatory mechanism works correctly - tries multiple URLs, succeeds on fallback
  - Real issue: State events not being synced via negentropy
- **Evidence:**
  - storyforge-ai: State event received → purgatory → fetch failed from shakespeare → succeeded from relay.ngit.dev ✅
  - 1-man-1-tank: State event NEVER received → never entered purgatory → empty git repo ❌
  - No logs for 1-man-1-tank in archive (event never arrived)
  - Archive has 500 announcements but only 336 state events (33% gap)
- **Purgatory Flow (Working):**
  ```
  State event received → Added to purgatory → Sync triggered → 
  Try URL 1 (failed) → Try URL 2 (succeeded) → Git data fetched → 
  State event released from purgatory → Stored in database
  ```
- **Why 1-man-1-tank Failed:**
  - State event created: Dec 17, 2025
  - Archive started: Jan 20, 2026 (34 days later)
  - Negentropy sync failed to transfer the state event
  - Without state event, no purgatory entry, no git fetch
- **Sync Status:**
  ```
  10:20:11 - WARN: Batch failed - ConnectedHistoricSyncFailures
  10:20:17 - Historic sync complete with failures
  ```
- **Files Created:**
  - `work/FETCH-FAILURE-DEEP-DIVE.md` - Comprehensive analysis report
- **Recommendations:**
  1. Investigate negentropy sync failures (why state events not transferring)
  2. Add more sync relays for redundancy
  3. Implement state event re-fetch when announcement exists but state missing
  4. Add monitoring for announcement/state event ratio
- **Migration Status:** ❌ **BLOCKED** - Must fix negentropy sync before proceeding
- **Next Steps:**
  1. Check if there are filters preventing old state event sync
  2. Restart archive to trigger fresh negentropy sync
  3. Monitor logs for 1-man-1-tank state event arrival
  4. Consider manual event injection for missing state events

### 2026-01-21 [Session 14:00] - Negentropy Fallback Analysis - SOLUTION IDENTIFIED ✅
- Completed: Architect review of negentropy fallback mechanism and sync failures
- **CRITICAL FINDING: Fallback IS working, but failed batches are NOT retried**
- **Negentropy Fallback Status:**
  - ✅ Fallback to REQ+EOSE IS implemented correctly (`src/sync/mod.rs:3152-3512`)
  - ✅ Archive detects ngit-relay doesn't support negentropy (NIP-77)
  - ✅ Falls back to REQ+EOSE for current batch
  - ❌ **Failed historic sync batches are NEVER retried** - this is the bug
- **Root Cause Confirmed:**
  1. Archive started Jan 20, 2026 (many state events from Dec 2025)
  2. Initial historic sync attempted negentropy against ngit-relay
  3. Negentropy timed out (ngit-relay doesn't support NIP-77)
  4. Batch marked as failed → status: `ConnectedHistoricSyncFailures`
  5. REQ+EOSE fallback executed for current batch only
  6. **Old state events from failed batches never retried** ❌
- **Why 500 Announcements vs 336 State Events:**
  - Announcements: Synced via live subscription (new events after Jan 20)
  - State events: Only received AFTER archive started
  - Missing ~164 events: From before Jan 20, lost in failed historic sync
- **Sync Status:** COMPLETE (with failures) - not in progress
  - `ConnectedHistoricSyncFailures` means historic sync finished but had failures
  - Live sync active for new events
  - No automatic retry for failed historic events
- **Solution: Disable Negentropy for Archive**
  - Add `NGIT_SYNC_DISABLE_NEGENTROPY=true` to archive config
  - Restart service to trigger fresh historic sync
  - Forces REQ+EOSE for all historic sync (avoids timeout)
  - Should sync all historic events successfully
- **Files Created:**
  - `work/NEGENTROPY-FALLBACK-ANALYSIS.md` - Detailed code analysis and findings
- **Code References:**
  - `src/sync/mod.rs:3152-3512` - Historic sync with fallback logic
  - `src/sync/relay_connection.rs:508-584` - Negentropy failure detection
  - `src/sync/mod.rs:1109-1113` - Batch failure handling (no retry)
  - `src/config.rs:409-413` - `sync_disable_negentropy` config option
- **Migration Status:** ❌ **BLOCKED** - Must apply config fix and restart archive
- **Next Steps:**
  1. Add `NGIT_SYNC_DISABLE_NEGENTROPY=true` to archive service config
  2. Restart archive service to trigger fresh historic sync
  3. Monitor sync progress (expect 500+ state events)
  4. Verify state event count matches production (>95% coverage)
  5. Proceed to Phase 3 validation once sync complete

### 2026-01-21 [Session 15:00] - Negentropy Fix Applied & Sync Query Investigation ⚠️
- Completed: Applied negentropy fix, verified working, but discovered sync query design issue
- **Negentropy Fix Applied Successfully:**
  - ✅ Issue b4d0 created: "Failed historic sync batches not retried after negentropy timeout"
  - ✅ Config updated: Added `syncDisableNegentropy = true` to archive service
  - ✅ Deployed and restarted: Service running with REQ+EOSE sync
  - ✅ Fix verified: Logs show `use_negentropy=false`, state events flowing
  - ✅ State event count: 336 → 339 (increasing)
  - ✅ Status: `Connected` (not `ConnectedHistoricSyncFailures`)
- **1-man-1-tank Investigation - CRITICAL FINDING:**
  - ❌ State event still NOT received after negentropy fix
  - ✅ Announcement (kind 30617) exists in archive
  - ❌ State event (kind 30618) does NOT exist in archive
  - ✅ State event exists on relay.ngit.dev and git.shakespeare.diy
  - ❌ Git repo empty (no refs, no objects)
- **ROOT CAUSE: Sync Query Design Flaw**
  - Announcements: Broad query `{"kinds": [10317, 30617]}` (no d-tag filter) ✅
  - State events: Narrow query `{"kinds": [30618], "#d": [list of known repos]}` ❌
  - **Problem:** State events only queried for repos archive already knows about
  - **Chicken-and-egg:** 1-man-1-tank state event never synced → not in query list → never queried → never synced
- **Why This Happens:**
  1. Archive receives announcement for 1-man-1-tank (broad query)
  2. Archive creates empty git repo
  3. Archive queries state events only for repos it already has state events for
  4. 1-man-1-tank NOT in list → state event never queried
  5. State event never arrives → git data never fetched → repo stays empty
- **Evidence from Logs:**
  ```
  # Broad announcement query (works)
  Sending '["REQ",...,{"kinds":[10317,30617]}]' to relay.ngit.dev
  
  # Narrow state event query (misses 1-man-1-tank)
  Sending '["REQ",...,{"#d":["BeatsBars","storyforge-ai",...],
                       "kinds":[30618]}]' to relay.ngit.dev
  # Note: "1-man-1-tank" NOT in d-tag list
  ```
- **Impact Assessment:**
  - Negentropy fix IS working (REQ+EOSE active, other repos syncing)
  - But won't help repos whose state events were never queried
  - Affects ALL repos where announcement arrived but state event didn't
  - Explains persistent gap between announcement count and state event count
- **Recommended Fixes:**
  1. **Option 1 (Best):** Add broad state event query `{"kinds": [30618]}` (no d-tag filter)
  2. **Option 2:** Query state events for all known announcements (when announcement received)
  3. **Option 3 (Immediate):** Manual re-sync by publishing state events to archive
- **Files Created:**
  - `work/NEGENTROPY-FIX-VERIFICATION.md` - Verification of negentropy fix
  - `work/1-MAN-1-TANK-SYNC-STATUS.md` - Detailed investigation of sync query issue
- **Migration Status:** ❌ **BLOCKED** - Sync query design prevents complete state event coverage
- **Next Steps:**
  1. ~~Create issue for sync query design flaw~~ - Architecture is correct, not a design flaw
  2. Investigate actual execution failure for 1-man-1-tank
  3. Verify where the flow breaks (announcement → state event)
  4. Fix identified issue
  5. Achieve >95% state event coverage before proceeding to Phase 3

### 2026-01-21 [Session 16:00] - ACTUAL ROOT CAUSE: RepoSyncIndex Not Populated from Database 🎯
- Completed: Re-investigation with corrected understanding of sync architecture
- **Previous analysis was WRONG** - Sync query design is CORRECT, not flawed
- **Actual Architecture (Verified from Code):**
  1. Layer 1 syncs announcements broadly: `{"kinds": [30617]}` (no d-tag filter)
  2. Accepted announcements broadcast to local relay: `local_relay.notify_event(event)`
  3. SelfSubscriber receives announcement, extracts identifier, queues for batch
  4. Batch processing builds new filters including the NEW repo
  5. State event filters created: `{"kinds": [30618], "#d": ["new-repo", ...]}`
  6. AddFilters sent to SyncManager → state events synced
- **Architecture is CORRECT** - The flow is designed to sync state events for newly discovered repos
- **ACTUAL ROOT CAUSE: RepoSyncIndex Not Populated from Database**
  - `RepoSyncIndex` is ONLY populated by SelfSubscriber receiving broadcasts
  - `RepoSyncIndex` is NOT populated from database on startup or daily sync
  - If broadcast is lost (channel overflow, timing), announcement in DB but NOT in RepoSyncIndex
  - State event filter never created for that repo
- **Why This Happens:**
  1. Archive starts, SelfSubscriber connects to local relay
  2. SyncManager syncs 500+ announcements quickly
  3. Each announcement broadcast via `notify_event()` (channel capacity: 1024)
  4. If SelfSubscriber can't keep up, broadcasts overflow and are lost
  5. Lost announcements never added to RepoSyncIndex
  6. State event filters never created for those repos
- **Why Daily Sync Doesn't Fix It:**
  - `fresh_start()` calls `recompute_new_sync_filters_for_relay()`
  - Uses `derive_relay_targets(&repo_index)` - the RepoSyncIndex
  - RepoSyncIndex STILL missing the announcement
  - State event filter still never created
- **Evidence for 1-man-1-tank:**
  - ✅ Announcement (30617) in database (verified via nak query)
  - ❌ State event (30618) NOT in database
  - ❌ Likely NOT in RepoSyncIndex (broadcast lost during initial sync)
  - ❌ State event filter never created
  - ❌ State event never synced
- **Code References:**
  - `src/sync/mod.rs:2256-2257` - Uses RepoSyncIndex for filter creation
  - `src/sync/mod.rs:2575-2577` - Broadcast to local relay
  - `src/sync/self_subscriber.rs:155-169` - SelfSubscriber receives broadcast
  - `src/sync/self_subscriber.rs:503-506` - Batch processing builds filters
  - Broadcast channel capacity: 1024 (can overflow during initial sync)
- **Recommended Fixes:**
  1. **Option 1 (Best):** Populate RepoSyncIndex from database on startup
  2. **Option 2:** Query database in `recompute_new_sync_filters_for_relay()`
  3. **Option 3:** Persist RepoSyncIndex to disk
  4. **Option 4:** Increase broadcast channel capacity (doesn't fully solve issue)
- **Files Created:**
  - `work/1-MAN-1-TANK-FLOW-ANALYSIS.md` - Comprehensive flow analysis and root cause
- **Migration Status:** ❌ **BLOCKED** - RepoSyncIndex bug prevents complete state event coverage
- **Next Steps:**
  1. Create issue for RepoSyncIndex database population bug
  2. Implement Option 1 (populate from database on startup)
  3. Restart archive service to trigger database population
  4. Verify 1-man-1-tank state event syncs successfully
  5. Achieve >95% state event coverage before proceeding to Phase 3

### 2026-01-21 [Session 17:00] - Validation Strategy Designed 📋
- Completed: Comprehensive validation strategy for RepoSyncIndex hypothesis
- **Validation Approach: 3-Phase Strategy**
  - **Phase 1 (Recommended First):** Diagnostic logging with `[DIAG:*]` prefixes
    - 7 strategic log points tracking database → broadcast → SelfSubscriber → RepoSyncIndex → filters
    - Minimal code changes, can deploy immediately
    - Provides direct evidence of where broadcasts are lost
  - **Phase 2:** Debug HTTP endpoints for real-time inspection
    - `/debug/sync-gap` - Compare database vs RepoSyncIndex (JSON)
    - `/debug/repo-sync-index/{repo_id}` - Check specific repo (e.g., 1-man-1-tank)
    - Allows validating fix effectiveness
  - **Phase 3:** Metrics for production monitoring
    - `ngit_sync_repos_in_index_total` - Current RepoSyncIndex size
    - `ngit_sync_gap_total` - Gap between database and index
    - `ngit_sync_broadcast_overflow_total` - Failed broadcasts
- **Validation Steps:**
  1. Baseline: Query database for 1-man-1-tank (exists) and state event (missing)
  2. Deploy Phase 1 logging
  3. Collect logs for 1-2 hours
  4. Analyze: Count events at each stage (`DIAG:DB` vs `DIAG:SELFSUBSCRIBER`)
  5. Confirm hypothesis: If DB count > SelfSubscriber count, broadcasts being lost
- **Success Criteria:**
  - Database has N announcements, RepoSyncIndex has M < N (gap exists)
  - 1-man-1-tank in database but NOT in RepoSyncIndex
  - `DIAG:BROADCAST.*FAILED` messages in logs
  - After fix (populate from DB), gap becomes 0 and state events sync
- **Timeline Estimate:**
  - Phase 1 only: 1 day (minimum viable validation)
  - Full validation + fix: 3-4 days
- **Files Created:**
  - `work/REPOSYNCINDEX-VALIDATION-STRATEGY.md` - 904-line comprehensive strategy document
- **Alternative Hypotheses (if disproved):**
  1. State event filters created but queries fail
  2. Remote relay doesn't have state events
  3. Write policy rejecting state events
  4. Purgatory blocking state events
- **Risk Assessment:**
  - Performance impact: Minimal (debug/trace level logs)
  - Operational risk: Low (appropriate log levels, gated endpoints)
  - Mitigation: Use structured logging, rate limiting, feature flags
- **Next Steps:**
  1. Decide: Validate hypothesis first OR implement fix directly
  2. If validating: Implement Phase 1 logging, collect data, analyze
  3. If fixing directly: Implement database population on startup
  4. Either way: Create issue for RepoSyncIndex bug

### 2026-01-21 [Session 18:00] - Migrating to Selective Archive Service ✅
- Completed: Migration from archive-all to selective relay.ngit.dev archive
- Context: grasp-audit package updated with `archiveGraspServices` option
- **Configuration changes made:**
  - Removed: `ngit-grasp-archive-all.nix` (archived all relays)
  - Added: `ngit-grasp-relay-ngit-dev.nix` (archives only relay.ngit.dev)
  - Updated: `services/default.nix` to import new service file
  - New data directory: `/persistent/grasp/relay-ngit-dev` (was `/persistent/grasp/sync-archive`)
  - New domain: `archive-relay-ngit-dev.internal` (was `archive.internal`)
  - Selective archiving: `archiveGraspServices = [ "relay.ngit.dev" ]`
- **Migration steps executed:**
  1. ✅ Stopped old archive-all service
  2. ✅ Deleted old archive-all data directory (freed 5.8GB)
  3. ✅ Updated NixOS configuration and deployed
  4. ✅ Created data directory manually (workaround for namespace issue)
  5. ✅ Verified new service started successfully
- **Verification results:**
  - Service status: ✅ Running (active since 16:09:47 CET)
  - Archive mode: ✅ Confirmed (`archive_all=false`, read_only=true)
  - Syncing from: ✅ `wss://relay.ngit.dev`
  - Data archived: ✅ 779MB (and growing)
  - Git fetches: ✅ Active (fetching repos from relay.ngit.dev)
  - Service domain: ✅ `archive-relay-ngit-dev.internal`
  - Port: ✅ 7443 (localhost only)
- **Disk space:**
  - Before: 26GB used, 21GB available
  - After cleanup: 20GB used, 27GB available (5.8GB freed)
  - Current: 20.8GB used, 26.2GB available (779MB new data)
- **Issues encountered:**
  - Mount namespace error on first deployment (data directory didn't exist)
  - Resolution: Created directory manually, service started successfully
  - Note: This is a known issue (b454) - module should create directories in ExecStartPre
- **Migration status:** ✅ **COMPLETE** - Selective archive service operational

### 2026-01-20 [Session 14:30] - Naughty List Blocker Identified
- Completed: Extensive state validation and root cause analysis
- **Blocker identified:** relay.ngit.dev on git "naughty list" preventing archive sync
- **Current metrics:**
  - State events: 318/580 (55% of target, need >99%)
  - Repos with git data: 297 (14% of total)
  - Repos in purgatory: 1,803 (85% of total repos)
  - Coverage: Only 55% (BLOCKED from proceeding)
- **Root cause analysis:**
  - Permission issues FIXED ✅ (18 repos, all HTTP 200)
  - Purgatory mechanism understood ✅ (30-min expiry, no 7-day block)
  - Service restarted and purgatory cleared ✅
  - BUT: Archive can't fetch from relay.ngit.dev (on naughty list)
- **Naughty list evidence:**
  - Logs show: "Domain is on git naughty list - skipping identifier=X domain=relay.ngit.dev"
  - BOTH relay.ngit.dev AND git.shakespeare.diy blocked
  - Archive blocking its own bootstrap relay (likely bug)
- **Impact:**
  - Many repos only have relay.ngit.dev clone URLs
  - Archive can't fetch from blocked domain
  - Repos stuck in purgatory without git data
  - Can't serve state events without git data
- **Files created:**
  - 7 analysis reports in `work/state-validation/reports/`
  - 9 data files in `work/state-validation/data/`
  - 3 metrics snapshots in `work/state-validation/metrics/`
  - `work/NEXT-SESSION-HANDOFF.md` - Comprehensive handoff document
  - `work/QUICK-REFERENCE.md` - Quick commands reference
  - `work/FILE-MANIFEST.md` - File index with descriptions
- **Next session tasks:**
  1. Find naughty list implementation in ngit-grasp codebase
  2. Understand why relay.ngit.dev was added (appears to be bug)
  3. Remove relay.ngit.dev from naughty list
  4. Wait 1-2 hours for archive to fetch from unblocked domain
  5. Reassess coverage (target: >580 state events)
- **Migration Status:** ❌ **BLOCKED** - Must resolve naughty list before Phase 3

### 2026-01-20 [Session 17:00] - Nostr Event Analysis Complete & Migration Guide Created
- Completed: Comprehensive Nostr event analysis comparing relay.ngit.dev and localhost archive
- Key Findings:
  - **99.86% sync accuracy** between relay.ngit.dev (690 events) and localhost:7334 (2,157 events)
  - Only **1 missing repository**: einundzwanzighh-gesundesgeld (created 2025-01-19)
  - localhost:7334 is authoritative source with 2-year history vs relay.ngit.dev's 8-month subset
  - boldwallet.git confirmed on localhost:7334 ✅, missing from relay.ngit.dev ❌
- Created: Comprehensive migration guide at `docs/how-to/ngit-relay-migration-guide.md`
  - Documented two-relay architecture discovery
  - Included helper scripts for querying, comparing, and validating relays
  - Detailed 5-phase migration plan with validation steps
  - Troubleshooting section for common issues
- Organized: Copied analysis files to `work/analysis/` for reference
  - CORRECTED-ANALYSIS.md - Full event count comparison
  - MISSING-ANALYSIS.md - Missing repository analysis
  - missing-announcements.jsonl - Event ready for import
  - missing-announcements-summary.txt - Human-readable summary
- Updated: Issue file reorganized to reference migration guide instead of duplicating content
  - Validation section now summarizes findings with link to detailed analysis
  - Plan section references migration guide for implementation details
  - Maintains issue focus on WHAT/WHY, guide documents HOW
- Decision: Sync from ws://localhost:7334 for complete coverage (1,244 repos vs 602)
- Next: Continue with Phase 2 validation using helper scripts from migration guide

### 2026-01-19 [Session 15:00] - REVISED MIGRATION PLAN: Archive-Based Approach ✅
- **MAJOR PIVOT:** Discovered critical data integrity issue that changes migration approach
- **Problem Identified:**
  - ngit-grasp only serves state events if ALL referenced git data is present
  - ngit-relay serves state events regardless of missing git data
  - Risk: Repos with missing data would lose main/master branch visibility after migration
  - ngit-grasp fetches git data from clone URLs, but filters out its own domain
- **Solution: Archive-Based Migration**
  - Deploy archive instance on VPS that syncs from relay.ngit.dev
  - Archive uses different domain (archive.internal) so doesn't filter relay.ngit.dev URLs
  - Archive fetches all git data from relay.ngit.dev during sync
  - Test instance syncs from archive to validate before production cutover
- **Disk Space Assessment:**
  - VPS has 49GB total, 21GB available (56% used)
  - Current ngit services: 17.2GB (relay.ngit.dev: 7.5GB, gitnostr.com: 6.7GB, ngit.danconwaydev.com: 77MB)
  - Old ngit-relay data: 2.9GB (deleted to free space)
  - Estimated archive size: 10-12GB (with doxbox blacklist)
  - After cleanup: 24GB available - sufficient for archive approach ✅
- **Archive Service Deployed:**
  - Service: `ngit-grasp-relay-ngit-archive` (running on VPS)
  - Domain: `archive.internal`
  - Mode: `archiveAll = true` with `repositoryBlacklist = ["doxbox"]`
  - Sync from: `wss://relay.ngit.dev`
  - Port: `7443` (localhost only)
  - Data dir: `/persistent/grasp/sync-archive`
  - Status: ✅ **ACTIVE and syncing** (330MB synced as of 17:30 CET)
- **Issues Created/Fixed:**
  - Issue b454: NixOS module should auto-create data directories in ExecStartPre
  - Temporary fix: Added tmpfiles rules to create directories before service starts
  - Permanent fix: Will be implemented in ngit-grasp module (tracked in b454)
- **Backup Created:**
  - Full backup of relay.ngit.dev data (7.5GB) created on VPS
  - Git repos: 7.1GB compressed
  - Relay DB: 18MB compressed
  - User downloading backup to laptop (slow connection, ~2 hours)
- **Updated Migration Plan:**
  1. ✅ **Phase 1: Archive Sync** (IN PROGRESS)
     - Archive service deployed and syncing from relay.ngit.dev
     - Estimated completion: 10-12GB sync (currently 330MB)
     - Monitor sync progress and final size
  2. **Phase 2: Validation** (NEXT)
     - Once archive sync complete, assess actual disk usage
     - Verify all repos synced successfully
     - Check for repos stuck in purgatory (missing git data)
     - Compare repo list: archive vs relay.ngit.dev
  3. **Phase 3: Test Instance** (PENDING)
     - Deploy test instance on laptop (not VPS - saves 15GB)
     - Test instance syncs from VPS archive
     - Validate all repos accessible
     - Verify no repos stuck in purgatory
  4. **Phase 4: Production Cutover** (PENDING)
     - Only after test instance validation passes
     - Stop ngit-relay, deploy ngit-grasp
     - Point to VPS archive or direct sync
     - Monitor intensively for 48 hours
- **Key Decisions:**
  - Test locally (laptop) instead of VPS to save disk space
  - Use doxbox blacklist to reduce archive size
  - Blank start archive on VPS (no upload needed - better bandwidth)
  - Archive fetches git data from relay.ngit.dev during sync
- **Timeline Revised:**
  - Archive sync: 1-2 days (depends on bandwidth and repo count)
  - Validation: 4-6 hours
  - Test instance: 1 day
  - Production cutover: 2-4 hours
  - Post-migration monitoring: 48 hours
  - **Total: 3-5 days** (from archive sync complete)
- **Next Steps:**
  1. Monitor archive sync progress: `ssh vps1 'du -sh /persistent/grasp/sync-archive'`
  2. Wait for archive sync to complete (10-12GB target)
  3. Validate archive completeness (repo count, git refs)
  4. Assess if enough disk space for test instance on VPS or test locally
  5. Proceed with Phase 2 validation
- **Status:** Archive syncing ✅, Migration approach validated ✅, Waiting for sync completion ⏳

## Validation Results

### Analysis Summary (2026-01-20)

Comprehensive Nostr event analysis completed comparing relay.ngit.dev and localhost archive:

**Event Counts:**
- **ws://localhost:7334** (archive relay): 2,157 events, 1,244 unique repos
- **relay.ngit.dev**: 690 events, 602 unique repos

**Sync Accuracy: 99.86%**
- Missing from localhost: 1 event (einundzwanzighh-gesundesgeld)
- Outdated events: 0
- Shared repos: 548

**Key Findings:**
- localhost:7334 is the **authoritative source** with complete history (Jan 2024 - Jan 2026)
- relay.ngit.dev is a filtered subset (May 2025 - Jan 2026)
- boldwallet.git found on localhost:7334 ✅, NOT on relay.ngit.dev ❌
- 696 repos exist ONLY on localhost (including boldwallet)
- 54 repos exist ONLY on relay.ngit.dev (newer additions)

**Recommendation:** Sync from ws://localhost:7334 for complete coverage.

**Detailed Analysis:** See [work/analysis/](../../work/analysis/) for complete event analysis and missing repository details.

**Migration Guide:** See [ngit-relay Migration Guide](../../docs/how-to/ngit-relay-migration-guide.md) for detailed validation steps, helper scripts, and troubleshooting procedures.

## Notes

- **Related issue:** 573b-production-timeout-diagnosis.md (timeout investigation that led to this)
- **Related issue:** 1ce6-purgatory-retry-mechanism.md (purgatory retry failures discovered during this migration)
- **Critical requirement:** Must collect baseline BEFORE migration for objective comparison
- **Success criteria:** Performance equal or better than ngit-relay baseline
  - No increase in CPU/memory usage
  - No new timeout patterns
  - Connection handling remains stable
  - Git operations perform equally well
- **Rollback plan:** Keep ngit-relay container for 2 weeks post-migration
  - Can switch back immediately if issues arise
  - Only remove after validation period passes
- **nginx metrics:** Implementation ready in ngit-relay repo
  - Provides connection counts, request rates, upstream health
  - Essential for before/after comparison
- **Current status:**
  - relay.ngit.dev: Running ngit-relay v0.0.5 with 2000 conn/IP rate limit
  - ngit.danconwaydev.com: Running ngit-grasp with 4096 max connections
  - Both services stable after Phase 1 improvements
- **Migration benefits:**
  - Single codebase to maintain and improve
  - Can focus logging/observability work on ngit-grasp
  - Proven performance (ngit.danconwaydev.com is stable)
  - Simplified deployment and configuration

### Accelerated Migration: Configuration Template

**ngit-grasp service config for relay.ngit.dev:**

```nix
# services/ngit-grasp.nix
{ inputs, ... }:

{
  imports = [ inputs.ngit-grasp.nixosModules.default ];

  services.ngit-grasp.relay-ngit-dev = {
    enable = true;
    domain = "relay.ngit.dev";
    
    # Network - bind to localhost, nginx handles TLS
    bindAddress = "127.0.0.1";
    port = 8082;  # Or whatever port ngit-relay was using
    
    # Storage - reuse existing data directory if possible
    dataDir = "/persistent/ngit-relay";  # Or new dir: /persistent/ngit-grasp
    
    # Identity - copy from ngit-relay config
    relayName = "relay.ngit.dev";
    relayDescription = "GRASP relay for git+nostr";
    relayOwnerNsecFile = "/persistent/ngit-relay/relay-owner.nsec";
    
    # Sync - bootstrap from ngit.danconwaydev.com
    syncBootstrapRelayUrl = "wss://ngit.danconwaydev.com";
    
    # Metrics - enable for monitoring
    metricsEnabled = true;
    
    # Logging - start with info, increase to debug if issues
    logLevel = "info";
  };

  # Nginx/Caddy reverse proxy (existing config, just update upstream)
  # Change upstream from ngit-relay container to 127.0.0.1:8082
}
```

**Key decisions:**
1. **Data directory:** Reuse `/persistent/ngit-relay` if compatible, or create new `/persistent/ngit-grasp`
2. **Port:** Use same port as ngit-relay container was using (check nginx config)
3. **Bootstrap relay:** Use ngit.danconwaydev.com (known good ngit-grasp instance)
4. **Metrics:** Enable from day 1 for comparison

### Accelerated Migration: Risk Assessment

**Risks with fast migration:**

| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Insufficient baseline data | Medium | Medium | Collect 48-72h minimum; focus on peak hours |
| Performance regression | Low | High | ngit-grasp proven on ngit.danconwaydev.com; instant rollback available |
| Data compatibility issues | Low | Medium | Both use same event format; test with sample data first |
| Configuration errors | Medium | Low | Pre-validate config; test build before migration |
| User disruption | Low | Medium | Migrate during off-peak; monitor intensively first 48h |
| Rollback complications | Low | High | Keep ngit-relay container; document exact rollback steps |

**Why acceptable to move fast:**
1. ✅ **Proven implementation:** ngit-grasp runs ngit.danconwaydev.com successfully
2. ✅ **Easy rollback:** Container-based deployment allows 5-min rollback
3. ✅ **Low user impact:** relay.ngit.dev is not mission-critical (dev/test relay)
4. ✅ **Stable baseline:** ngit-relay v0.0.5 is stable after timeout fixes
5. ✅ **Intensive monitoring:** 48h of close monitoring catches issues early

**What we're trading off:**
- ❌ Detailed performance comparison (1-2 weeks baseline → 48-72h baseline)
- ❌ Comprehensive testing (full test suite → basic functionality tests)
- ❌ Gradual rollout (immediate cutover → no canary deployment)

**Acceptable because:**
- This is a development/test relay, not production-critical
- We have a proven implementation (ngit.danconwaydev.com)
- Rollback is trivial (restart container)
- Benefits (consolidated codebase) outweigh risks

### Accelerated Migration: Monitoring Checklist

**Pre-migration baseline (48-72h):**
```bash
# Capture these metrics before migration
echo "=== CPU Baseline ===" > baseline.txt
top -b -n 1 | grep ngit-relay >> baseline.txt

echo "=== Memory Baseline ===" >> baseline.txt
docker stats ngit-relay --no-stream >> baseline.txt

echo "=== Connection Baseline ===" >> baseline.txt
ss -s >> baseline.txt

echo "=== Error Baseline ===" >> baseline.txt
journalctl -u ngit-relay --since "24 hours ago" | grep -i error | wc -l >> baseline.txt
```

**Post-migration monitoring (48h intensive):**
```bash
# Run every 15 min (hour 1), hourly (hours 2-6), then decreasing frequency

# 1. Service health
systemctl status ngit-grasp-relay-ngit-dev

# 2. Resource usage
top -b -n 1 | grep ngit-grasp

# 3. Connection count
ss -s

# 4. Error check
journalctl -u ngit-grasp-relay-ngit-dev --since "15 minutes ago" | grep -i error

# 5. Metrics endpoint
curl http://localhost:8082/metrics | grep -E "ngit_(connections|events|git)"

# 6. Functional test
# WebSocket: Use websocat or browser
# Git: git ls-remote https://relay.ngit.dev/<npub>/<repo>.git
```

**Rollback trigger conditions:**
- 🚨 Service crashes or fails to restart
- 🚨 Error rate >10x baseline
- 🚨 Memory usage >4x baseline (sustained >30 min)
- 🚨 CPU usage >90% sustained >15 min
- 🚨 Git operations failing
- 🚨 WebSocket connections rejected

**Success indicators:**
- ✅ Service uptime >48h continuous
- ✅ Error rate ≤ baseline
- ✅ Resource usage ≤ 2x baseline
- ✅ All functional tests passing
- ✅ No user complaints