ngit-grasp/docs/reference/test-strategy.md

Test Strategy for ngit-grasp

Overview

This document describes the testing strategy for ngit-grasp, including the grasp-audit reusable compliance testing tool and the integration tests in the main repository.

Testing Philosophy

1. Specification-Driven: Tests mirror GRASP-01 protocol structure exactly
2. Compliance-First: Every requirement in the spec has a corresponding test
3. Reusable: The grasp-audit tool can validate any GRASP implementation
4. Isolated: Each test runs with its own relay instance via TestRelay
5. Clear Failures: Test failures cite exact spec requirements

Test Pyramid

                    ╱╲
                   ╱  ╲
                  ╱ E2E╲              ~ 10%  End-to-end with real Git
                 ╱──────╲
                ╱        ╲
               ╱Compliance╲           ~ 30%  GRASP-01 spec validation
              ╱────────────╲                  (grasp-audit)
             ╱              ╲
            ╱  Integration   ╲        ~ 30%  Component interaction
           ╱──────────────────╲              (tests/)
          ╱                    ╲
         ╱   Unit Tests         ╲     ~ 30%  Individual functions
        ╱────────────────────────╲           (src/**/tests)

Project Structure

Actual Test Layout

ngit-grasp/
├── tests/                          # Integration tests for ngit-grasp
│   ├── common/
│   │   ├── mod.rs                  # Test utilities module
│   │   └── relay.rs                # TestRelay fixture
│   ├── nip01_compliance.rs         # NIP-01 relay compliance
│   ├── nip11_document.rs           # NIP-11 document tests
│   ├── nip34_announcements.rs      # Repository announcement tests
│   ├── repository_creation.rs      # Git repo creation tests
│   ├── push_authorization.rs       # Push validation tests
│   ├── cors.rs                     # CORS header tests
│   └── git_clone.rs                # Git clone tests
│
└── grasp-audit/                    # Reusable GRASP compliance tool
    ├── Cargo.toml
    ├── flake.nix
    └── src/
        ├── lib.rs                  # Public API
        ├── client.rs               # AuditClient
        ├── audit.rs                # AuditConfig, cleanup tags
        ├── fixtures.rs             # Test fixtures
        └── specs/
            └── grasp01/            # GRASP-01 specification tests
                ├── mod.rs          # Module exports
                ├── nip01_smoke.rs  # NIP-01 smoke tests
                ├── nip11_document.rs
                ├── event_acceptance_policy.rs
                ├── cors.rs
                ├── git_clone.rs
                ├── push_authorization.rs
                ├── repository_creation.rs
                └── spec_requirements.rs  # Requirement definitions

Integration Tests (tests/)

TestRelay Fixture

The TestRelay fixture provides automatic relay lifecycle management:

// From tests/common/relay.rs

/// Test relay fixture that manages relay lifecycle
///
/// Automatically starts and stops the ngit-grasp relay for testing.
/// Uses a random port to avoid conflicts and cleans up created repositories.
pub struct TestRelay {
    process: Child,
    url: String,
    port: u16,
}

impl TestRelay {
    /// Start a test relay instance
    pub async fn start() -> Self { ... }
    
    /// Get the relay WebSocket URL
    pub fn url(&self) -> &str { ... }
    
    /// Get the relay domain (host:port)
    pub fn domain(&self) -> String { ... }
    
    /// Stop the relay
    pub async fn stop(mut self) { ... }
}

Using TestRelay in Integration Tests

From tests/nip01_compliance.rs:

use common::TestRelay;
use grasp_audit::*;

/// Macro to generate isolated integration tests
macro_rules! isolated_test {
    ($test_name:ident) => {
        #[tokio::test]
        async fn $test_name() {
            let relay = TestRelay::start().await;
            let config = AuditConfig::isolated();
            let client = AuditClient::new(relay.url(), config)
                .await
                .expect("Failed to create audit client");

            let result = specs::Nip01SmokeTests::$test_name(&client).await;

            relay.stop().await;

            assert!(
                result.passed,
                "{} failed: {}",
                stringify!($test_name),
                result.error.as_deref().unwrap_or("unknown error")
            );
        }
    };
}

// Generate isolated tests for all NIP-01 smoke tests
isolated_test!(test_websocket_connection);
isolated_test!(test_send_receive_event);
isolated_test!(test_create_subscription);

Running Integration Tests

# Run all integration tests
cargo test --test '*'

# Run specific test file
cargo test --test nip01_compliance

# Run with output
cargo test --test nip01_compliance -- --nocapture

GRASP Audit Tool (grasp-audit/)

Purpose

The grasp-audit tool is a reusable GRASP compliance testing library that can:

• Test ngit-grasp for self-validation
• Test any other GRASP implementation (like ngit-relay)
• Run in CI/CD for continuous compliance verification
• Generate compliance reports

Test Suites

From grasp-audit/src/specs/grasp01/mod.rs:

Suite	Description	Requirements
Nip01SmokeTests	Basic NIP-01 relay functionality	WebSocket only
Nip11DocumentTests	NIP-11 relay information document	WebSocket only
EventAcceptancePolicyTests	Event acceptance rules	WebSocket only
CorsTests	CORS headers on Git HTTP endpoints	git-data-dir
GitCloneTests	Git clone operations	git-data-dir
PushAuthorizationTests	Push authorization	git-data-dir
RepositoryCreationTests	Repository creation	git-data-dir

Spec Requirements Database

From grasp-audit/src/specs/grasp01/spec_requirements.rs:

pub struct SpecRequirement {
    pub id: &'static str,           // e.g., "GRASP-01:L9"
    pub section: &'static str,      // e.g., "Nostr Relay"
    pub level: RequirementLevel,    // MUST, SHOULD, MAY
    pub text: &'static str,         // Exact text from spec
    pub line: u32,                  // Line number in spec
}

pub enum RequirementLevel {
    Must,
    Should,
    May,
}

Automatic Cleanup Tags

All audit events include cleanup tags for production safety (from grasp-audit/src/audit.rs):

// Automatically added to EVERY audit event:
["t", "grasp-audit-test-event"]              // Marker
["t", "audit-{run_id}"]                      // Run isolation  
["t", "audit-cleanup-after-{unix_timestamp}"] // Cleanup time

Running grasp-audit

Testing the reference implementation (ngit-relay):

# Use test-ngit-relay.sh for automated relay management
cd grasp-audit && nix develop -c bash test-ngit-relay.sh --mode test

# Or manually:
docker run --rm -p 18081:8081 ghcr.io/danconwaydev/ngit-relay:latest
cd grasp-audit
RELAY_URL="ws://localhost:18081" nix develop -c cargo test --lib -- --ignored --nocapture

Testing ngit-grasp (the main project):

# Integration tests use TestRelay fixture - just run:
cargo test --test '*'

Test Patterns

Isolated Test Pattern

Each test runs with its own fresh relay instance:

#[tokio::test]
async fn test_something() {
    // Start fresh relay
    let relay = TestRelay::start().await;
    
    // Run test
    let client = AuditClient::new(relay.url(), AuditConfig::isolated()).await?;
    // ... test logic ...
    
    // Cleanup
    relay.stop().await;
}

Macro-Based Test Generation

For test suites that follow the same pattern, use macros:

macro_rules! isolated_test {
    ($test_name:ident) => {
        #[tokio::test]
        async fn $test_name() {
            let relay = TestRelay::start().await;
            // ... standard setup and teardown ...
        }
    };
}

isolated_test!(test_websocket_connection);
isolated_test!(test_send_receive_event);

Coverage Targets

Test Type	Coverage Target
Unit Tests	>80% line coverage of src/
Integration Tests	All critical user paths
GRASP-01 Compliance	100% of MUST requirements

CI/CD Integration

Running All Tests

# Unit tests (fast, no external dependencies)
cargo test --lib

# Integration tests (requires relay binary built)
cargo build --release
cargo test --test '*'

# Compliance tests against ngit-relay reference
cd grasp-audit && nix develop -c bash test-ngit-relay.sh --mode test

Summary

What	Where	Purpose
Unit tests	src/**/tests modules	Test individual functions
Integration tests	tests/*.rs	Test ngit-grasp as a whole
TestRelay fixture	tests/common/relay.rs	Manage relay lifecycle
GRASP audit library	grasp-audit/	Reusable compliance testing
GRASP-01 specs	grasp-audit/src/specs/grasp01/	Spec requirement tests

Related Documentation

• Architecture - System design
• GRASP-01 Implementation Learnings - Patterns and lessons
• GRASP Audit Learnings - Audit tool patterns