mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-07-22 07:48:27 +00:00
Merge pull request #3626 from vitorpamplona/claude/graperank-algorithm-improvements-oco5ue
Add follower crawl and trusted-follower metrics to GrapeRank
This commit is contained in:
@@ -389,8 +389,9 @@ HTTP endpoint. Reuses quartz's `Nip86Client` and the shared `Nip86Retriever`
|
||||
| `amy profile show [USER]` | Print kind:0 metadata. USER accepts npub/nprofile/hex/NIP-05; defaults to self. |
|
||||
| `amy profile edit --name … --about … --picture URL …` | Patch and re-publish your kind:0. |
|
||||
| `amy follow USER` / `amy unfollow USER` | Add/remove USER from your kind:3 contact list (fetches the freshest list first). |
|
||||
| `amy graperank [OBSERVER] [--offline] [--min-rank N]` | Crawl + score: compute GrapeRank web-of-trust scores (0..1) over the follow/mute/report graph, then persist the result. Exhaustively crawls each user's kind:10002 outbox for their latest kind:3/10000/1984 until every discovered user is checked (no user cap), dropping reports the author retracted via NIP-09. **Every score run persists its result locally**: the ranks (cutoff `--min-rank`, default 2) are reconciled into the shared store as NIP-85 kind:30382 cards signed by a per-observer **service key** — changed ranks re-signed, unchanged skipped (no event-id churn), dropped targets retracted (kind:5). `--offline` skips the crawl. |
|
||||
| `amy graperank [OBSERVER] [--offline] [--min-rank N]` | Crawl + score: compute GrapeRank web-of-trust scores (0..1) over the follow/mute/report graph, then persist the result. Exhaustively crawls each user's kind:10002 outbox for their latest kind:3/10000/1984 until every discovered user is checked (no user cap), dropping reports the author retracted via NIP-09. **Every score run persists its result locally**: the cards (rank cutoff `--min-rank`, default 2) are reconciled into the shared store as NIP-85 kind:30382 cards signed by a per-observer **service key** — each carries `rank`, `followers` (trusted-follower count, cutoff `--followers-threshold`, default 0.02) and `hops` (follow distance from the observer); changed cards re-signed, unchanged skipped (no event-id churn), dropped targets retracted (kind:5). `--offline` skips the crawl. |
|
||||
| `amy graperank crawl [OBSERVER] [--max-hops N] [--no-preconnect]` | Pipeline stage 1 — network only: crawl the follow/mute/report graph (kind 3/10000/1984/10002) into the local store, no scoring. Idempotent and cumulative: run it a few times to load everything, then `score`. |
|
||||
| `amy graperank followers [OBSERVER] [--relay URL[,URL…]]` | The **reverse** crawl: find every user who *follows* the observer by asking as many relays as possible for kind:3 lists that `#p`-tag them (paged past each relay's cap). The outbox model can't find followers — you don't know one exists until you've seen their list — so this casts a wide net over the whole relay universe (reachability-cache live set + every kind:10002/30166 relay in the store + index/aggregator relays), skipping proven-dead relays. Persists each follower's list, enriching the graph a later `score` builds. Idempotent and cumulative. |
|
||||
| `amy graperank score [OBSERVER]` | Pipeline stage 2 — local only: score from the store and persist the cards (identical to bare `--offline`; same scoring flags). No network, so re-run with different `--rigor`/`--attenuation`/`--min-rank` without re-crawling. |
|
||||
| `amy graperank publish [OBSERVER] [--relay URL[,URL…]]` | Pipeline stage 3 — transport only: make the operator relay(s) converge to the locally persisted card set — one NIP-77 up-only reconcile per relay over the service key's kind:30382 + kind:5 (nothing is re-scored or re-signed; a relay that can't reconcile gets the full set published instead). Also refreshes the observer's kind:10040 pointer when we hold their key. |
|
||||
| `amy graperank rank USER [--provider PUBKEY] [--refresh]` | The consumer side: read the kind:30382 cards about USER — one rank per provider, newest card each. Local store first; `--refresh` (or a miss) drains the operator relays, the relays your kind:10040 declares, and the bootstrap set. |
|
||||
@@ -420,12 +421,16 @@ means re-signing **replaces** a target's card instead of orphaning it — and
|
||||
losing everything but the master seed still re-derives every key.
|
||||
|
||||
**Every score run persists its cards.** After scoring, Amy reconciles the result
|
||||
into the local store: new or changed ranks (≥ `--min-rank`, default 2) are
|
||||
signed; unchanged ranks are skipped (no new event id); and any card whose target
|
||||
into the local store: new or changed cards (rank ≥ `--min-rank`, default 2) are
|
||||
signed; unchanged cards are skipped (no new event id); and any card whose target
|
||||
dropped out of the graph or fell below the cutoff is **retracted** with a kind:5
|
||||
(the store applies it; the tombstone is kept). The local store is the source of
|
||||
truth — `graperank rank USER` reads it offline, and `graperank publish` mirrors
|
||||
it out:
|
||||
(the store applies it; the tombstone is kept). Each card carries three public
|
||||
tags: `rank` (`round(score*100)`), `followers` — the number of the target's
|
||||
followers whose own score clears `--followers-threshold` (default 0.02, matching
|
||||
Brainstorm's trusted-follower cutoff) — and `hops`, the shortest follow-graph
|
||||
distance from the observer (1 = a direct follow). A change to *any* of the three
|
||||
re-signs the card. The local store is the source of truth — `graperank rank USER`
|
||||
reads it offline, and `graperank publish` mirrors it out:
|
||||
|
||||
```bash
|
||||
amy graperank operator relay wss://relay.example.com # where all cards live
|
||||
|
||||
@@ -26,6 +26,7 @@ import com.vitorpamplona.amethyst.cli.DataDir
|
||||
import com.vitorpamplona.amethyst.cli.Output
|
||||
import com.vitorpamplona.amethyst.commons.defaults.Constants
|
||||
import com.vitorpamplona.amethyst.commons.defaults.DefaultIndexerRelayList
|
||||
import com.vitorpamplona.quartz.experimental.graperank.FollowerCrawler
|
||||
import com.vitorpamplona.quartz.experimental.graperank.GrapeRank
|
||||
import com.vitorpamplona.quartz.experimental.graperank.GrapeRankCrawler
|
||||
import com.vitorpamplona.quartz.experimental.graperank.GrapeRankParams
|
||||
@@ -146,6 +147,10 @@ object GrapeRankCommand {
|
||||
|
||||
private const val PROBE_TIMEOUT_MS = 2000
|
||||
|
||||
// Default score cutoff for counting a follower as "trusted" (the `followers`
|
||||
// tag). Matches NosFabrica Brainstorm's verifiedFollowersInfluenceCutoff.
|
||||
private const val DEFAULT_FOLLOWERS_THRESHOLD = 0.02
|
||||
|
||||
// args.bool on a mistyped flag silently returns false, so the one flag read from
|
||||
// three different functions goes through a compile-time-checked name.
|
||||
private const val NO_REACHABILITY_CACHE_FLAG = "no-reachability-cache"
|
||||
@@ -209,6 +214,7 @@ object GrapeRankCommand {
|
||||
"providers" -> providers(dataDir, tail.drop(1).toTypedArray())
|
||||
"operator" -> operator(dataDir, tail.drop(1).toTypedArray())
|
||||
"crawl" -> crawl(dataDir, tail.drop(1).toTypedArray())
|
||||
"followers" -> followers(dataDir, tail.drop(1).toTypedArray())
|
||||
"status" -> status(dataDir)
|
||||
// The relay census outgrew graperank (it feeds the shared NIP-66
|
||||
// reachability cache every command reads) and moved to `amy relay
|
||||
@@ -248,6 +254,10 @@ object GrapeRankCommand {
|
||||
// retracted. Rank is round(score*100), so 2 drops the ~0.015-and-below
|
||||
// barely-trusted tail.
|
||||
val minRank = args.intFlag("min-rank", 2)
|
||||
// A "trusted follower" (the `followers` tag) is a follower whose own score is
|
||||
// at or above this. Mirrors Brainstorm's verifiedFollowersInfluenceCutoff
|
||||
// (0.02), which is the same 0.02 score == rank 2 line as the default min-rank.
|
||||
val followersThreshold = args.flag("followers-threshold")?.toDoubleOrNull() ?: DEFAULT_FOLLOWERS_THRESHOLD
|
||||
|
||||
val params =
|
||||
GrapeRankParams(
|
||||
@@ -255,8 +265,15 @@ object GrapeRankCommand {
|
||||
rigor = args.flag("rigor")?.toDoubleOrNull() ?: GrapeRankParams().rigor,
|
||||
)
|
||||
|
||||
Context.open(dataDir).use { ctx ->
|
||||
// Crawl never signs and score signs cards with the machine-level operator
|
||||
// key (`~/.amy/operator/`, independent of any account), so neither needs a
|
||||
// personal account — run anonymously when there is none, requiring an
|
||||
// explicit observer since there's no logged-in user to default to.
|
||||
Context.openOrAnonymous(dataDir).use { ctx ->
|
||||
ctx.prepare()
|
||||
if (ctx.anonymous && observerArg == null) {
|
||||
return Output.error("bad_args", "no account — pass an OBSERVER (npub / hex / nprofile / NIP-05)")
|
||||
}
|
||||
val observer = observerArg?.let { ctx.requireUserHex(it) } ?: ctx.identity.pubKeyHex
|
||||
|
||||
// Contact lists stream straight into a compact int-CSR structure as the
|
||||
@@ -327,6 +344,14 @@ object GrapeRankCommand {
|
||||
val scoringMs = (System.nanoTime() - scoreStart) / 1_000_000
|
||||
System.err.println("[graperank] scored ${rankedIds.size} users in $scoringMs ms")
|
||||
|
||||
// Two derived per-user metrics persisted alongside the rank on each card:
|
||||
// - trusted-follower count: how many of a user's followers score at or
|
||||
// above the cutoff (Brainstorm's trustedFollowers).
|
||||
// - hops: shortest follow-graph distance from the observer (1 = direct
|
||||
// follow). Both are pure functions over the same graph + scores.
|
||||
val followerCounts = graph.trustedFollowerCounts(scores, followersThreshold)
|
||||
val hops = graph.hopsFrom(observer)
|
||||
|
||||
val hopHistogram = crawlStats?.hopHistogram.orEmpty()
|
||||
val result =
|
||||
linkedMapOf<String, Any?>(
|
||||
@@ -352,9 +377,16 @@ object GrapeRankCommand {
|
||||
"graph_build_ms" to buildMs,
|
||||
"scoring_ms" to scoringMs,
|
||||
"scoring_sweeps" to sweeps,
|
||||
"followers_threshold" to followersThreshold,
|
||||
"scores" to
|
||||
rankedIds.take(limit).map {
|
||||
mapOf("pubkey" to graph.pubkeyOf(it), "score" to scores[it], "rank" to rankOf(scores[it]))
|
||||
mapOf(
|
||||
"pubkey" to graph.pubkeyOf(it),
|
||||
"score" to scores[it],
|
||||
"rank" to rankOf(scores[it]),
|
||||
"followers" to followerCounts[it],
|
||||
"hops" to hops[it],
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
@@ -372,7 +404,16 @@ object GrapeRankCommand {
|
||||
val desiredCards =
|
||||
rankedIds
|
||||
.filter { rankOf(scores[it]) >= minRank }
|
||||
.map { graph.pubkeyOf(it) to rankOf(scores[it]) }
|
||||
.map { id ->
|
||||
GrapeRankPublisher.ScoredCard(
|
||||
target = graph.pubkeyOf(id),
|
||||
rank = rankOf(scores[id]),
|
||||
followers = followerCounts[id],
|
||||
// A scored user always has a follow path from the observer,
|
||||
// so hops is ≥ 1; guard the UNREACHABLE sentinel just in case.
|
||||
hops = hops[id].takeIf { it >= 1 },
|
||||
)
|
||||
}
|
||||
|
||||
val cardsStart = System.nanoTime()
|
||||
val local =
|
||||
@@ -524,8 +565,13 @@ object GrapeRankCommand {
|
||||
): Int {
|
||||
val args = Args(rest)
|
||||
val observerArg = args.positionalOrNull(0)
|
||||
Context.open(dataDir).use { ctx ->
|
||||
// Crawl never signs, so no account is needed — run anonymously when there is
|
||||
// none, requiring an explicit observer (no logged-in user to default to).
|
||||
Context.openOrAnonymous(dataDir).use { ctx ->
|
||||
ctx.prepare()
|
||||
if (ctx.anonymous && observerArg == null) {
|
||||
return Output.error("bad_args", "no account — pass an OBSERVER (npub / hex / nprofile / NIP-05)")
|
||||
}
|
||||
val observer = observerArg?.let { ctx.requireUserHex(it) } ?: ctx.identity.pubKeyHex
|
||||
// Persist-only crawl: no in-memory graph (null builder); every event
|
||||
// still lands in the store for a later `score`.
|
||||
@@ -554,6 +600,131 @@ object GrapeRankCommand {
|
||||
return 0
|
||||
}
|
||||
|
||||
/**
|
||||
* `amy graperank followers [OBSERVER] [flags]` — the reverse crawl: find every
|
||||
* user who FOLLOWS the observer and persist their kind:3 contact lists.
|
||||
*
|
||||
* The outbox model (what `crawl` uses) walks follows *outward* and can't find
|
||||
* followers — you don't know a follower exists until you've seen their list, so
|
||||
* you can't route to their outbox first. This casts a wide net instead: it asks
|
||||
* as many relays as possible for kind:3 events that `#p`-tag the observer, paging
|
||||
* each relay past its per-REQ cap. The relay universe is "all possible relays":
|
||||
* the reachability-cache live set + every kind:10002 read/write relay and every
|
||||
* kind:30166 monitored relay in the store + the index/aggregator relays that hold
|
||||
* reverse-follow data for the network. Proven-dead relays are skipped
|
||||
* (`--no-reachability-cache` to query them anyway).
|
||||
*
|
||||
* Each follower's list lands in the store, so it also enriches the graph a later
|
||||
* `graperank score` builds — every follower becomes a FOLLOW edge into the
|
||||
* observer. Idempotent and cumulative; run it a few times for completeness.
|
||||
*
|
||||
* Flags: `--relay URL[,URL…]` (query only these instead of the whole universe),
|
||||
* `--max N` (cap followers pulled per relay; default: pull every follower each
|
||||
* relay holds), `--timeout SECS` (per-page EOSE watchdog, default 15),
|
||||
* `--relay-concurrency N` (relays paged at once, default 16), `--insert-batch N`
|
||||
* (events per store commit, default 500).
|
||||
*/
|
||||
private suspend fun followers(
|
||||
dataDir: DataDir,
|
||||
rest: Array<String>,
|
||||
): Int {
|
||||
val args = Args(rest)
|
||||
val observerArg = args.positionalOrNull(0)
|
||||
val relayArg = args.flag("relay")
|
||||
val relayConcurrency = args.intFlag("relay-concurrency", args.intFlag("concurrency", 16))
|
||||
|
||||
// Read-only + never signs, so it runs anonymously — but then an OBSERVER
|
||||
// positional is required (there's no account to default to).
|
||||
Context.openOrAnonymous(dataDir).use { ctx ->
|
||||
ctx.prepare()
|
||||
if (ctx.anonymous && observerArg == null) {
|
||||
return Output.error("bad_args", "usage: amy graperank followers OBSERVER [flags] (no account — pass an observer)")
|
||||
}
|
||||
val observer = observerArg?.let { ctx.requireUserHex(it) } ?: ctx.identity.pubKeyHex
|
||||
|
||||
val relays =
|
||||
relayArg
|
||||
?.split(",")
|
||||
?.mapNotNull { RelayUrlNormalizer.normalizeOrNull(it.trim()) }
|
||||
?.toSet()
|
||||
?.takeIf { it.isNotEmpty() }
|
||||
?: allKnownRelays(ctx, args)
|
||||
|
||||
if (relays.isEmpty()) {
|
||||
return Output.error("no_relays", "no relays to query — run `amy relay probe` / `amy graperank crawl` first, or pass --relay")
|
||||
}
|
||||
|
||||
System.err.println("[followers] querying ${relays.size} relays for kind:3 #p=${observer.take(8)}…")
|
||||
val crawler =
|
||||
FollowerCrawler(
|
||||
client = ctx.client,
|
||||
store = ctx.store,
|
||||
config =
|
||||
FollowerCrawler.Config(
|
||||
relays = relays,
|
||||
// Default null → pull EVERY follower each relay holds; --max
|
||||
// N caps the total per relay for a quick spot check.
|
||||
maxPerRelay = args.flag("max")?.toIntOrNull(),
|
||||
timeoutMs = args.longFlag("timeout", 15L) * 1000,
|
||||
maxConcurrentRelays = relayConcurrency,
|
||||
insertBatchSize = args.intFlag("insert-batch", 500),
|
||||
),
|
||||
log = { System.err.println(it) },
|
||||
)
|
||||
val stats = crawler.crawl(observer)
|
||||
|
||||
Output.emit(
|
||||
linkedMapOf<String, Any?>(
|
||||
"observer" to observer,
|
||||
"relays_queried" to stats.relaysQueried,
|
||||
"relays_answered" to stats.relaysAnswered,
|
||||
"followers_found" to stats.followersFound,
|
||||
"events_stored" to stats.eventsStored,
|
||||
"download_ms" to stats.downloadMs,
|
||||
),
|
||||
)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
/**
|
||||
* "All possible relays" for the follower crawl: the reachability-cache live set,
|
||||
* every kind:10002 read/write relay and every kind:30166 monitored relay the
|
||||
* store knows, and the index/aggregator relays that hold reverse-follow data —
|
||||
* minus the ones a prior probe proved dead (unless `--no-reachability-cache`).
|
||||
*/
|
||||
private suspend fun allKnownRelays(
|
||||
ctx: Context,
|
||||
args: Args,
|
||||
): Set<NormalizedRelayUrl> {
|
||||
// Read the reachability records (kind:30166) with a throwaway signer, NOT
|
||||
// ctx.reachability — the latter derives the monitor key and would CREATE the
|
||||
// operator master (a passphrase prompt) purely to read a cache. The follower
|
||||
// crawl never signs, so keep it truly account-and-key-free. snapshot() only
|
||||
// reads; the signer is used solely for writes. Same trick as `status`.
|
||||
val reach = RelayReachabilityStore(store = ctx.store, signer = NostrSignerInternal(KeyPair())).snapshot()
|
||||
val relays = LinkedHashSet<NormalizedRelayUrl>()
|
||||
relays.addAll(reach.live)
|
||||
// Every advertised outbox/inbox relay in the store — the homes of the users
|
||||
// whose followers we're hunting are exactly where those followers publish too.
|
||||
for (event in ctx.store.query<Event>(Filter(kinds = listOf(AdvertisedRelayListEvent.KIND)))) {
|
||||
if (event is AdvertisedRelayListEvent) relays.addAll(event.relaysNorm())
|
||||
}
|
||||
// Relays a NIP-66 monitor has ever seen (kind:30166) — the widest census.
|
||||
for (event in ctx.store.query<Event>(Filter(kinds = listOf(RelayDiscoveryEvent.KIND)))) {
|
||||
if (event is RelayDiscoveryEvent) event.relay()?.let { relays.add(it) }
|
||||
}
|
||||
// Aggregators/indexers hold reverse-follow data for users whose own outbox
|
||||
// we've never learned — the biggest completeness lever for followers.
|
||||
relays.addAll(CONTENT_AGGREGATOR_RELAYS)
|
||||
relays.addAll(EXTRA_DISCOVERY_RELAYS)
|
||||
relays.addAll(ctx.bootstrapRelays())
|
||||
relays.addAll(Constants.eventFinderRelays)
|
||||
|
||||
if (!args.bool(NO_REACHABILITY_CACHE_FLAG)) relays.removeAll(reach.dead)
|
||||
return relays
|
||||
}
|
||||
|
||||
/**
|
||||
* `amy graperank status` — read-only inventory of everything a GrapeRank run
|
||||
* depends on, straight from the local store: WoT record counts (the "do I
|
||||
@@ -908,6 +1079,8 @@ object GrapeRankCommand {
|
||||
linkedMapOf<String, Any?>(
|
||||
"provider" to card.pubKey,
|
||||
"rank" to card.rank(),
|
||||
"followers" to card.followerCount(),
|
||||
"hops" to card.hops(),
|
||||
"observer" to providerToObserver[card.pubKey],
|
||||
"created_at" to card.createdAt,
|
||||
"event_id" to card.id,
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.experimental.graperank
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.core.HexKey
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.ParallelEventVerifier
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.fetchAllPagesFromPool
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.nip01Core.store.IEventStore
|
||||
import com.vitorpamplona.quartz.nip01Core.tags.people.isTaggedUser
|
||||
import com.vitorpamplona.quartz.nip02FollowList.ContactListEvent
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.IO
|
||||
import kotlinx.coroutines.channels.Channel
|
||||
import kotlinx.coroutines.coroutineScope
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.coroutines.withContext
|
||||
import kotlin.time.TimeSource
|
||||
|
||||
/**
|
||||
* The reverse of [GrapeRankCrawler]: finds every user who **follows** the observer
|
||||
* by asking as many relays as possible for kind:3 contact lists that `#p`-tag the
|
||||
* observer, and persists each one to [store]. The outbox model can't find these —
|
||||
* you don't know a follower until you've seen their list, so you can't route to
|
||||
* their outbox first — so this casts a wide net across the whole relay universe
|
||||
* ([Config.relays], assembled by the caller from the reachability cache, the
|
||||
* kind:10002 relays in the store, and the index/aggregator relays that hold
|
||||
* reverse-follow data for the network).
|
||||
*
|
||||
* Each discovered kind:3 is a follower's full contact list, so persisting it also
|
||||
* enriches the graph a later `graperank score` builds from the store — the
|
||||
* follower becomes a FOLLOW edge into the observer (and into everyone else it
|
||||
* follows). A popular observer can have far more followers than a relay's per-REQ
|
||||
* cap, so every relay is paged on its own `created_at` cursor via
|
||||
* [fetchAllPagesFromPool].
|
||||
*
|
||||
* Transport-agnostic within quartz: it takes an [INostrClient] and an
|
||||
* [IEventStore]; relay *policy* (which relays make up "all possible relays") is the
|
||||
* caller's, injected through [Config]. Progress is emitted through [log].
|
||||
*/
|
||||
class FollowerCrawler(
|
||||
private val client: INostrClient,
|
||||
private val store: IEventStore,
|
||||
private val config: Config,
|
||||
private val log: (String) -> Unit = {},
|
||||
) {
|
||||
/**
|
||||
* @param relays the relay universe to query — "all possible relays" is the
|
||||
* caller's policy (reachability-cache live set + every kind:10002 relay in the
|
||||
* store + the index/aggregator relays). Empty means nothing to do.
|
||||
* @param maxPerRelay TOTAL cap on followers pulled from each relay, or `null`
|
||||
* (default) to pull **every** follower a relay holds. This is NOT a page size:
|
||||
* [fetchAllPages] treats a filter's `limit` as the total across all pages and
|
||||
* stops paging once it's reached, so a non-null value cuts the crawl short at
|
||||
* that many per relay. Leave it null for completeness; set it only to bound a
|
||||
* spot check. The per-page size is the relay's own default either way.
|
||||
* @param timeoutMs per-page EOSE timeout for a relay before its next page fires.
|
||||
* @param maxConcurrentRelays how many relays page at once (a global fan-out cap).
|
||||
* @param insertBatchSize verified events group-committed per [IEventStore.batchInsert].
|
||||
*/
|
||||
class Config(
|
||||
val relays: Set<NormalizedRelayUrl>,
|
||||
val maxPerRelay: Int? = null,
|
||||
val timeoutMs: Long = 15_000,
|
||||
val maxConcurrentRelays: Int = 16,
|
||||
val insertBatchSize: Int = 500,
|
||||
)
|
||||
|
||||
/** What the follower crawl found. */
|
||||
class Stats(
|
||||
val relaysQueried: Int,
|
||||
/** Relays that returned at least one valid follower list. */
|
||||
val relaysAnswered: Int,
|
||||
/** Distinct followers (kind:3 authors that `#p`-tag the observer). */
|
||||
val followersFound: Int,
|
||||
/** Verified kind:3 events handed to the store (superseded versions included). */
|
||||
val eventsStored: Long,
|
||||
val downloadMs: Long,
|
||||
)
|
||||
|
||||
/**
|
||||
* Find and persist every follower of [observer] across [Config.relays]. Verifies
|
||||
* each event (a relay can lie), keeps only kind:3 that actually `#p`-tag the
|
||||
* observer (a relay can over-return), dedups by event id, and group-commits to
|
||||
* the store. Returns [Stats].
|
||||
*/
|
||||
suspend fun crawl(observer: HexKey): Stats =
|
||||
withContext(Dispatchers.IO) {
|
||||
if (config.relays.isEmpty()) return@withContext Stats(0, 0, 0, 0L, 0L)
|
||||
|
||||
val mark = TimeSource.Monotonic.markNow()
|
||||
// No limit by default → fetchAllPages walks the whole result set page by
|
||||
// page; a non-null maxPerRelay caps the total per relay (see Config).
|
||||
val filter = Filter(kinds = FOLLOW_KINDS, tags = mapOf("p" to listOf(observer)), limit = config.maxPerRelay)
|
||||
val perRelay = config.relays.associateWith { listOf(filter) }
|
||||
|
||||
// These three are read/written ONLY by the verifier's single drain
|
||||
// coroutine (ParallelEventVerifier dispatches onVerified in order from one
|
||||
// coroutine), so plain collections are safe — no concurrent access.
|
||||
val followers = HashSet<HexKey>()
|
||||
val seen = HashSet<HexKey>()
|
||||
val answered = HashSet<NormalizedRelayUrl>()
|
||||
var eventsStored = 0L
|
||||
|
||||
// Verified follower lists cross to a single persister coroutine that can
|
||||
// suspend on the store; onVerified itself must not (it's a plain callback).
|
||||
val toPersist = Channel<Event>(Channel.UNLIMITED)
|
||||
|
||||
coroutineScope {
|
||||
val persister =
|
||||
launch(Dispatchers.IO) {
|
||||
val flushAt = config.insertBatchSize.coerceAtLeast(1)
|
||||
val buffer = ArrayList<Event>(flushAt)
|
||||
|
||||
suspend fun flush() {
|
||||
if (buffer.isEmpty()) return
|
||||
store.batchInsert(buffer)
|
||||
eventsStored += buffer.size
|
||||
buffer.clear()
|
||||
}
|
||||
for (event in toPersist) {
|
||||
buffer.add(event)
|
||||
if (buffer.size >= flushAt) flush()
|
||||
}
|
||||
flush()
|
||||
}
|
||||
|
||||
val verifier =
|
||||
ParallelEventVerifier<NormalizedRelayUrl>(
|
||||
scope = this,
|
||||
onVerified = { event, relay ->
|
||||
// A relay can over-return; keep only genuine followers, and
|
||||
// dedup by id so a list mirrored across relays is stored once.
|
||||
if (event is ContactListEvent && event.isTaggedUser(observer) && seen.add(event.id)) {
|
||||
followers.add(event.pubKey)
|
||||
answered.add(relay)
|
||||
toPersist.trySend(event)
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
client.fetchAllPagesFromPool(
|
||||
filters = perRelay,
|
||||
timeoutMs = config.timeoutMs,
|
||||
maxConcurrentRelays = config.maxConcurrentRelays,
|
||||
onRelayComplete = { relay, total ->
|
||||
if (total > 0) log("[followers] ${relay.url}: $total kind:3 pages drained")
|
||||
},
|
||||
) { event, relay ->
|
||||
// onEvent runs on the relay reader thread and must not suspend;
|
||||
// submit() is a non-suspending channel send that backpressures.
|
||||
if (event.kind == ContactListEvent.KIND) verifier.submit(event, relay)
|
||||
}
|
||||
|
||||
// No more events will arrive: drain the verifier, then the persister.
|
||||
verifier.close()
|
||||
verifier.join()
|
||||
toPersist.close()
|
||||
persister.join()
|
||||
}
|
||||
|
||||
Stats(
|
||||
relaysQueried = config.relays.size,
|
||||
relaysAnswered = answered.size,
|
||||
followersFound = followers.size,
|
||||
eventsStored = eventsStored,
|
||||
downloadMs = mark.elapsedNow().inWholeMilliseconds,
|
||||
)
|
||||
}
|
||||
|
||||
companion object {
|
||||
/** Reverse-follow lookup is kind:3 only — a follower IS the author of a kind:3 that `#p`-tags the observer. */
|
||||
private val FOLLOW_KINDS = listOf(ContactListEvent.KIND)
|
||||
}
|
||||
}
|
||||
@@ -31,6 +31,11 @@ import com.vitorpamplona.quartz.nip01Core.signers.NostrSigner
|
||||
import com.vitorpamplona.quartz.nip01Core.store.IEventStore
|
||||
import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.ContactCardEvent
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.followerCount
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.hops
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.rank
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.FollowerCountTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.HopsTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.RankTag
|
||||
import com.vitorpamplona.quartz.utils.TimeUtils
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
@@ -62,6 +67,21 @@ class GrapeRankPublisher(
|
||||
private val store: IEventStore,
|
||||
private val log: (String) -> Unit = {},
|
||||
) {
|
||||
/**
|
||||
* One desired kind:30382 card: the GrapeRank result for [target] the caller
|
||||
* wants persisted. [rank] (`round(score*100)`) is always written; [followers]
|
||||
* (trusted-follower count) and [hops] (follow-graph distance from the observer)
|
||||
* are optional — a `null` omits that tag, so a caller that only knows the rank
|
||||
* still produces a valid card, and older cards missing those tags reconcile
|
||||
* cleanly against a run that now supplies them.
|
||||
*/
|
||||
class ScoredCard(
|
||||
val target: HexKey,
|
||||
val rank: Int,
|
||||
val followers: Int? = null,
|
||||
val hops: Int? = null,
|
||||
)
|
||||
|
||||
/** Outcome of one [reconcileLocal]: what was signed, retracted, and left alone. */
|
||||
class LocalResult(
|
||||
/** New or rank-changed cards signed and inserted into the store. */
|
||||
@@ -73,25 +93,27 @@ class GrapeRankPublisher(
|
||||
)
|
||||
|
||||
/**
|
||||
* Reconcile the desired [scored] `(target, rank)` set (the caller has already
|
||||
* applied any rank cutoff) into the local store, signed by [providerSigner]:
|
||||
* upsert the changed cards, retract the stale ones, skip the unchanged rest.
|
||||
* Every card and retraction is stamped [createdAt], so a replacement is
|
||||
* strictly newer than what it displaces.
|
||||
* Reconcile the desired [scored] card set (the caller has already applied any
|
||||
* rank cutoff) into the local store, signed by [providerSigner]: upsert the
|
||||
* changed cards, retract the stale ones, skip the unchanged rest. Every card and
|
||||
* retraction is stamped [createdAt], so a replacement is strictly newer than what
|
||||
* it displaces.
|
||||
*/
|
||||
suspend fun reconcileLocal(
|
||||
providerSigner: NostrSigner,
|
||||
providerPubkey: HexKey,
|
||||
scored: List<Pair<HexKey, Int>>,
|
||||
scored: List<ScoredCard>,
|
||||
createdAt: Long = TimeUtils.now(),
|
||||
): LocalResult {
|
||||
val existing = existingCards(providerPubkey)
|
||||
val desiredTargets = scored.mapTo(HashSet()) { it.first }
|
||||
val desiredTargets = scored.mapTo(HashSet()) { it.target }
|
||||
|
||||
// Upsert targets whose rank tag STRING would change (or that have no card
|
||||
// yet). RankTag.assemble writes rank.toString(), so we diff that exact
|
||||
// string — an unchanged score never produces a new signature.
|
||||
val changed = scored.filter { (target, rank) -> existing[target]?.let(::rankTagValue) != rank.toString() }
|
||||
// Upsert targets whose card values would change (or that have no card yet).
|
||||
// We diff every tag we write — rank, follower count, and hops — so a card
|
||||
// re-signs when any of them moves, but an all-unchanged run never churns an
|
||||
// event id. A card predating the follower/hops tags reads null for them and
|
||||
// so re-signs once to pick them up.
|
||||
val changed = scored.filter { card -> existing[card.target]?.let(::cardValues) != desiredValues(card) }
|
||||
|
||||
// Retract cards whose target is no longer desired — it dropped out of the
|
||||
// graph, or fell below the caller's cutoff. No stale assertion is left standing.
|
||||
@@ -203,15 +225,15 @@ class GrapeRankPublisher(
|
||||
}.toMap()
|
||||
|
||||
/**
|
||||
* The raw `rank` tag value string on a card — exactly what a client diffs, so an
|
||||
* unchanged score never produces a new signature. Our cards carry only a `rank`
|
||||
* tag (plus the d-tag target), so this one value decides whether a re-sign
|
||||
* would differ.
|
||||
* The (rank, followers, hops) triple stored on an existing card — the values a
|
||||
* re-sign would overwrite. Compared against [desiredValues] so an unchanged run
|
||||
* produces no new signature; a missing tag reads null and so differs from a
|
||||
* desired non-null value (the one-time migration onto the new tags).
|
||||
*/
|
||||
private fun rankTagValue(card: ContactCardEvent): String? =
|
||||
card.tags.firstNotNullOfOrNull { tag ->
|
||||
if (tag.size > 1 && tag[0] == RankTag.TAG_NAME) tag[1] else null
|
||||
}
|
||||
private fun cardValues(card: ContactCardEvent): Triple<Int?, Int?, Int?> = Triple(card.rank(), card.followerCount(), card.hops())
|
||||
|
||||
/** The (rank, followers, hops) triple a [ScoredCard] would write. */
|
||||
private fun desiredValues(card: ScoredCard): Triple<Int?, Int?, Int?> = Triple(card.rank, card.followers, card.hops)
|
||||
|
||||
/**
|
||||
* Build + sign one kind:30382 card per (target, rank) — fanned out on
|
||||
@@ -221,7 +243,7 @@ class GrapeRankPublisher(
|
||||
*/
|
||||
private suspend fun signAndInsertCards(
|
||||
signer: NostrSigner,
|
||||
cards: List<Pair<HexKey, Int>>,
|
||||
cards: List<ScoredCard>,
|
||||
createdAt: Long,
|
||||
): Int {
|
||||
if (cards.isEmpty()) return 0
|
||||
@@ -230,13 +252,17 @@ class GrapeRankPublisher(
|
||||
val signedBatch =
|
||||
coroutineScope {
|
||||
batch
|
||||
.map { (target, rank) ->
|
||||
.map { card ->
|
||||
async(Dispatchers.Default) {
|
||||
ContactCardEvent.create(
|
||||
targetUser = target,
|
||||
targetUser = card.target,
|
||||
signer = signer,
|
||||
createdAt = createdAt,
|
||||
publicInitializer = { add(RankTag.assemble(rank)) },
|
||||
publicInitializer = {
|
||||
add(RankTag.assemble(card.rank))
|
||||
card.followers?.let { add(FollowerCountTag.assemble(it)) }
|
||||
card.hops?.let { add(HopsTag.assemble(it)) }
|
||||
},
|
||||
)
|
||||
}
|
||||
}.awaitAll()
|
||||
|
||||
@@ -60,9 +60,11 @@ class TrustGraph internal constructor(
|
||||
// each packing source id (low 29 bits) + relation code (top bits).
|
||||
internal val inOffsets: IntArray,
|
||||
internal val inPacked: IntArray,
|
||||
// CSR by source: out-neighbour targets of node s are outTargets[outOffsets[s] until outOffsets[s+1]].
|
||||
// CSR by source: out-edges of node s are outPacked[outOffsets[s] until outOffsets[s+1]],
|
||||
// each packing the neighbour (target) id (low 29 bits) + relation code (top bits) —
|
||||
// same layout as inPacked, so a forward walk can filter by relation.
|
||||
internal val outOffsets: IntArray,
|
||||
internal val outTargets: IntArray,
|
||||
internal val outPacked: IntArray,
|
||||
) {
|
||||
/** Node id for [pubkey], or `-1` if it never appeared in the graph. */
|
||||
fun idOf(pubkey: HexKey): Int = ids[pubkey] ?: -1
|
||||
@@ -72,10 +74,87 @@ class TrustGraph internal constructor(
|
||||
|
||||
fun edgeCount(): Int = inPacked.size
|
||||
|
||||
/**
|
||||
* Hop distance from [observer] to every node along **FOLLOW edges only** — a
|
||||
* plain BFS over the follow graph (mutes/reports are not reachability). Indexed
|
||||
* by node id: the observer is `0`, a user the observer follows directly is `1`,
|
||||
* a user followed by a 1-hop user is `2`, and so on. A node with no follow path
|
||||
* from the observer stays [UNREACHABLE]. Returns an all-[UNREACHABLE] array (bar
|
||||
* a missing observer's own 0) when the observer isn't in the graph.
|
||||
*
|
||||
* BFS over the compact int-CSR: an `IntArray` ring queue and the distance array
|
||||
* itself as the visited set, so a whole-network graph costs one int per node
|
||||
* plus the queue — no boxed collections.
|
||||
*/
|
||||
fun hopsFrom(observer: HexKey): IntArray {
|
||||
val hops = IntArray(nodeCount) { UNREACHABLE }
|
||||
val observerId = idOf(observer)
|
||||
if (observerId < 0) return hops
|
||||
|
||||
hops[observerId] = 0
|
||||
// Ring buffer sized to the node count — BFS enqueues each node at most once.
|
||||
val queue = IntArray(nodeCount)
|
||||
var head = 0
|
||||
var tail = 0
|
||||
queue[tail++] = observerId
|
||||
while (head < tail) {
|
||||
val node = queue[head++]
|
||||
val nextHop = hops[node] + 1
|
||||
var i = outOffsets[node]
|
||||
val end = outOffsets[node + 1]
|
||||
while (i < end) {
|
||||
val packed = outPacked[i]
|
||||
if ((packed ushr SOURCE_BITS) == TrustRelation.FOLLOW.code) {
|
||||
val target = packed and SOURCE_MASK
|
||||
if (hops[target] == UNREACHABLE) {
|
||||
hops[target] = nextHop
|
||||
queue[tail++] = target
|
||||
}
|
||||
}
|
||||
i++
|
||||
}
|
||||
}
|
||||
return hops
|
||||
}
|
||||
|
||||
/**
|
||||
* For every node, how many of its **followers** score at or above [minScore] in
|
||||
* [scores] — the "trusted follower" count Brainstorm records on its `ScoreCard`
|
||||
* (its `verifiedFollowersInfluenceCutoff` defaults to 0.02). A follower is the
|
||||
* source of an incoming FOLLOW edge; mutes/reports don't count. Indexed by node
|
||||
* id. [scores] must be the array [GrapeRank.compute] returned for this graph.
|
||||
*/
|
||||
fun trustedFollowerCounts(
|
||||
scores: DoubleArray,
|
||||
minScore: Double,
|
||||
): IntArray {
|
||||
val counts = IntArray(nodeCount)
|
||||
var target = 0
|
||||
while (target < nodeCount) {
|
||||
var count = 0
|
||||
var i = inOffsets[target]
|
||||
val end = inOffsets[target + 1]
|
||||
while (i < end) {
|
||||
val packed = inPacked[i]
|
||||
if ((packed ushr SOURCE_BITS) == TrustRelation.FOLLOW.code) {
|
||||
val source = packed and SOURCE_MASK
|
||||
if (scores[source] >= minScore) count++
|
||||
}
|
||||
i++
|
||||
}
|
||||
counts[target] = count
|
||||
target++
|
||||
}
|
||||
return counts
|
||||
}
|
||||
|
||||
companion object {
|
||||
const val SOURCE_BITS = 29
|
||||
const val SOURCE_MASK = (1 shl SOURCE_BITS) - 1
|
||||
const val MAX_NODES = SOURCE_MASK // ids must fit in the low 29 bits
|
||||
|
||||
/** [hopsFrom] distance for a node with no follow path from the observer. */
|
||||
const val UNREACHABLE = -1
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -108,20 +108,24 @@ class TrustGraphBuilder {
|
||||
inPacked[inCursor[t]++] = edgeSourcesPacked.get(i)
|
||||
}
|
||||
|
||||
// Outgoing CSR (by source).
|
||||
// Outgoing CSR (by source). Each entry packs the target id + relation code
|
||||
// in the same layout as the incoming CSR, so a forward walk (e.g. the
|
||||
// follow-only BFS in TrustGraph.hopsFrom) can filter edges by relation.
|
||||
val outOffsets = IntArray(n + 1)
|
||||
for (i in 0 until m) {
|
||||
val s = edgeSourcesPacked.get(i) and TrustGraph.SOURCE_MASK
|
||||
outOffsets[s + 1]++
|
||||
}
|
||||
for (i in 1..n) outOffsets[i] += outOffsets[i - 1]
|
||||
val outTargets = IntArray(m)
|
||||
val outPacked = IntArray(m)
|
||||
val outCursor = outOffsets.copyOf()
|
||||
for (i in 0 until m) {
|
||||
val s = edgeSourcesPacked.get(i) and TrustGraph.SOURCE_MASK
|
||||
outTargets[outCursor[s]++] = edgeTargets.get(i)
|
||||
val packedSource = edgeSourcesPacked.get(i)
|
||||
val s = packedSource and TrustGraph.SOURCE_MASK
|
||||
val relationCode = packedSource ushr TrustGraph.SOURCE_BITS
|
||||
outPacked[outCursor[s]++] = edgeTargets.get(i) or (relationCode shl TrustGraph.SOURCE_BITS)
|
||||
}
|
||||
|
||||
return TrustGraph(n, pubkeys.toTypedArray(), ids, inOffsets, inPacked, outOffsets, outTargets)
|
||||
return TrustGraph(n, pubkeys.toTypedArray(), ids, inOffsets, inPacked, outOffsets, outPacked)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -61,6 +61,8 @@ class ContactCardEvent(
|
||||
|
||||
fun followerCount() = tags.followerCount()
|
||||
|
||||
fun hops() = tags.hops()
|
||||
|
||||
fun firstCreatedAt() = tags.firstCreatedAt()
|
||||
|
||||
fun postCount() = tags.postCount()
|
||||
|
||||
@@ -25,6 +25,7 @@ import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.ActiveHoursEnd
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.ActiveHoursStartTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.FirstCreatedAtTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.FollowerCountTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.HopsTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.PetNameTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.PostCountTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.RankTag
|
||||
@@ -45,6 +46,8 @@ fun TagArrayBuilder<ContactCardEvent>.rank(rank: Int) = addUnique(RankTag.assemb
|
||||
|
||||
fun TagArrayBuilder<ContactCardEvent>.followers(count: Int) = addUnique(FollowerCountTag.assemble(count))
|
||||
|
||||
fun TagArrayBuilder<ContactCardEvent>.hops(hops: Int) = addUnique(HopsTag.assemble(hops))
|
||||
|
||||
fun TagArrayBuilder<ContactCardEvent>.firstCreatedAt(timestamp: Long) = addUnique(FirstCreatedAtTag.assemble(timestamp))
|
||||
|
||||
fun TagArrayBuilder<ContactCardEvent>.postCount(count: Int) = addUnique(PostCountTag.assemble(count))
|
||||
|
||||
@@ -26,6 +26,7 @@ import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.ActiveHoursEnd
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.ActiveHoursStartTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.FirstCreatedAtTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.FollowerCountTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.HopsTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.PetNameTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.PostCountTag
|
||||
import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.RankTag
|
||||
@@ -46,6 +47,8 @@ fun TagArray.rank() = fastFirstNotNullOfOrNull(RankTag::parse)
|
||||
|
||||
fun TagArray.followerCount() = fastFirstNotNullOfOrNull(FollowerCountTag::parse)
|
||||
|
||||
fun TagArray.hops() = fastFirstNotNullOfOrNull(HopsTag::parse)
|
||||
|
||||
fun TagArray.firstCreatedAt() = fastFirstNotNullOfOrNull(FirstCreatedAtTag::parse)
|
||||
|
||||
fun TagArray.postCount() = fastFirstNotNullOfOrNull(PostCountTag::parse)
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.has
|
||||
import com.vitorpamplona.quartz.utils.ensure
|
||||
|
||||
/**
|
||||
* The number of follow hops from the observer to the card's target — the length
|
||||
* of the shortest follow path in the observer's web of trust. A user the observer
|
||||
* follows directly is 1 hop; a user followed by someone the observer follows is 2;
|
||||
* and so on. Mirrors the `hops` field on Brainstorm's GrapeRank `ScoreCard`.
|
||||
*/
|
||||
class HopsTag {
|
||||
companion object {
|
||||
const val TAG_NAME = "hops"
|
||||
|
||||
fun parse(tag: Array<String>): Int? {
|
||||
ensure(tag.has(1)) { return null }
|
||||
ensure(tag[0] == TAG_NAME) { return null }
|
||||
ensure(tag[1].isNotEmpty()) { return null }
|
||||
return tag[1].toIntOrNull()
|
||||
}
|
||||
|
||||
fun assemble(hops: Int) = arrayOf(TAG_NAME, hops.toString())
|
||||
}
|
||||
}
|
||||
@@ -20,6 +20,7 @@
|
||||
*/
|
||||
package com.vitorpamplona.quartz.experimental.graperank
|
||||
|
||||
import com.vitorpamplona.quartz.experimental.graperank.GrapeRankPublisher.ScoredCard
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
@@ -64,7 +65,7 @@ class GrapeRankPublisherTest {
|
||||
val c = hexKey(0xC)
|
||||
|
||||
// First run: every card is new.
|
||||
val r1 = publisher.reconcileLocal(signer, provider, listOf(a to 50, b to 10), createdAt = 1_000L)
|
||||
val r1 = publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, 50), ScoredCard(b, 10)), createdAt = 1_000L)
|
||||
assertEquals(2, r1.signed)
|
||||
assertEquals(0, r1.unchanged)
|
||||
assertEquals(0, r1.retracted)
|
||||
@@ -76,7 +77,7 @@ class GrapeRankPublisherTest {
|
||||
.toSet()
|
||||
|
||||
// Same ranks again: nothing is re-signed, no event id churns.
|
||||
val r2 = publisher.reconcileLocal(signer, provider, listOf(a to 50, b to 10), createdAt = 2_000L)
|
||||
val r2 = publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, 50), ScoredCard(b, 10)), createdAt = 2_000L)
|
||||
assertEquals(0, r2.signed)
|
||||
assertEquals(2, r2.unchanged)
|
||||
assertEquals(0, r2.retracted)
|
||||
@@ -88,7 +89,7 @@ class GrapeRankPublisherTest {
|
||||
assertEquals(firstIds, secondIds)
|
||||
|
||||
// a's rank moved, b dropped out, c is new: a + c signed, b retracted.
|
||||
val r3 = publisher.reconcileLocal(signer, provider, listOf(a to 60, c to 5), createdAt = 3_000L)
|
||||
val r3 = publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, 60), ScoredCard(c, 5)), createdAt = 3_000L)
|
||||
assertEquals(2, r3.signed)
|
||||
assertEquals(0, r3.unchanged)
|
||||
assertEquals(1, r3.retracted)
|
||||
@@ -113,16 +114,54 @@ class GrapeRankPublisherTest {
|
||||
|
||||
val a = hexKey(0xA)
|
||||
|
||||
publisher.reconcileLocal(signer, provider, listOf(a to 40), createdAt = 1_000L)
|
||||
publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, 40)), createdAt = 1_000L)
|
||||
publisher.reconcileLocal(signer, provider, emptyList(), createdAt = 2_000L)
|
||||
assertEquals(emptyMap(), cardsByTarget(store, provider))
|
||||
|
||||
// A NEWER card outranks the older kind:5 (NIP-09 deletions only cover
|
||||
// versions up to their created_at), so the target comes back cleanly.
|
||||
val r = publisher.reconcileLocal(signer, provider, listOf(a to 45), createdAt = 3_000L)
|
||||
val r = publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, 45)), createdAt = 3_000L)
|
||||
assertEquals(1, r.signed)
|
||||
assertEquals(mapOf(a to 45), cardsByTarget(store, provider))
|
||||
|
||||
store.close()
|
||||
}
|
||||
|
||||
@Test
|
||||
fun writesFollowerAndHopTagsAndReSignsWhenTheyMove() =
|
||||
runBlocking {
|
||||
val store = EventStore(null)
|
||||
val signer = NostrSignerInternal(KeyPair())
|
||||
val provider = signer.pubKey
|
||||
val publisher = GrapeRankPublisher(store)
|
||||
|
||||
val a = hexKey(0xA)
|
||||
|
||||
suspend fun cardFor(target: String): ContactCardEvent =
|
||||
store
|
||||
.query<Event>(Filter(kinds = listOf(ContactCardEvent.KIND), authors = listOf(provider)))
|
||||
.filterIsInstance<ContactCardEvent>()
|
||||
.first { it.aboutUser() == target }
|
||||
|
||||
// A card carrying rank + followers + hops persists all three tags.
|
||||
publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, rank = 50, followers = 7, hops = 2)), createdAt = 1_000L)
|
||||
cardFor(a).let {
|
||||
assertEquals(50, it.rank())
|
||||
assertEquals(7, it.followerCount())
|
||||
assertEquals(2, it.hops())
|
||||
}
|
||||
|
||||
// Rank unchanged but the follower count moved → the card re-signs.
|
||||
val moved = publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, rank = 50, followers = 9, hops = 2)), createdAt = 2_000L)
|
||||
assertEquals(1, moved.signed)
|
||||
assertEquals(0, moved.unchanged)
|
||||
assertEquals(9, cardFor(a).followerCount())
|
||||
|
||||
// Everything identical → no re-sign.
|
||||
val same = publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, rank = 50, followers = 9, hops = 2)), createdAt = 3_000L)
|
||||
assertEquals(0, same.signed)
|
||||
assertEquals(1, same.unchanged)
|
||||
|
||||
store.close()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -104,4 +104,76 @@ class TrustGraphBuilderTest {
|
||||
assertEquals(3, graph.nodeCount, "alice, bob, carol interned once each")
|
||||
assertEquals(3, graph.edgeCount())
|
||||
}
|
||||
|
||||
private fun TrustGraph.hop(
|
||||
observer: HexKey,
|
||||
target: HexKey,
|
||||
): Int {
|
||||
val id = idOf(target)
|
||||
return if (id < 0) TrustGraph.UNREACHABLE else hopsFrom(observer)[id]
|
||||
}
|
||||
|
||||
@Test
|
||||
fun hopsCountFollowDistanceFromTheObserver() {
|
||||
val b = TrustGraphBuilder()
|
||||
b.addFollows(alice, listOf(bob)) // alice -> bob (1 hop)
|
||||
b.addFollows(bob, listOf(carol)) // bob -> carol (2 hops)
|
||||
b.addFollows(carol, listOf(dave)) // carol -> dave (3 hops)
|
||||
val graph = b.build()
|
||||
|
||||
assertEquals(0, graph.hop(alice, alice), "the observer is 0 hops from itself")
|
||||
assertEquals(1, graph.hop(alice, bob), "a direct follow is 1 hop")
|
||||
assertEquals(2, graph.hop(alice, carol))
|
||||
assertEquals(3, graph.hop(alice, dave))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun hopsTakeTheShortestFollowPath() {
|
||||
val b = TrustGraphBuilder()
|
||||
b.addFollows(alice, listOf(bob, dave)) // dave is also a direct follow…
|
||||
b.addFollows(bob, listOf(carol))
|
||||
b.addFollows(carol, listOf(dave)) // …as well as reachable at 3 hops
|
||||
val graph = b.build()
|
||||
assertEquals(1, graph.hop(alice, dave), "BFS keeps the shortest of two follow paths")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun hopsIgnoreMuteAndReportEdges() {
|
||||
val b = TrustGraphBuilder()
|
||||
b.addMutes(alice, listOf(bob)) // a mute is not reachability
|
||||
b.addReports(alice, listOf(carol)) // neither is a report
|
||||
val graph = b.build()
|
||||
assertEquals(TrustGraph.UNREACHABLE, graph.hop(alice, bob), "a muted user is not reachable by follows")
|
||||
assertEquals(TrustGraph.UNREACHABLE, graph.hop(alice, carol), "a reported user is not reachable by follows")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun hopsAreUnreachableWithNoFollowPath() {
|
||||
val b = TrustGraphBuilder()
|
||||
b.addFollows(alice, listOf(bob))
|
||||
b.addFollows(carol, listOf(dave)) // a separate island
|
||||
val graph = b.build()
|
||||
assertEquals(TrustGraph.UNREACHABLE, graph.hop(alice, dave))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun trustedFollowerCountsOnlyCountFollowersAboveTheThreshold() {
|
||||
val b = TrustGraphBuilder()
|
||||
// carol is followed by alice, bob and dave (three follow edges in).
|
||||
b.addFollows(alice, listOf(carol))
|
||||
b.addFollows(bob, listOf(carol))
|
||||
b.addFollows(dave, listOf(carol))
|
||||
// dave also MUTES carol — a mute must never count as a follower.
|
||||
b.addMutes(dave, listOf(carol))
|
||||
val graph = b.build()
|
||||
|
||||
val carolId = graph.idOf(carol)
|
||||
val scores = DoubleArray(graph.nodeCount)
|
||||
scores[graph.idOf(alice)] = 0.9 // trusted
|
||||
scores[graph.idOf(bob)] = 0.01 // below the 0.02 cutoff
|
||||
scores[graph.idOf(dave)] = 0.5 // trusted
|
||||
|
||||
val counts = graph.trustedFollowerCounts(scores, minScore = 0.02)
|
||||
assertEquals(2, counts[carolId], "only alice and dave clear the cutoff; bob is too low and dave's mute doesn't count")
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user