Binary Nostr Events (.bne): Unified Binary Protocol Architecture

1. Overview & Motivation

In standard Nostr (NIP-01), an event is defined as a JSON string containing text-encoded metadata and content. Media and binary attachments (blobs) are stored on separate HTTP servers (BUD-01, NIP-96), referenced via URLs inside event content or tags (NIP-94, NIP-92).

This document outlines an architectural proposal for Binary Nostr Events (.bne): a canonical, close-to-the-metal binary event envelope. By making the event structure binary at its foundation, the protocol eliminates the divide between "text events" and "binary blobs". Everything—social posts, file attachments, media, encrypted payloads, and protocol signals—becomes a binary-encoded event.

2. Binary Event Envelope Memory Layout

A Binary Event (.bne) is serialized into a byte sequence with fixed-width fields for fast zero-copy memory access and length-prefixed variables.

+---------------------------------------------------------------------------------+
| Offset | Field          | Type               | Size (Bytes)  | Description      |
+--------+----------------+--------------------+---------------+------------------+
| 0      | version        | uint8              | 1             | Format version 1 |
| 1      | pubkey         | byte[32]           | 32            | secp256k1 key    |
| 33     | created_at     | uint64 (be)        | 8             | Unix timestamp   |
| 41     | kind           | uint16 (be)        | 2             | Event kind       |
| 43     | tags_count     | uint16 (be)        | 2             | Number of tags   |
| 45     | tags_bytes     | variable           | dynamic       | TLV tag table    |
| 45+T   | content_length | uint32 (be)        | 4             | Payload byte len |
| 49+T   | content        | byte[content_len]  | dynamic       | Raw binary body  |
| 49+T+C | sig            | byte[64]           | 64            | Schnorr signature|
+---------------------------------------------------------------------------------+

Tag Encoding (TLV)

Each tag in tags_bytes is encoded as:

element_count (uint8): Number of string/byte elements in the tag array (e.g., 2 for ["e", "<id>"]).
For each element:
- element_len (uint16): Length of element in bytes.
- element_data (byte[element_len]): Raw string or hex bytes.

3. Cryptographic Verification & Event ID

The id of a Binary Event is computed by applying SHA-256 over the canonical binary serialization of all fields excluding the sig field (offsets 0 through 49 + T + C):

\text{id} = \text{SHA256}(\text{version} \parallel \text{pubkey} \parallel \text{created\_at} \parallel \text{kind} \parallel \text{tags} \parallel \text{content\_length} \parallel \text{content})

The sig field is a standard Schnorr signature over the 32-byte id digest using pubkey.

flowchart TD
    subgraph Event Envelope Serialization
        V[version: 1B]
        PK[pubkey: 32B]
        TS[created_at: 8B]
        K[kind: 2B]
        T[tags: variable]
        CL[content_length: 4B]
        C[content payload: variable]
        S[sig: 64B]
    end

    V & PK & TS & K & T & CL & C -->|SHA256| ID[32-Byte Event ID]
    ID & PK & S -->|Schnorr Verify| Valid{Valid Signature?}

4. Content Identification Strategy: Kind-Based Parsing

To maintain maximum performance without adding header flags, .bne parsers identify the content type using Event Kinds combined with Magic Bytes:

                          .bne Event Received
                                   |
                          Check Event 'kind'
                                   |
             +---------------------+---------------------+
             |                                           |
    Text Kind (0, 1, 30023...)               Binary Kind (1063, 24242...)
             |                                           |
    Decode content as UTF-8                      Inspect content
             |                                 Magic Bytes / Header
    Render text / JSON                                   |
                                              +----------+----------+
                                              |                     |
                                         JPEG (0xFFD8FF)      PNG (0x89504E)
                                              |                     |
                                         Render Image          Render Image

Text & Protocol Kinds (kind: 0, 1, 30023...):
- The content bytes are directly interpreted as raw UTF-8 text strings or JSON objects.
Binary & Media Kinds (kind: 1063 NIP-94 File Metadata, kind: 24242 BUD-11...):
- The content section contains raw binary payload bytes (e.g. JPEG, PNG, MP4, PDF, or encrypted bytes).
- Parsers check magic bytes at the start of the content buffer (e.g., 0xFF 0xD8 0xFF for JPEG, 0x89 0x50 0x4E 0x47 for PNG, 0x25 0x50 0x44 0x46 for PDF) to identify media types instantly.
- Optional m (MIME) tags (["m", "image/jpeg"]) provide explicit MIME fallbacks for custom binary streams.

5. Media Polyglot Events (JPEG/EXIF, PNG Chunks, ID3)

Beyond wrapping media inside a .bne envelope, standard media formats can be structured as Polyglot Files—files that function as standard media files in operating systems while simultaneously serving as valid self-authenticated Nostr events!

+---------------------------------------------------------------------------+
| JPEG SOI (0xFFD8) | APP1 / EXIF Header (Nostr Metadata) | Compressed JPEG Data|
+---------------------------------------------------------------------------+
| Media Reader Sees: Valid JPEG Image                                       |
| Nostr Relay Sees: Signed Event (Pubkey + Sig in APP1 + Image Content Hash)|
+---------------------------------------------------------------------------+

Supported Media Metadata Containers

JPEG (.jpg): Nostr event headers (pubkey, created_at, kind, tags, sig) are embedded inside an APP1 (EXIF) or COM (Comment) segment. The signature signs the compressed image frame.
PNG (.png): Embedded inside a custom ancillary chunk named nOST or standard tEXt / iTXt metadata chunks.
MP3 / Audio (.mp3): Embedded inside an ID3v2 frame (e.g., GEOB frame).
MP4 / WebM Video: Embedded inside a custom moov / meta box atom.

Advantages of Media Polyglots

Universal Viewability: Double-clicking photo.jpg opens it in any image editor or web browser.
In-Band Signature Verification: Opening photo.jpg in a Nostr-aware client extracts the EXIF metadata, verifies the author's Schnorr signature against the image payload, and renders author provenance directly over the media!

6. Unifying Events and Blobs

Under this model:

Small & Medium Blobs as Direct Content: Images, thumbnails, audio clips, and encrypted media are stored directly in content.
Self-Authenticating Media: The blob is authenticated by the event signature and addressable by event id or content hash tag (["x", "<sha256>"]).
Multi-Chunk Large Blobs: Files exceeding single event storage limits use sequential addressable events with index tags (["chunk", "0"], ["chunk", "1"]).
Transport Agnosticism: Binary events can be sent over WebSockets, TCP streams, Bluetooth LE, QUIC, UDP, or saved directly as static files (.bne or polyglot .jpg) on disk/CDNs without string escaping or Base64 encoding.

7. Compatibility & Transcoding Gateway

To allow smooth coexistence with JSON-based Nostr (NIP-01):

[ Binary Event Client ] <---> ( Binary Protocol ) <---> [ Transcoding Gateway ] <---> ( JSON Protocol ) <---> [ Legacy Relay ]

Binary (.bne) to JSON Transcoding:
- pubkey, id, sig are converted from raw bytes to lowercase hex strings.
- content is UTF-8 decoded if text kind, or Base64/Data-URI encoded if binary kind.
- tags are unpacked into JSON string arrays.
Dual-Digest Mapping:
- For native binary events, gateways add a tag ["binary_id", "<hex_id>"] when publishing to legacy JSON relays so clients can map binary IDs to legacy JSON event IDs.

8. Performance Advantages

Zero-Copy Parsing: Relays and clients slice memory buffers directly without allocating JSON parse trees or escaping characters.
Bandwidth Reduction: Eliminates JSON structural overhead, hex encoding (50% key size reduction), and Base64 padding (33% content size reduction).
Storage Efficiency: Direct disk mmap for database indexes and fast streaming queries.

8.7 KiB Raw Permalink Blame History