diff --git a/44.md b/44.md index a7c13f12..47b89906 100644 --- a/44.md +++ b/44.md @@ -84,10 +84,12 @@ NIP-44 version 2 has the following design characteristics: - Slice 76-byte HKDF output into: `chacha_key` (bytes 0..32), `chacha_nonce` (bytes 32..44), `hmac_key` (bytes 44..76) 4. Add padding - Content must be encoded from UTF-8 into byte array - - Validate plaintext length. Minimum is 1 byte, maximum is 65535 bytes - - Padding format is: `[plaintext_length: u16][plaintext][zero_bytes]` + - Validate plaintext length. Minimum is 1 byte, maximum is 4,294,967,295 bytes - Padding algorithm is related to powers-of-two, with min padded msg size of 32 bytes - - Plaintext length is encoded in big-endian as first 2 bytes of the padded blob + - Plaintext length prefix is encoded in big-endian: + - If length is less than 65536: prefix is 2 bytes (`u16`), format is `[plaintext_length: u16][plaintext][zero_bytes]` + - If length is 65536 or greater: prefix is 6 bytes (2 zero bytes + `u32`), format is `[0x00, 0x00][plaintext_length: u32][plaintext][zero_bytes]` + - A zero value in the first 2 bytes signals the extended format; since valid plaintext is at least 1 byte, a u16 length of 0 is otherwise invalid 5. Encrypt padded content - Use ChaCha20, with key and nonce from step 3 6. Calculate MAC (message authentication code) @@ -112,8 +114,8 @@ validation rules, refer to BIP-340. 2. Decode base64 - Base64 is decoded into `version, nonce, ciphertext, mac` - If the version is unknown, implementations must indicate that the encryption version is not supported - - Validate length of base64 message to prevent DoS on base64 decoder: it can be in range from 132 to 87472 chars - - Validate length of decoded message to verify output of the decoder: it can be in range from 99 to 65603 bytes + - Validate minimum length of base64 message to prevent DoS on base64 decoder: it must be at least 132 chars + - Validate minimum length of decoded message to verify output of the decoder: it must be at least 99 bytes 3. Calculate conversation key - See step 1 of [encryption](#Encryption) 4. Calculate message keys @@ -124,10 +126,23 @@ validation rules, refer to BIP-340. 6. Decrypt ciphertext - Use ChaCha20 with key and nonce from step 3 7. Remove padding - - Read the first two BE bytes of plaintext that correspond to plaintext length - - Verify that the length of sliced plaintext matches the value of the two BE bytes + - Read the first 2 bytes as a big-endian u16 + - If zero, read the next 4 bytes as a big-endian u32 plaintext length (6-byte prefix total) + - Otherwise, use those 2 bytes as the u16 plaintext length (2-byte prefix total) + - Verify that the length of sliced plaintext matches the decoded length - Verify that calculated padding from step 3 of the [encryption](#Encryption) process matches the actual padding +### Implementation considerations + +The theoretical maximum plaintext size is 2^32 - 1 bytes (~4 GB). Implementations SHOULD enforce +their own maximum payload size based on platform and resource constraints, rejecting oversized payloads +early in `decode_payload` (before base64 decoding) to prevent denial-of-service. Decryption may require +several times the payload size in working memory due to base64 decoding, byte array slicing, and +padding operations. For reference, JVM-based systems are limited to ~2 GB contiguous arrays, and mobile +devices may have significantly less available memory. Note that `calc_padded_len` can return values up +to 2^32, which exceeds the range of unsigned 32-bit integers; implementations must use 64-bit (or +larger) arithmetic for padding calculations. + ### Details - Cryptographic methods @@ -149,7 +164,8 @@ validation rules, refer to BIP-340. `i`-th byte (inclusive) to the `j`-th byte (exclusive) of `x`. - Constants `c`: - `min_plaintext_size` is 1. 1 byte msg is padded to 32 bytes. - - `max_plaintext_size` is 65535 (64kB - 1). It is padded to 65536 bytes. + - `max_plaintext_size` is 4294967295 (2^32 - 1). + - `extended_prefix_threshold` is 65536. Lengths below this use a 2-byte u16 prefix; lengths at or above use a 6-byte prefix (2 zero bytes + u32). - Functions - `base64_encode(string)` and `base64_decode(bytes)` are Base64 ([RFC 4648](https://datatracker.ietf.org/doc/html/rfc4648), with padding) - `concat` refers to byte array concatenation @@ -157,6 +173,9 @@ validation rules, refer to BIP-340. - `utf8_encode(string)` and `utf8_decode(bytes)` transform string to byte array and back - `write_u8(number)` restricts number to values 0..255 and encodes into Big-Endian uint8 byte array - `write_u16_be(number)` restricts number to values 0..65535 and encodes into Big-Endian uint16 byte array + - `write_u32_be(number)` restricts number to values 0..4294967295 and encodes into Big-Endian uint32 byte array + - `read_uint16_be(bytes)` reads 2 bytes as a Big-Endian unsigned 16-bit integer + - `read_uint32_be(bytes)` reads 4 bytes as a Big-Endian unsigned 32-bit integer - `zeros(length)` creates byte array of length `length >= 0`, filled with zeros - `floor(number)` and `log2(number)` are well-known mathematical methods @@ -176,40 +195,51 @@ def calc_padded_len(unpadded_len): if unpadded_len <= 32: return 32 else: - return chunk * (floor((len - 1) / chunk) + 1) + return chunk * (floor((unpadded_len - 1) / chunk) + 1) # Converts unpadded plaintext to padded bytearray def pad(plaintext): unpadded = utf8_encode(plaintext) - unpadded_len = len(plaintext) + unpadded_len = len(unpadded) if (unpadded_len < c.min_plaintext_size or unpadded_len > c.max_plaintext_size): raise Exception('invalid plaintext length') - prefix = write_u16_be(unpadded_len) + if unpadded_len >= c.extended_prefix_threshold: + prefix = concat([0, 0], write_u32_be(unpadded_len)) # 6 bytes + else: + prefix = write_u16_be(unpadded_len) # 2 bytes suffix = zeros(calc_padded_len(unpadded_len) - unpadded_len) return concat(prefix, unpadded, suffix) # Converts padded bytearray to unpadded plaintext def unpad(padded): - unpadded_len = read_uint16_be(padded[0:2]) - unpadded = padded[2:2+unpadded_len] + first_two = read_uint16_be(padded[0:2]) + if first_two == 0: + unpadded_len = read_uint32_be(padded[2:6]) + if unpadded_len < c.extended_prefix_threshold: raise Exception('invalid padding') + prefix_len = 6 + else: + unpadded_len = first_two + prefix_len = 2 + unpadded = padded[prefix_len:prefix_len+unpadded_len] if (unpadded_len == 0 or len(unpadded) != unpadded_len or - len(padded) != 2 + calc_padded_len(unpadded_len)): raise Exception('invalid padding') + len(padded) != prefix_len + calc_padded_len(unpadded_len)): raise Exception('invalid padding') return utf8_decode(unpadded) -# metadata: always 65b (version: 1b, nonce: 32b, max: 32b) -# plaintext: 1b to 0xffff -# padded plaintext: 32b to 0xffff -# ciphertext: 32b+2 to 0xffff+2 -# raw payload: 99 (65+32+2) to 65603 (65+0xffff+2) -# compressed payload (base64): 132b to 87472b +# metadata: always 65b (version: 1b, nonce: 32b, mac: 32b) +# plaintext: 1b to 0xffffffff +# padded plaintext (small, <65536): 32b to 0x10000, with 2b prefix -> 34b to 0x10000+2 +# padded plaintext (large, >=65536): 0x10000 to 0x100000000, with 6b prefix -> 0x10006 to 0x100000000+6 +# ciphertext: same as padded plaintext (chacha20 doesn't change length) +# raw payload (small): 99 (65+34) to 65603 (65+0x10000+2) +# raw payload (large): 65607 (65+0x10006) to 4294967367 (65+0x100000000+6) def decode_payload(payload): plen = len(payload) if plen == 0 or payload[0] == '#': raise Exception('unknown version') - if plen < 132 or plen > 87472: raise Exception('invalid payload size') + if plen < 132: raise Exception('invalid payload size') data = base64_decode(payload) - dlen = len(d) - if dlen < 99 or dlen > 65603: raise Exception('invalid data size'); + dlen = len(data) + if dlen < 99: raise Exception('invalid data size'); vers = data[0] if vers != 2: raise Exception('unknown version ' + vers) nonce = data[1:33] @@ -295,3 +325,27 @@ The file also contains intermediate values. A quick guidance with regards to its - `invalid.encrypt_msg_lengths` - `invalid.get_conversation_key`: calculating conversation_key must throw an error - `invalid.decrypt`: decrypting message content must throw an error + +#### Extended length prefix test vectors + +The following test vectors exercise the boundary between the 2-byte u16 prefix and the 6-byte +extended prefix. Since the payloads are too large to include inline, SHA-256 checksums of the +plaintext and base64-encoded payload are provided (following the `encrypt_decrypt_long_msg` pattern). + +All vectors use the same `conversation_key` and `nonce` as above. Plaintext is the byte `0x61` +(`'a'`) repeated to the specified length. + +``` +conversation_key: c41c775356fd92eadc63ff5a0dc1da211b268cbea22316767095b2871ea1412d +nonce: 0000000000000000000000000000000000000000000000000000000000000001 +``` + +| plaintext_len | prefix | padded_len | plaintext_sha256 | payload_sha256 | +|---|---|---|---|---| +| 65535 | u16 (2 bytes) | 65536 | `6e1bebca6a8229364a162a72ef064826c4cd7457bf54f190ef782bd9deff3e42` | `6d8c2810d1e870fbaa1f0a0937126cca837a15f9260e27060c331d70a3c0bc84` | +| 65536 | extended (6 bytes) | 65536 | `bf718b6f653bebc184e1479f1935b8da974d701b893afcf49e701f3e2f9f9c5a` | `b7b4edb36ba92e267d322d56d9aebc22e7fa96ff52e3c12adc07f07a43cbc616` | +| 65537 | extended (6 bytes) | 81920 | `008ffc88d3c96a9f307524eb361e47c5222a887fc45fa0c1fb8d429c5c23b430` | `eeb7c7c5373894ea2c1547cfd3ccb15d5a0b2d619da852e5c79df792dcc9e435` | + +Note that 65535 and 65536 both have a `padded_len` of 65536, but the total padded-with-prefix +sizes differ: 65538 (2 + 65536) vs 65542 (6 + 65536). The jump to 65537 triggers the next +padding bucket at 81920.