Encrypting Files Peer-to-Peer with the Noise Protocol?

When I previously introduced the Noise Protocol, I wrote about this kind of application:

In addition, the Noise Protocol can also be used to encrypt files—as long as I have your public key, I can use a one-way handshake (7.4) to encrypt a file (along with the message sent during the handshake), then transmit it to an untrusted location (such as a cloud drive, FTP, IPFS, or even a blockchain). Only the person who holds the private key can decrypt that file. This application is fascinating because it provides the security of asymmetric encryption while retaining the encryption/decryption speed of symmetric encryption.

Later, when I reread this passage, it suddenly struck me that this is a very interesting idea. I searched around and it seemed no one had done it, so I decided to roll up my sleeves and try it myself. Conveniently, there is an excellent implementation of the Noise Protocol in Rust: snow. Libra also uses this library (through rust-libp2p), so I set out to experiment with snow.

Choosing a Pattern

Section 7.4 of the Noise Protocol mentions several “one-way handshakes”: N, K, and X:

• N: the sender does not use a static key pair.
• K: the receiver knows the sender’s static public key in advance.
• X: the sender’s static key pair is transmitted to the receiver encrypted over the network.

In any of these, the sender needs the receiver’s static public key during the handshake. These patterns differ only in how the sender’s public key is provided to the receiver. Pattern N is essentially the most traditional DH-based public-key encryption algorithm; it does not authenticate the sender—even if the receiver can decrypt the content she receives, she cannot confirm that its source is trustworthy, so it is not perfect. Patterns K and X do not have this problem, but Pattern K places too high a requirement on the receiver, limiting its practical use. Therefore, Pattern X, which transmits the public key during the handshake, became my first choice (and, in fact, my only choice).

Pattern X’s handshake:

<- s
...
-> e, es, s, ss

Note that although we use the word “handshake” here, it is actually decided unilaterally by the sender; there is no real handshake process. The sender only needs to generate the public key of an ephemeral key e, perform DH between e and the other party’s public key, use the DH result to encrypt its own public key, and then perform DH between its own public key and the other party’s public key. The key generated throughout this process is the key needed for the subsequent encryption.

How Do We Encrypt Files with This Pattern?

The main application of the Noise Protocol is securing network traffic, so the largest message the protocol can handle at a time is 65,535 bytes. Why? Because that is the maximum length of a logical IP packet (physically, depending on the MTU—Maximum Transmit Unit—an IP packet is split into several fragments; Ethernet’s MTU, for example, is 1,500). Since the encrypted message adds 16 bytes of AEAD (Authenticated Encryption with Associated Data) authentication data compared with the plaintext, the actual pre-encryption size is 65,535 − 16. That means a file must be split into chunks of 65,519 bytes to satisfy the maximum encrypted length.

Thus we arrive at an encryption flow like this:

Now the question is: the data is encrypted, but how does the receiver decrypt it? This requires careful thought.

The best idea I can come up with is to serialize the handshake information generated by the sender during the handshake and write it into the encrypted file header. That way, the receiver can read the handshake information, establish the corresponding handshake state, then switch to transport mode and decrypt.

How Should the Encrypted File Header Be Designed?

Considering potential future upgrades, I used protobuf for the file header. I also considered capnp and flatbuffers, which have some hard-to-resist advantages (such as zero-copy, and the ability to read parts of the content locally rather than having to read the entire structure before decoding, as protobuf requires). But first, protobuf is simple and direct; second, it makes the message more secure (after all, it needs encode/decode preprocessing); and third, for my application, protobuf’s efficiency disadvantage is almost negligible. So I used protobuf, defined as follows:

syntax = "proto3";
package header;

enum CipherMode {
  ChaChaPoly = 0;
  AESGCM = 1;
}

enum HashMode {
  BLAKE2b = 0;
  BLAKE2s = 1;
  SHA512 = 2;
  SHA256 = 3;
}

// this is the file header for encrypted files. Note that we deliberately fixed
// the DH function to ed25519 since it provides same level of security as ed448,
// but performance is better. The default choise of if all the parameters are
// default value: Noise_X_25519_ChaChaPoly_BLAKE2b. This fits the most of the
// security standards. However, initiator and responder can use a pre shared key
// for extra level of security.
message Header {
  // cipher function used by noise protocol
  CipherMode cipher = 1;
  // hash function used by noise protocol
  HashMode hash = 2;
  // if true, we will do a Xpsk1. Note user shall provide the psk upon Session::new.
  // <- s
  // ...
  // -> e, es, s, ss, psk
  bool use_psk = 3;
  // handshake message from the initiator
  bytes handshake_message = 4;
}

handshake_message here is the handshake message generated by the sender.

Some readers may wonder: is it insecure to put the handshake message in the file header?

No problem. First, the handshake process is originally transmitted over the insecure internet; putting it in the file header is just as insecure as putting it on the network. Second, what is transmitted during the handshake is specified by the protocol. Anyone can parse out the public key of the ephemeral key pair e, but since they do not have the private key, they cannot compute DH(e, s), and therefore cannot derive any of the subsequent steps. So this process is secure.

How Does the Receiver Decrypt?

If you have followed this far, the decryption flow is presumably already clear to you:

Comparison with Traditional Symmetric Encryption

The biggest problem with traditional symmetric encryption (such as AES-256 or ChaChaPoly) is the key: if the same key is used for every encryption, the leakage or brute-force cracking of any one key will leave all encrypted files using that key unprotected; if a different key is used each time, the burden of key management and decryption increases greatly (you need to know which key corresponds to which file). In contrast, if you use the Noise Protocol or directly use a DH algorithm to “negotiate” the key required for the ciphertext unilaterally, each encryption generates a completely random key. Therefore, as long as the receiver’s private key is not leaked, both security and usability are better than traditional approaches. In addition, the receiver can verify that the ciphertext comes from a sender it trusts. The only drawback of this approach is that the ciphertext is longer than the plaintext by a file header of about 100 bytes plus an HMAC for every 64 KB. But this is an acceptable compromise.

Source Code

If you want to see the concrete implementation, go to: github.com/tyrchen/conceal. This code was written in mid-December last year. Later, I wanted to use mmap to optimize file read/write efficiency, but I got stuck on the mmap write problem and never had time to solve it (see the Pull Request), so I put it aside. The unoptimized code on master works fine, for your reference.