Last active
October 29, 2017 09:45
-
-
Save pnarayanan/40478d72249f4645f5afbbabd7c4eef5 to your computer and use it in GitHub Desktop.
Gist of the change within Message Format to support encryption at rest
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To support encryption, Message Format associated with messages now needs to store the encryption key. | |
Since we need to encrypt/decrypt both the User Metadata and the Blob payload, we have two choices: | |
a) Duplicate the key within both the User Metadata subrecord and Blob subrecord. This is the least | |
invasive change and has some merits, but has the burden of duplication of keys. | |
b) Create a new Blob Encryption Key record and store the key just once. | |
We decided to go with the second approach that avoids duplication and lends itself better to potential | |
use cases such as key replacements. | |
This patch involves adding a new Blob Encryption Key Record and a new version of the Header as shown | |
at the end of this document. | |
One additional change is the ability to associate Message Format specific metadata with outgoing messages | |
from the server. This is added for the following reason: Since a GetRequest can request multiple keys | |
from multiple partitions in the same request, a GetResponse can consist of multiple PartitionResponseInfos, | |
each containing multiple messages within it. The receiver of the GetResponse needs to know whether and | |
which of the messages in a PartitionResponseInfo are encrypted. The metadata associated at the parition | |
level is therefore not sufficient to convey this information, something at the message level is required. | |
One option is to embed this information in the message level MessageInfo that gets sent with the response | |
to also include this info. However, MessageInfo is used widely within the store among other places and | |
keeps information available exclusively from the Index, and the message format metadata is out of place. | |
This patch therefore introduces a MessageMetadata class which will contain MessageFormat specific metadata | |
associated with any message. This will be a versioned object. Additionally, rather than simply conveying | |
that a message is encrypted or not, and sending the encryption key as part of the zero-copy, an optimization | |
is made to actually store the encryption key within the MessageMetadata itself, if there is one. This | |
means that within MessageFormatSend, we will only do zero copies for the actual payload | |
(usermetadata/blob property/blob info/blob data). The encryption key record will be deserialized while crafting | |
the response, which is cheap and keeps things simple. | |
The new formats are the following: | |
BlobEncryptionKey_Format_V1: | |
* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | |
* | | | | | | |
* | version | size | blob encryption key | Crc | | |
* |(2 bytes)| (4 bytes) | (n bytes) | (8 bytes) | | |
* | | | | | | |
* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | |
* version - The version of the blob key | |
* | |
* size - The size of the blob key | |
* | |
* blob encryption key - The blob encryption key in bytes | |
* | |
* crc - The crc of the blob key record | |
* | |
MessageHeader_Format_V2: | |
* | |
* - - - - - - - - - - - - - - - - - - -- - -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | |
* | | | | | | | | | | |
* | version | payload size | Blob Encryption | Blob Property | Delete | User Metadata | Blob | Crc | | |
* |(2 bytes)| (8 bytes) | Key Relative | Relative | Relative | Relative | Relative | (8 bytes) | | |
* | | | Offset | Offset | Offset | Offset | Offset | | | |
* | | | (4 bytes) | (4 bytes) | (4 bytes) | (4 bytes) | (4 bytes) | | | |
* | | | | | | | | | | |
* - - - - - - - - - - - - - - - - - - -- - -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | |
* | |
* version - The version of the message header | |
* | |
* payload size - The size of the message payload. | |
* Blob Encryption Key Record Size (if present) + (Blob prop record size or delete record size) + | |
* user metadata size + blob size | |
* | |
* Blob Encryption - The offset at which the blob encryption key record is located relative to this message. | |
* Key relative Non-existence of blob key record is indicated by -1. Blob Keys are optionally present for Put | |
* offset records. Blob Keys will be absent for Delete records. | |
* | |
* blob property - The offset at which the blob property record is located relative to this message. Only one of | |
* relative offset blob property/delete relative offset field can exist. Non existence is indicated by -1 | |
* | |
* delete - The offset at which the delete record is located relative to this message. Only one of blob | |
* relative offset property/delete relative offset field can exist. Non existence is indicated by -1 | |
* | |
* user metadata - The offset at which the user metadata record is located relative to this message. This exist | |
* relative offset only when blob property record and blob record exist | |
* | |
* blob metadata - The offset at which the blob record is located relative to this message. This exist only when | |
* relative offset blob property record and user metadata record exist | |
* | |
* crc - The crc of the message header | |
* | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment