parquet-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From b...@apache.org
Subject [parquet-format] branch encryption updated: PARQUET-1450: Detailed crypto specification (#114)
Date Thu, 17 Jan 2019 00:33:39 GMT
This is an automated email from the ASF dual-hosted git repository.

blue pushed a commit to branch encryption
in repository https://gitbox.apache.org/repos/asf/parquet-format.git


The following commit(s) were added to refs/heads/encryption by this push:
     new 41b9701  PARQUET-1450: Detailed crypto specification (#114)
41b9701 is described below

commit 41b970156f223ab68b594a625d094759f3e41b6b
Author: ggershinsky <ggershinsky@users.noreply.github.com>
AuthorDate: Thu Jan 17 02:33:35 2019 +0200

    PARQUET-1450: Detailed crypto specification (#114)
---
 Encryption.md                         | 675 ++++++++++++++++++++++++----------
 README.md                             |   8 +-
 doc/images/FileLayoutEncryptionEF.jpg | Bin 123128 -> 0 bytes
 doc/images/FileLayoutEncryptionEF.png | Bin 0 -> 81119 bytes
 doc/images/FileLayoutEncryptionPF.jpg | Bin 117345 -> 0 bytes
 doc/images/FileLayoutEncryptionPF.png | Bin 0 -> 79304 bytes
 6 files changed, 481 insertions(+), 202 deletions(-)

diff --git a/Encryption.md b/Encryption.md
index 156d9ce..63dfadf 100644
--- a/Encryption.md
+++ b/Encryption.md
@@ -19,216 +19,495 @@
 
 # Parquet Modular Encryption
 
-Parquet files, containing sensitive information, can be protected by the modular
-encryption mechanism, that encrypts and authenticates the file data and metadata - 
-while allowing for a regular Parquet functionality (columnar projection, 
-predicate pushdown, encoding and compression). The mechanism also enables column access 
-control, via support for encryption of different columns with different keys.
-
-## Problem Statement
-The existing data protection solutions (such as flat encryption of files, in-storage 
-encryption, or a use of an encrypting storage client) can be applied to Parquet files,
-but have various security or performance issues. An encryption mechanism, integrated in
-the Parquet format, allows for an optimal combination of data security, processing
-speed and access control granularity.
-
-
-## Goals
+Parquet files containing sensitive information can be protected by the modular encryption

+mechanism that encrypts and authenticates the file data and metadata - while allowing 
+for a regular Parquet functionality (columnar projection, predicate pushdown, encoding 
+and compression). 
+
+## 1 Problem Statement
+Existing data protection solutions (such as flat encryption of files, in-storage encryption,

+or use of an encrypting storage client) can be applied to Parquet files, but have various

+security or performance issues. An encryption mechanism, integrated in the Parquet format,

+allows for an optimal combination of data security, processing speed and encryption granularity.
+
+## 2 Goals
 1. Protect Parquet data and metadata by encryption, while enabling selective reads 
 (columnar projection, predicate push-down).
 2. Implement "client-side" encryption/decryption (storage client). The storage server 
 must not see plaintext data, metadata or encryption keys.
-3. Leverage authenticated encryption that allows clients to check integrity of the 
-retrieved data - making sure the file (or file parts) had not been replaced with a 
-wrong version, or tampered with otherwise.
-4. Support column access control - by enabling different encryption keys for different 
-columns, and for the footer.
+3. Leverage authenticated encryption that allows clients to check integrity of the retrieved

+data - making sure the file (or file parts) have not been replaced with a wrong version,
or 
+tampered with otherwise.
+4. Enable different encryption keys for different columns and for the footer.
 5. Allow for partial encryption - encrypt only column(s) with sensitive data.
 6. Work with all compression and encoding mechanisms supported in Parquet.
-7. Support multiple encryption algorithms, to account for different security and 
-performance requirements.
-8. Enable two modes for metadata protection:
+7. Support multiple encryption algorithms, to account for different security and performance

+requirements.
+8. Enable two modes for metadata protection -
    * full protection of file metadata
-   * partial protection of file metadata, that allows legacy readers to access unencrypted

- columns in an encrypted file.
-9. Miminize overhead of encryption: in terms of size of encrypted files, and throughput
+   * partial protection of file metadata that allows legacy readers to access unencrypted

+columns in an encrypted file.
+9.	Minimize overhead of encryption - in terms of size of encrypted files, and throughput

 of write/read operations.
 
 
-## Technical Approach
-
-Each Parquet module (footer, page headers, pages, column indexes, column metadata) is 
-encrypted separately. Then it is possible to fetch and decrypt the footer, find the 
-offset of a required page, fetch it and decrypt the data. In this document, the term 
-“footer” always refers to the regular Parquet footer - the `FileMetaData` structure,
and 
-its nested fields (row groups / column chunks).
-
-The results of compression of column pages are encrypted, before being written to the 
-output stream. A new Thrift structure, with a column crypto metadata, is added to 
-column chunks of the encrypted columns. This metadata provides information about the 
-column encryption keys.
-
-The results of Thrift serialization of metadata structures are encrypted, before being 
-written to the output stream.
-
-## Encryption algorithms
-
-Parquet encryption algorithms are based on the standard AES ciphers for symmetric 
-encryption. AES is supported in Intel and other CPUs with hardware acceleration of 
-crypto operations (“AES-NI”) - that can be leveraged by e.g. Java programs 
-(automatically via HotSpot), or C++ programs (via EVP-* functions in OpenSSL).
-
-Initially, two algorithms are implemented, one based on a GCM mode of AES, and the other

-on a combination of GCM and CTR modes.
-
-AES-GCM is an authenticated encryption. Besides the data confidentiality (encryption), it

-supports two levels of integrity verification / authentication: of the data (default), and

-of the data combined with an optional AAD (“additional authenticated data”). The default

-authentication allows to make sure the data has not been tampered with. An AAD is a free

-text to be signed, together with the data. The user can, for example, pass the file name

-with its version (or creation timestamp) as the AAD, to verify the file has not been 
-replaced with an older version.
-
-Sometimes, a hardware acceleration of AES is unavialable (e.g. in Java 8). Then AES crypto

-operations are implemented in software, and can be somewhat slow, becoming a performance

-bottleneck in certain workloads. AES-CTR is a regular (not authenticated) cipher.
-It is faster than AES-GCM, since it doesn’t perform integrity verification and doesn’t

-calculate the authentication tag. For applications running without AES acceleration and 
-willing to compromise on content verification, AES-CTR can provide a boost in Parquet 
-write/read throughput. The second Parquet algorithm encrypts the data content (pages) 
-with AES-CTR, and the metadata (Thrift structures) with AES-GCM. This allows to encrypt/decrypt

-the bulk of the data faster, while still verifying the metadata integrity and making sure

-the file had not been replaced with a wrong version. However, tampering with the page data

-might go unnoticed. 
+## 3 Technical Approach
+Parquet files are comprised of separately serialized components: pages, page headers, column

+indexes, offset indexes, a footer. Parquet encryption mechanism denotes them as “modules”

+and encrypts each module separately – making it possible to fetch and decrypt the footer,

+find the offset of required pages, fetch the pages and decrypt the data. In this document,

+the term “footer” always refers to the regular Parquet footer - the `FileMetaData` structure,

+and its nested fields (row groups / column chunks). 
 
-The `AesGcmV1` and `AesGcmCtrV1` structures contain an optional `aad_metadata` field that
can 
-be used by a reader to retrieve the AAD string used for file encryption. The maximal allowed
-length of `aad_metadata` is 512 bytes.
-
-Parquet-mr/-cpp implementations use the RBG-based IV construction as defined in the NIST

-SP 800-38D document for the GCM ciphers (section 8.2.2).
-
-
-### AES_GCM_V1
-All modules are encrypted with the AES-GCM cipher. The authentication tags (16 bytes) are

-written after each ciphertext. The IVs (12 bytes) are written before each ciphertext.
-
-### AES_GCM_CTR_V1
-Thrift modules are encrypted with the AES-GCM cipher, as described above. 
-The pages are encrypted with AES-CTR, where the IVs (16 bytes) are written before each 
-ciphertext.
-
-
-## File Format
-
-The encrypted Parquet files have a different extension - “.parquet.encrypted”.
-
-The encryption is flexible - each column and the footer can be encrypted with the same key,

+File encryption is flexible - each column and the footer can be encrypted with the same key,

 with a different key, or not encrypted at all.
 
-The metadata structures (`PageHeader`, `ColumnIndex`, `OffsetIndex`; and sometimes `FileMetaData`
and 
-`ColumnMetaData`, see below) are encrypted after Thrift serialization. For each structure,

-the encryption buffer is comprised of an IV, ciphertext and tag, as described in the 
-Algorithms section. The length of the encryption buffer (a 4-byte little endian) is 
-written to the output stream, followed by the buffer itself.
-
-The column pages (data and dictionary) are encrypted after compression. For each page, 
-the encryption buffer is comprised of an IV, ciphertext and (in case of AES_GCM_V1) of a

-tag, as described in the Algorithms section. Only the buffer is written to the output 
-stream - not need to write the length of the buffer, since the length (size of the page after
-compression and encryption) is kept in the page headers.
-
-A `crypto_meta_data` field in set in each `ColumnChunk` in the encrypted columns. 
-`ColumnCryptoMetaData` is a union - the actual structure is chosen depending on whether the

-column is encrypted with the footer key, or with a column-specific key. For the latter, 
-a key metadata can be specified, with a maximal length of 512. Key metadata is a free-form
-byte array that can be used by a reader to retrieve the column encryption key. 
-
-Parquet file footer, and its nested structures, contain sensitive information - ranging 
+The results of compression of column pages are encrypted before being written to the output

+stream. A new Thrift structure, with column crypto metadata, is added to column chunks of

+the encrypted columns. This metadata provides information about the column encryption keys.
+
+The results of serialization of Thrift structures are encrypted, before being written 
+to the output stream. 
+
+The file footer can be either encrypted or left as a plaintext. In an encrypted footer mode,

+a new Thrift structure with file crypto metadata is added to the file. This metadata provides

+information about the file encryption algorithm and the footer encryption key. 
+
+In a plaintext footer mode, the contents of the footer structure is visible and signed 
+in order to verify its integrity. New footer fields keep an
+information about the file encryption algorithm and the footer signing key.
+
+For encrypted columns, the following modules are always encrypted, with the same column key:

+pages and  page headers (both dictionary and data), column indexes, offset indexes.  If the

+column key is different from the footer encryption key, the column metadata is serialized

+separately and encrypted with the column key. In this case, the column metadata is also 
+considered to be a module.  
+
+There are two module types: data modules (pages) and Thrift modules (all Thrift structures
that 
+are serialized separately).
+
+
+## 4 Encryption Algorithms and Keys
+Parquet encryption algorithms are based on the standard AES ciphers for symmetric encryption.

+AES is supported in Intel and other CPUs with hardware acceleration of crypto operations

+(“AES-NI”) - that can be leveraged, for example, by Java programs (automatically via
HotSpot), 
+or C++ programs (via EVP-* functions in OpenSSL). Parquet supports all standard AES key sizes:

+128, 192 and 256 bits. 
+
+Initially, two algorithms have been implemented, one based on a GCM mode of AES, and the

+other on a combination of GCM and CTR modes.
+
+### 4.1 AES modes used in Parquet
+
+#### 4.1.1 AES GCM
+AES GCM is an authenticated encryption. Besides the data confidentiality (encryption), it

+supports two levels of integrity verification (authentication): of the data (default), 
+and of the data combined with an optional AAD (“additional authenticated data”). The

+authentication allows to make sure the data has not been tampered with. An AAD 
+is a free text to be authenticated, together with the data. The user can, for example, pass
the 
+file name with its version (or creation timestamp) as an AAD input, to verify that the 
+file has not been replaced with an older version. The details on how Parquet creates 
+and uses AADs are provided in the section 4.4.
+
+#### 4.1.2 AES CTR
+AES CTR is a regular (not authenticated) cipher. It is faster than the GCM cipher, since
it 
+doesn’t perform integrity verification and doesn’t calculate an authentication tag. 
+Actually, GCM is a combination of the CTR cipher and an 
+authentication layer called GMAC. For applications running without AES acceleration 
+(e.g. on Java versions before Java 9) and willing to compromise on content verification,

+CTR cipher can provide a boost in encryption/decryption throughput.
+
+
+#### 4.1.3 Nonces and IVs
+GCM and CTR ciphers require a unique vector to be provided for each encrypted stream. 
+In this document, the unique input to GCM encryption is called nonce (“number used once”).
+The unique input to CTR encryption is called IV ("initialization vector"), and is comprised
of two 
+parts: a nonce and an initial counter field. 
+
+Parquet encryption uses the RBG-based (random bit generator) nonce construction as defined
in 
+the section 8.2.2 of the NIST SP 800-38D document. For each encrypted module, Parquet generates
a 
+unique nonce with a length of 12 bytes (96 bits). Notice: the NIST 
+specification uses a term “IV” for what is called “nonce” in the Parquet encryption
design.
+
+
+### 4.2 Parquet encryption algorithms
+
+#### 4.2.1 AES_GCM_V1
+This Parquet algorithm encrypts all modules by the GCM cipher, without padding. The AES GCM
cipher
+must be implemented by a cryptographic provider according to the NIST SP 800-38D specification.

+
+In Parquet, an input to the GCM cipher is an encryption key, a 12-byte nonce, a plaintext
and an 
+AAD. The output is a ciphertext with the length equal to that of plaintext, and a 16-byte
authentication 
+tag used to verify the ciphertext and AAD integrity.
+
+
+#### 4.2.2 AES_GCM_CTR_V1
+In this Parquet algorithm, all Thrift modules are encrypted with the GCM cipher, as described

+above, but the pages are encrypted by the CTR cipher without padding. This allows to encrypt/decrypt

+the bulk of the data faster, while still verifying the metadata integrity and making 
+sure the file has not been replaced with a wrong version. However, tampering with the 
+page data might go unnoticed. The AES CTR cipher
+must be implemented by a cryptographic provider according to the NIST SP 800-38A specification.

+
+In Parquet, an input to the CTR cipher is an encryption key, a 16-byte IV and a plaintext.
IVs are comprised of 
+a 12-byte nonce and a 4-byte initial counter field. The first 31 bits of the initial counter
field are set 
+to 0, the last bit is set to 1. The output is a ciphertext with the length equal to that
of plaintext.
+
+### 4.3 Key metadata
+A wide variety of services and tools for management of encryption keys exist in the 
+industry today. Public clouds offer different key management services (KMS), and 
+organizational IT systems either build proprietary key managers in-house or adopt open source

+tools for on-premises deployment. Besides the diversity of management tools, there are many

+ways to generate and handle the keys themselves (generate Data keys inside KMS – or locally

+upon data encryption; use Data keys only, or use Master keys to encrypt the Data keys; 
+store the encrypted key material inside the data file, or at a separate location; etc). There

+is also a large variety of authorization and certification methods, required to control the

+access to encryption keys.
+
+Parquet is not limited to a single KMS, key generation/wrapping method, or authorization
service. 
+Instead, Parquet provides a developer with a simple interface that can be utilized for implementation

+of any key management scheme. For each column or footer key, a file writer can generate and
pass an 
+arbitrary `key_metadata` byte array that will be stored in the file. This field is made available
to 
+file readers to enable recovery of the key. For example, the key_metadata 
+can keep a serialized
+
+   * String ID of a Data key. This enables direct retrieval of the Data key from a KMS.
+   * Encrypted Data key, and string ID of a Master key. The Data key is generated randomly
and 
+   encrypted with a Master key either remotely in a KMS, or locally after retrieving the
Master key from a KMS.
+   Master key rotation requires modification of the data file footer.
+   * Short ID (counter) of a Data key inside the Parquet data file. The Data key is encrypted
with a 
+   Master key using one of the options described above – but the resulting key material
is stored 
+   separately, outside the data file, and will be retrieved using the counter and file path.
+   Master key rotation doesn't require modification of the data file.
+   
+Key metadata can also be empty - in a case the encryption keys are fully managed by the caller

+code, and passed explicitly to Parquet readers for the file footer and each encrypted column.
+
+### 4.4 Additional Authenticated Data
+The AES GCM cipher protects against byte replacement inside a ciphertext - but, without an
AAD, 
+it can't prevent replacement of one ciphertext with another (encrypted with the same key).

+Parquet modular encryption leverages AADs to protect against swapping ciphertext modules
(encrypted 
+with AES GCM) inside a file or between files. Parquet can also protect against swapping full

+files - for example, replacement of a file with an old version, or replacement of one table

+partition with another. AADs are built to reflects the identity of a file and of the modules

+inside the file. 
+
+Parquet constructs a module AAD from two components: an optional AAD prefix - a string provided

+by the user for the file, and an AAD suffix, built internally for each GCM-encrypted module

+inside the file. The AAD prefix should reflect the target identity that helps to detect file

+swapping (a simple example - table name with a date and partition, e.g. "employees_23May2018.part0").

+The AAD suffix reflects the internal identity of modules inside the file, which for example

+prevents replacement of column pages in row group 0 by pages from the same column in row

+group 1. The module AAD is a direct concatenation of the prefix and suffix parts. 
+
+#### 4.4.1 AAD prefix
+File swapping can be prevented by an AAD prefix string, that uniquely identifies the file
and 
+allows to differentiate it e.g. from older versions of the file or from other partition files
in the same 
+data set (table). This string is optionally passed by a writer upon file creation. If provided,
+the AAD prefix is stored in an `aad_prefix` field in the file, and is made available to the
readers. 
+This field is not encrypted. If a user is concerned about keeping the file identity inside
the file, 
+the writer code can explicitly request Parquet not to store the AAD prefix. Then the aad_prefix
field 
+will be empty; AAD prefixes must be fully managed by the caller code and supplied explictly
to Parquet 
+readers for each file.
+
+The protection against swapping full files is optional. It is not enabled by default because

+it requires the writers to generate and pass an AAD prefix.
+
+A reader of a file created with an AAD prefix, should be able to verify the prefix (file
identity)
+by comparing it with e.g. the target table name, using a convention accepted in the organization.
+Readers of data sets, comprised of multiple partition files, can verify data set integrity
by 
+checking the number of files and the AAD prefix of each file. For example, a reader that
needs to 
+process the employee table, a May 23 version, knows (via the convention) that 
+the AAD prefix must be "employees_23May2018.partN" in 
+each corresponding table file. If a file AAD prefix is "employees_23May2018.part0", the reader

+will know it is fine, but if the prefix is "employees_23May2016.part0" or "contractors_23May2018.part0"
- 
+the file is wrong. The reader should also know the number of table partitions and verify
availability 
+of all partition files (prefixes) from 0 to N-1.
+
+   
+#### 4.4.2 AAD suffix
+The suffix part of a module AAD protects against module swapping inside a file. It also protects
against 
+module swapping between files  - in situations when an encryption key is re-used in multiple
files and the 
+writer has not provided a unique AAD prefix for each file. 
+
+Unlike AAD prefix, a suffix is built internally by Parquet, by direct concatenation of the
following parts: 
+1.	[All modules] internal file identifier - a random byte array generated for each file (implementation-defined
length)
+2.	[All modules] module type (1 byte)
+3.	[All modules except footer] row group ordinal (2 byte short, little endian)
+4.	[All modules except footer] column ordinal (2 byte short, little endian)
+5.	[Data page and header only] page ordinal (2 byte short, little endian)
+
+The following module types are defined:  
+
+   * Footer (0)
+   * ColumnMetaData (1)
+   * Data Page (2)
+   * Dictionary Page (3)
+   * Data PageHeader (4)
+   * Dictionary PageHeader (5)
+   * ColumnIndex (6)
+   * OffsetIndex (7)
+
+
+|                      | Internal File ID | Module type | Row group ordinal | Column ordinal
| Page ordinal|
+|----------------------|------------------|-------------|-------------------|----------------|-------------|
+| Footer               |       yes        |   yes (0)   |        no         |      no   
    |     no      |
+| ColumnMetaData       |       yes        |   yes (1)   |        yes        |      yes  
    |     no      |
+| Data Page            |       yes        |   yes (2)   |        yes        |      yes  
    |     yes     |
+| Dictionary Page      |       yes        |   yes (3)   |        yes        |      yes  
    |     no      |
+| Data PageHeader      |       yes        |   yes (4)   |        yes        |      yes  
    |     yes     |
+| Dictionary PageHeader|       yes        |   yes (5)   |        yes        |      yes  
    |     no      |
+| ColumnIndex          |       yes        |   yes (6)   |        yes        |      yes  
    |     no      |
+| OffsetIndex          |       yes        |   yes (7)   |        yes        |      yes  
    |     no      |
+
+
+
+## 5 File Format
+
+### 5.1 Encrypted module serialization
+The Thrift modules are encrypted with the GCM cipher. In the AES_GCM_V1 algorithm, 
+the column pages (data modules) are also encrypted with AES GCM. For each module, the GCM
encryption 
+buffer is comprised of a nonce, ciphertext and tag, described in the Algorithms section.
The length of 
+the encryption buffer (a 4-byte little endian) is written to the output stream, followed
by the buffer itself.
+
+|length (4 bytes) | nonce (12 bytes) | ciphertext (length-28 bytes) | tag (16 bytes) |
+|-----------------|------------------|------------------------------|----------------|
+
+
+In the AES_GCM_CTR_V1 algorithm, the column pages are encrypted with AES CTR.
+For each page, the CTR encryption buffer is comprised of a nonce and ciphertext, 
+described in the Algorithms section. The length of the encryption buffer 
+(a 4-byte little endian) is written to the output stream, followed by the buffer itself.
+
+|length (4 bytes) | nonce (12 bytes) | ciphertext (length-12 bytes) |
+|-----------------|------------------|------------------------------|
+
+
+### 5.2 Crypto structures
+Parquet file encryption algorithm is specified in a union of the following Thrift structures:
+
+```c
+struct AesGcmV1 {
+  /** AAD prefix **/
+  1: optional binary aad_prefix
+
+  /** Unique file identifier part of AAD suffix **/
+  2: optional binary aad_file_unique
+  
+  /** In files encrypted with AAD prefix without storing it,
+   * readers must supply the prefix **/
+  3: optional bool supply_aad_prefix
+}
+
+struct AesGcmCtrV1 {
+  /** AAD prefix **/
+  1: optional binary aad_prefix
+
+  /** Unique file identifier part of AAD suffix **/
+  2: optional binary aad_file_unique
+  
+  /** In files encrypted with AAD prefix without storing it,
+   * readers must supply the prefix **/
+  3: optional bool supply_aad_prefix
+}
+
+union EncryptionAlgorithm {
+  1: AesGcmV1 AES_GCM_V1
+  2: AesGcmCtrV1 AES_GCM_CTR_V1
+}
+```
+
+If a writer provides an AAD prefix, it will be used for enciphering the file and stored in
the 
+`aad_prefix` field. However, the writer can request Parquet not to store the prefix in the
file. In 
+this case, the `aad_prefix` field will not be set, and the `supply_aad_prefix` field will
be set 
+to _true_ to inform readers they must supply the AAD prefix for this file in order to be
able to 
+decrypt it.
+
+The row group ordinal, required for AAD suffix calculation, is set in the RowGroup structure:
+
+```c
+struct RowGroup {
+...
+  /** Row group ordinal in the file **/
+  7: optional i16 ordinal
+}
+```
+
+A `crypto_metadata` field is set in each ColumnChunk in the encrypted columns. ColumnCryptoMetaData

+is a union - the actual structure is chosen depending on whether the column is encrypted
with the 
+footer encryption key, or with a column-specific key. For the latter, a key metadata can
be specified.
+
+```c
+struct EncryptionWithFooterKey {
+}
+
+struct EncryptionWithColumnKey {
+  /** Column path in schema **/
+  1: required list<string> path_in_schema
+  
+  /** Retrieval metadata of column encryption key **/
+  2: optional binary key_metadata
+}
+
+union ColumnCryptoMetaData {
+  1: EncryptionWithFooterKey ENCRYPTION_WITH_FOOTER_KEY
+  2: EncryptionWithColumnKey ENCRYPTION_WITH_COLUMN_KEY
+}
+
+struct ColumnChunk {
+...
+  /** Crypto metadata of encrypted columns **/
+  8: optional ColumnCryptoMetaData crypto_metadata
+}
+```
+
+
+### 5.3 Protection of sensitive metadata
+The Parquet file footer, and its nested structures, contain sensitive information - ranging

 from a secret data (column statistics) to other information that can be exploited by an 
-attacker (e.g. schema, num_values, key_value_metadata, column data offset and size, encoding
and crypto_meta_data). 
-This information is automatically protected when the footer and secret columns are encrypted

-with the same key. In other cases - when column(s) and the footer are encrypted with 
-different keys; or column(s) are encrypted and the footer is not - an extra measure is 
-required to protect the column-specific information in the file footer. In these cases, 
-the column-specific information (kept in `ColumnMetaData` structures) is split from the 
-footer, by utilizing the `required i64 file_offset` parameter in the `ColumnChunk` 
-structure. This allows to serialize each `ColumnMetaData` structure separately, and encrypt

-it with a column-specific key, thus protecting the column stats and other metadata. 
-
-### Encrypted footer mode
-
+attacker (e.g. schema, num_values, key_value_metadata, encoding 
+and crypto_metadata). This information is automatically protected when the footer and 
+secret columns are encrypted with the same key. In other cases - when column(s) and the 
+footer are encrypted with different keys; or column(s) are encrypted and the footer is not,

+an extra measure is required to protect the column-specific information in the file footer.

+In these cases, the `ColumnMetaData` structures are Thrift-serialized separately and encrypted

+with a column-specific key, thus protecting the column stats and 
+other metadata. The column metadata module is encrypted with the GCM cipher, serialized 
+according to the section 5.1 instructions and stored in an `optional binary encrypted_column_metadata`

+field in the `ColumnChunk`.
+
+```c
+struct ColumnChunk {
+...
+  
+  /** Column metadata for this chunk.. **/
+  3: optional ColumnMetaData meta_data
+..
+  /** Crypto metadata of encrypted columns **/
+  8: optional ColumnCryptoMetaData crypto_metadata
+  
+  /** Encrypted column metadata for this chunk **/
+  9: optional binary encrypted_column_metadata
+}
+```
+
+
+### 5.4 Encrypted footer mode
 In files with sensitive column data, a good security practice is to encrypt not only the

-secret columns, but also the file footer metadata, with a separate footer key. This hides
-the file schema / column names, number of rows, key-value properties, column sort order,

-column data offset and size, list of encrypted columns and metadata of the column encryption
keys. 
-It also makes the footer tamper-proof.
-
-The columns encrypted with the same key as the footer, don't split the ColumnMetaData from
the 
-ColumnChunks, leaving it at the original location, `optional ColumnMetaData meta_data`. This
field
-is not set for columns enrypted with a column-specific key.
-
-A Thrift-serialized `FileCryptoMetaData` structure is written after the footer. It contains

-information on the file encryption algorithm and on the footer (offset in 
-the file, and an optional key metadata, with a maximal length of 512). Then 
-the length of this structure is written, as a 4-byte little endian integer. Then a final

-magic string, "PARE".
-
-Only the `FileCryptoMetaData` is written as a plaintext, all other file parts are protected
-(as needed) with appropriate keys.
-
- ![File Layout - Encrypted footer](doc/images/FileLayoutEncryptionEF.jpg)
-
-### Plaintext footer mode
-
-This mode allows legacy Parquet versions (released before the encryption support) to access
unencrypted 
-columns in encrypted files - at a price of leaving certain metadata fields unprotected in
these files 
-(not encrypted or tamper-proofed). The plaintext footer mode can be useful during a transitional
period 
-in organizations 
-where some frameworks can't be upgraded to a new Parquet library for a while. Data writers
will
-upgrade and run with a new Parquet version, producing encrypted files in this mode. Data
readers, 
-working with a sensitive data, will also upgrade to a new Parquet library. But other readers
that
-don't need the sensitive columns, can continue working with an older Parquet version. They
will be 
-able to access plaintext columns in encrypted files. A legacy reader, trying to access a
sensitive 
-column data in a ".parquet.encrypted" file with a plaintext footer, will get an  exception.
More
-specifically, a Thrift parsing exception on an encrypted `PageHeader` structure. Again, using
legacy
-Parquet readers for encrypted files is a temporary solution.
-
-In the plaintext footer mode, the `optional ColumnMetaData meta_data` is set in the `ColumnChunk`

-structure for all columns, but is stripped of the statistics for the sensitive (encrypted)
columns. 
-These statistics are available for new readers with the column key - they fetch the split
ColumnMetaData, 
-and decrypt it to get all metadata values. The legacy readers are not aware of the split
metadata, 
-they parse the embedded field as usual. While they can't read the data of the encrypted columns,
they 
-read the metadata to exract the offset and size of the column data - required for input vectorization
-(see the next section).
-
-An `encryption_algorithm` field is set at the FileMetaData structure. Then the footer is
written as usual, 
-followed by its length (4-byte little endian integer) and a final magic string, "PAR1".
-
- ![File Layout: Plaintext footer](doc/images/FileLayoutEncryptionPF.jpg)
-
-
-### New fields for vectorized readers
-
-Apache Spark and other vectorized readers slice a file by using the information on offset
-and size of each row group. In the legacy readers, this is done by running over a list of
all column chunks
-in a row group, reading the relevant information from the column metadata, adding up the
size values
-and picking the offset of the first column as the row group offset. However, vectorization
-needs only a row group metadata, not metadata of individual columns. Also, in files written
in an
-encrypted footer mode, the column metadata is not available to readers without the column
key. Therefore, 
-two new fields are added to the
-`RowGroup` structure - `file_offset` and `total_compressed_size` - that are set upon file
-writing, and allow vectorized readers to query a file even if keys to certain columns are
-not available ('hidden columns'). Naturally, the query itself should not try to access the

-hidden column data.
-
-## Encryption overhead
-
-The size overhead of Parquet modular encryption is negligible, since the most of the encryption

-operations are performed on pages (the minimal unit of Parquet data storage and compression).
The
-overhead order of magnitude is adding ~ 1 byte per each 10,000 bytes of original data.
-
-The throughput overhead of Parquet modular encryption depends on whether AES enciphering
is done
-in software or hardware. In both cases, performing encryption on full pages (~1MB buffers)
instead of
-on much smaller individual data values causes AES to work at its maximal speed. Preliminary
tests
-show Parquet modular encryption throughput overhead to be up to a few percents in Java 9
workloads.
+secret columns, but also the file footer metadata. This hides the file schema, 
+number of rows, key-value properties, column sort order, names of the encrypted columns 
+and metadata of the column encryption keys. 
+
+The columns encrypted with the same key as the footer must leave the column metadata at the
original 
+location, `optional ColumnMetaData meta_data` in the `ColumnChunk` structure. 
+This field is not set for columns encrypted with a column-specific key - instead, the `ColumnMetaData`
+is Thrift-serialized, encrypted with the column key and written to the `encrypted_column_metadata`

+field in the `ColumnChunk` structure, as described in the section 5.3.
+
+A Thrift-serialized `FileCryptoMetaData` structure is written before the encrypted footer.

+It contains information on the file encryption algorithm and on the footer key metadata.
Then 
+the combined length of this structure and of the encrypted footer is written as a 4-byte

+little endian integer, followed by a final magic string, "PARE". The same magic bytes are

+written at the beginning of the file (offset 0). Parquet readers start file parsing by 
+reading and checking the magic string. Therefore, the encrypted footer mode uses a new 
+magic string ("PARE") in order to instruct readers to look for a file crypto metadata 
+before the footer - and also to immediately inform legacy readers (expecting ‘PAR1’ 
+bytes) that they can’t parse this file.
+
+```c
+/** Crypto metadata for files with encrypted footer **/
+struct FileCryptoMetaData {
+  /** 
+   * Encryption algorithm. This field is only used for files
+   * with encrypted footer. Files with plaintext footer store algorithm id
+   * inside footer (FileMetaData structure).
+   */
+  1: required EncryptionAlgorithm encryption_algorithm
+    
+  /** Retrieval metadata of key used for encryption of footer, 
+   *  and (possibly) columns **/
+  2: optional binary key_metadata
+}
+```
+
+ ![File Layout - Encrypted footer](doc/images/FileLayoutEncryptionEF.png)
+ 
+ 
+### 5.5 Plaintext footer mode
+This mode allows legacy Parquet versions (released before the encryption support) to access

+unencrypted columns in encrypted files - at a price of leaving certain metadata fields 
+unprotected in these files. 
+
+The plaintext footer mode can be useful during a transitional period in organizations where

+some frameworks can't be upgraded to a new Parquet library for a while. Data writers will

+upgrade and run with a new Parquet version, producing encrypted files in this mode. Data

+readers working with sensitive data will also upgrade to a new Parquet library. But other

+readers that don't need the sensitive columns, can continue working with an older Parquet

+version. They will be able to access plaintext columns in encrypted files. A legacy reader,

+trying to access a sensitive column data in an encrypted file with a plaintext footer, will

+get an exception. More specifically, a Thrift parsing exception on an encrypted page header

+structure. Again, using legacy Parquet readers for encrypted files is a temporary solution.
+
+In the plaintext footer mode, the `optional ColumnMetaData meta_data` is set in the `ColumnChunk`
+structure for all columns, but is stripped of the statistics for the sensitive (encrypted)

+columns. These statistics are available for new readers with the column key - they decrypt

+the `encrypted_column_metadata` field, described in the section 5.3, and parse it to get
statistics 
+and all other column metadata values. The legacy readers are not aware of the encrypted metadata
field; 
+they parse the regular (plaintext) field as usual. While they can't read the data of encrypted

+columns, they read their metadata to extract the offset and size of encrypted column data,

+required for column chunk vectorization.
+
+The plaintext footer is signed in order to prevent tampering with the 
+`FileMetaData` contents. The footer signing is done by encrypting the serialized `FileMetaData`

+structure with the 
+AES GCM algorithm - using a footer signing key, and an AAD constructed according to the instructions

+of the section 4.4. Only the nonce and GCM tag are stored in the file – as a 28-byte 
+fixed-length array, written right after  the footer itself. The ciphertext is not stored,

+because it is not required for footer integrity verification by readers.
+
+| nonce (12 bytes) |  tag (16 bytes) |
+|------------------|-----------------|
+
+
+The plaintext footer mode sets the following fields in the the FileMetaData structure:
+
+```c
+struct FileMetaData {
+...
+  /** 
+   * Encryption algorithm. This field is set only in encrypted files
+   * with plaintext footer. Files with encrypted footer store algorithm id
+   * in FileCryptoMetaData structure.
+   */
+  8: optional EncryptionAlgorithm encryption_algorithm
+
+  /** 
+   * Retrieval metadata of key used for signing the footer. 
+   * Used only in encrypted files with plaintext footer. 
+   */ 
+  9: optional binary footer_signing_key_metadata
+}
+```
+ 
+The `FileMetaData` structure is Thrift-serialized and written to the output stream.
+The 28-byte footer signature is written after the plaintext footer, followed by a 4-byte
little endian integer 
+that contains the combined length of the footer and its signature. A final magic string,

+"PAR1", is written at the end of the 
+file. The same magic string is written at the beginning of the file (offset 0). The magic
bytes 
+for plaintext footer mode are ‘PAR1’ to allow legacy readers to read projections of the
file 
+that do not include encrypted columns.
+
+ ![File Layout - Encrypted footer](doc/images/FileLayoutEncryptionPF.png)
+
+## 6. Encryption Overhead
+The size overhead of Parquet modular encryption is negligible, since most of the encryption

+operations are performed on pages (the minimal unit of Parquet data storage and compression).

+The overhead order of magnitude is adding 1 byte per each ~30,000 bytes of original 
+data - calculated by comparing the page encryption overhead (nonce + tag + length = 32 bytes)

+to the default page size (1 MB). This is a rough estimation, and can change with the encryption
+algorithm (no 16-byte tag in AES_GCM_CTR_V1) and with page configuration or data encoding/compression.
+
+The throughput overhead of Parquet modular encryption depends on whether AES enciphering
is 
+done in software or hardware. In both cases, performing encryption on full pages (~1MB buffers)

+instead of on much smaller individual data values causes AES to work at its maximal speed.

+
diff --git a/README.md b/README.md
index b048c77..951aaa0 100644
--- a/README.md
+++ b/README.md
@@ -221,11 +221,11 @@ file reference multiple parquet files.
 Parquet files, containing sensitive information, can be protected by the modular
 encryption mechanism, that encrypts and authenticates the file data and metadata - 
 while allowing for a regular Parquet functionality (columnar projection, 
-predicate pushdown, encoding and compression). The mechanism also enables column access 
-control, via support for encryption of different columns with different keys.
+predicate pushdown, encoding and compression). The mechanism supports encryption 
+of different columns with different keys.
 Each Parquet module (footer, page headers, pages, column indexes, column metadata) is 
-encrypted separately. Then it is possible to fetch and decrypt the footer, find the 
-offset of a required page, fetch it and decrypt the data. 
+encrypted separately - making it possible to fetch and decrypt the footer, find the 
+offset of required pages, fetch them and decrypt the data. 
 See [Encryption.md](Encryption.md) for details.
 
 
diff --git a/doc/images/FileLayoutEncryptionEF.jpg b/doc/images/FileLayoutEncryptionEF.jpg
deleted file mode 100644
index c589d9c..0000000
Binary files a/doc/images/FileLayoutEncryptionEF.jpg and /dev/null differ
diff --git a/doc/images/FileLayoutEncryptionEF.png b/doc/images/FileLayoutEncryptionEF.png
new file mode 100755
index 0000000..f8fc3ef
Binary files /dev/null and b/doc/images/FileLayoutEncryptionEF.png differ
diff --git a/doc/images/FileLayoutEncryptionPF.jpg b/doc/images/FileLayoutEncryptionPF.jpg
deleted file mode 100644
index c5300c7..0000000
Binary files a/doc/images/FileLayoutEncryptionPF.jpg and /dev/null differ
diff --git a/doc/images/FileLayoutEncryptionPF.png b/doc/images/FileLayoutEncryptionPF.png
new file mode 100755
index 0000000..b7046a1
Binary files /dev/null and b/doc/images/FileLayoutEncryptionPF.png differ


Mime
View raw message