hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7544) Transparent table/CF encryption
Date Sat, 12 Jan 2013 00:24:13 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551696#comment-13551696

Andrew Purtell commented on HBASE-7544:

bq. the 'hbase' user would have to have to have access to all the keys, and that user is the
only one who would have access to the on-disk files

The aim is to protect sensitive data against accidental leakage and to facilitate auditable
compliance according to the regulations under which several industries operate. We assume
under normal circumstances that the 'hbase' user is the only one who would have access to
on-disk files. However this does not guarantee leakage isn't possible if HDFS configuration
is incorrect -- HDFS and HBase might be independently managed -- or if a server is decommissioned
from the cluster and mishandled. The usual rationale for this type of feature. 

Schema design considerations are similar to those of HFile compression. Some tables might
only have one sensitive column encrypted, to minimize performance impacts. We might also not
want to encrypt every type of block in the HFile (nor compress them). 

There would be a master key supplied to HBase processes, managed by the cluster administrator,
protected by the Java Keystore, perhaps residing on a hardware security module. Within HBase,
per-table and per-CF keys are created on demand. There are a couple of reasons why the 2-tier
key architecture is good (reduction of scope of compromise, facilitating lazy key rotation,
etc.) The administrator would need to run HBCK on a system with access to the master key material
in order to take recovery actions.

I will attach a design doc and patch for consideration, once I have the go ahead. :-)
> Transparent table/CF encryption
> -------------------------------
>                 Key: HBASE-7544
>                 URL: https://issues.apache.org/jira/browse/HBASE-7544
>             Project: HBase
>          Issue Type: New Feature
>          Components: HFile, io
>    Affects Versions: 0.96.0
>            Reporter: Andrew Purtell
>            Assignee: Andrew Purtell
> Introduce transparent encryption of HBase on disk data.
> Depends on a separate contribution of an encryption codec framework to Hadoop core and
an AES-NI (native code) codec. This is work done in the context of MAPREDUCE-4491 but I'd
gather there will be additional JIRAs for common and HDFS parts of it.
> Requirements:
> - Transparent encryption at the CF or table level
> - Protect against all data leakage from files at rest
> - Two-tier key architecture for consistency with best practices for this feature in the
RDBMS world
> - Built-in key management
> - Flexible and non-intrusive key rotation
> - Mechanisms not exposed to or modifiable by users
> - Hardware security module integration (via Java KeyStore)
> - HBCK support for transparently encrypted files (+ plugin architecture for HBCK)
> Additional goals:
> - Shell support for administrative functions
> - Avoid performance impact for the null crypto codec case
> - Play nicely with other changes underway: in HFile, block coding, etc.
> We're aiming for rough parity with Oracle's transparent tablespace encryption feature,
described in http://www.oracle.com/technetwork/database/owp-security-advanced-security-11gr-133411.pdf
> {quote}
> “Transparent Data Encryption uses a 2-tier key architecture for flexible and non-intrusive
key rotation and least operational and performance impact: Each application table with at
least one encrypted column has its own table key, which is applied to all encrypted columns
in that table. Equally, each encrypted tablespace has its own tablespace key. Table keys are
stored in the data dictionary of the database, while tablespace keys are stored in the header
of the tablespace and additionally, the header of each underlying OS file that makes up the
tablespace.  Each of these keys is encrypted with the TDE master encryption key, which is
stored outside of the database in an external security module: either the Oracle Wallet (a
PKCS#12 formatted file that is encrypted using a passphrase supplied either by the designated
security administrator or DBA during setup),  or a Hardware Security Module (HSM) device for
higher assurance […]”
> {quote}
> Further design details forthcoming in a design document and patch as soon as we have
all of the clearances in place.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message