Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0ACBA10D7C for ; Tue, 26 Nov 2013 22:52:40 +0000 (UTC) Received: (qmail 47040 invoked by uid 500); 26 Nov 2013 22:52:39 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 47003 invoked by uid 500); 26 Nov 2013 22:52:39 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 46994 invoked by uid 99); 26 Nov 2013 22:52:39 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Nov 2013 22:52:39 +0000 Date: Tue, 26 Nov 2013 22:52:39 +0000 (UTC) From: "Hudson (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-7544) Transparent table/CF encryption MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-7544?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1383= 3185#comment-13833185 ]=20 Hudson commented on HBASE-7544: ------------------------------- SUCCESS: Integrated in HBase-TRUNK #4699 (See [https://builds.apache.org/jo= b/HBase-TRUNK/4699/]) Amend HBASE-7544. Fix javadoc typo for Cipher#createDecryptionStream (apurt= ell: rev 1545790) * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /Cipher.java HBASE-7544. Transparent CF encryption (apurtell: rev 1545536) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDe= scriptor.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDes= criptor.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/security/= EncryptionUtil.java * /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/security * /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/security/= TestEncryptionUtil.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstant= s.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /Cipher.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /CipherProvider.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /Context.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /Decryptor.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /DefaultCipherProvider.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /Encryption.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /Encryptor.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /KeyProvider.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /KeyStoreKeyProvider.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /aes * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /aes/AES.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /aes/AESDecryptor.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto= /aes/AESEncryptor.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encodi= ng/HFileBlockDefaultDecodingContext.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encodi= ng/HFileBlockDefaultEncodingContext.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/= HFileContext.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/= HFileContextBuilder.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Byte= s.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto= /KeyProviderForTesting.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto= /TestCipherProvider.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto= /TestEncryption.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto= /TestKeyProvider.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto= /TestKeyStoreKeyProvider.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto= /aes * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/io/crypto= /aes/TestAES.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTe= stIngestWithEncryption.java * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobu= f/generated/EncryptionProtos.java * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobu= f/generated/HFileProtos.java * /hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobu= f/generated/WALProtos.java * /hbase/trunk/hbase-protocol/src/main/protobuf/Encryption.proto * /hbase/trunk/hbase-protocol/src/main/protobuf/HFile.proto * /hbase/trunk/hbase-protocol/src/main/protobuf/WAL.proto * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/HalfSt= oreFileReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/= AbstractHFileReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/= FixedFileTrailer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/= HFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/= HFileBlock.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/= HFilePrettyPrinter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/= HFileReaderV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/= HFileReaderV3.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/= HFileWriterV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/= HFileWriterV3.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce= /LoadIncrementalHFiles.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionser= ver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionser= ver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionser= ver/StoreFileInfo.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionser= ver/wal/ProtobufLogReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionser= ver/wal/ProtobufLogWriter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionser= ver/wal/SecureProtobufLogReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionser= ver/wal/SecureProtobufLogWriter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionser= ver/wal/SecureWALCellCodec.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionser= ver/wal/WALCellCodec.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionser= ver/wal/WriterBase.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Comp= ressionTest.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBas= eFsck.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/hbck= /HFileCorruptionChecker.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HFilePerf= ormanceEvaluation.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHa= lfStoreFileReader.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/= RandomSeek.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/= TestCacheOnWrite.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/= TestFixedFileTrailer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/= TestHFile.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/= TestHFileBlockIndex.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/= TestHFileEncryption.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/= TestHFileInlineToRootChunkConversion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/= TestHFilePerformance.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/= TestHFileSeek.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/= TestReseekTo.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/= TestSeekTo.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce= /TestHFileOutputFormat.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce= /TestLoadIncrementalHFiles.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionser= ver/TestEncryptionKeyRotation.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionser= ver/TestEncryptionRandomKeying.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionser= ver/TestStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionser= ver/TestStoreFile.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionser= ver/wal/HLogPerformanceEvaluation.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionser= ver/wal/SequenceFileLogWriter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionser= ver/wal/TestSecureHLog.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionser= ver/wal/TestSecureWALReplay.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/Load= TestTool.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/Test= HBaseFsckEncryption.java * /hbase/trunk/hbase-shell/src/main/ruby/hbase/admin.rb > Transparent table/CF encryption > ------------------------------- > > Key: HBASE-7544 > URL: https://issues.apache.org/jira/browse/HBASE-7544 > Project: HBase > Issue Type: New Feature > Components: HFile, io > Reporter: Andrew Purtell > Assignee: Andrew Purtell > Fix For: 0.98.0 > > Attachments: 7544-addendum-1.patch, 7544-final.patch, 7544.patch,= 7544.patch, 7544.patch, 7544.patch, 7544.patch, 7544.patch, 7544p1.patch, = 7544p1.patch, 7544p2.patch, 7544p2.patch, 7544p3.patch, 7544p3.patch, 7544p= 4.patch, historical-7544.patch, historical-7544.pdf, historical-shell.patch= , latency-single.7544.xlsx > > > Introduce transparent encryption of HBase on disk data. > Depends on a separate contribution of an encryption codec framework to Ha= doop core and an AES-NI (native code) codec. This is work done in the conte= xt of MAPREDUCE-4491 but I'd gather there will be additional JIRAs for comm= on and HDFS parts of it. > Requirements: > - Transparent encryption at the CF or table level > - Protect against all data leakage from files at rest > - Two-tier key architecture for consistency with best practices for this = feature in the RDBMS world > - Built-in key management > - Flexible and non-intrusive key rotation > - Mechanisms not exposed to or modifiable by users > - Hardware security module integration (via Java KeyStore) > - HBCK support for transparently encrypted files (+ plugin architecture f= or HBCK) > Additional goals: > - Shell support for administrative functions > - Avoid performance impact for the null crypto codec case > - Play nicely with other changes underway: in HFile, block coding, etc. > We're aiming for rough parity with Oracle's transparent tablespace encryp= tion feature, described in http://www.oracle.com/technetwork/database/owp-s= ecurity-advanced-security-11gr-133411.pdf as > {quote} > =E2=80=9CTransparent Data Encryption uses a 2-tier key architecture for f= lexible and non-intrusive key rotation and least operational and performanc= e impact: Each application table with at least one encrypted column has its= own table key, which is applied to all encrypted columns in that table. Eq= ually, each encrypted tablespace has its own tablespace key. Table keys are= stored in the data dictionary of the database, while tablespace keys are s= tored in the header of the tablespace and additionally, the header of each = underlying OS file that makes up the tablespace. Each of these keys is enc= rypted with the TDE master encryption key, which is stored outside of the d= atabase in an external security module: either the Oracle Wallet (a PKCS#12= formatted file that is encrypted using a passphrase supplied either by the= designated security administrator or DBA during setup), or a Hardware Sec= urity Module (HSM) device for higher assurance [=E2=80=A6]=E2=80=9D > {quote} > Further design details forthcoming in a design document and patch as soon= as we have all of the clearances in place. -- This message was sent by Atlassian JIRA (v6.1#6144)