hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "cheng xu" <cheng.a...@intel.com>
Subject Re: Review Request 33956: HIVE-9614: Encrypt mapjoin tables
Date Thu, 21 May 2015 08:49:57 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33956/#review84671
-----------------------------------------------------------


Thank you for this patch. I have some questions and will have another round of review after
understanding these questions. Thank you!


common/src/java/org/apache/hive/common/util/HdfsEncryptionUtilities.java
<https://reviews.apache.org/r/33956/#comment136026>

    Why not use the isPathEncrypted from HdfsEncryptionShim directly?



common/src/java/org/apache/hive/common/util/HdfsEncryptionUtilities.java
<https://reviews.apache.org/r/33956/#comment136027>

    The same as above.



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
<https://reviews.apache.org/r/33956/#comment136025>

    Is it possible to get the FsPermission from org.apache.hadoop.fs.FileContext?



ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
<https://reviews.apache.org/r/33956/#comment136022>

    I am a little confused here. How can a local path be converted to a hdfs path? The original
code is trying to create a tar file from a local path and uploading it to the hdfs with replication
information. The new code path will lose the replication information. And the previous code
path will only be executed in a local file or pfile schema in test.



ql/src/test/queries/clientpositive/encryption_map_join_select.q
<https://reviews.apache.org/r/33956/#comment136021>

    drop table encryptedTable PURGE;


- cheng xu


On May 7, 2015, 9:23 p.m., Sergio Pena wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33956/
> -----------------------------------------------------------
> 
> (Updated May 7, 2015, 9:23 p.m.)
> 
> 
> Review request for hive, Brock Noland and cheng xu.
> 
> 
> Bugs: HIVE-9614
>     https://issues.apache.org/jira/browse/HIVE-9614
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> The security issue here is that encrypted tables used on MAP-JOIN queries, and stored
on the distribute cache, are first copied to the client local filesystem in an unencrypted
form in order to compress it there.
> 
> This patch avoids the local copy if the table is encrypted on HDFS. It keeps the hash
table on HDFS, compresses the table in HDFS, and then adds it to the distributed cache.
> 
> Files that are copied to the datanodes by the distributed cache are still unencrypted.
This is a limitation we have from HDFS.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/common/CompressionUtils.java 0e0d538c2faf1c52c4d8378df013294ae4efa41c

>   common/src/java/org/apache/hive/common/util/HdfsEncryptionUtilities.java PRE-CREATION

>   itests/src/test/resources/testconfiguration.properties 3eff7d010923a4e07d5024904f1531ca52473aa2

>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java ad5c8f8302de2a15b1703161799f71cd81a94475

>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java d7a08ecf1c183fe56b5ca41c2c69d413874418bb

>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 4d84f0f76ce17711077ceadf23e6b9ed12e6a414

>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MapJoinResolver.java c0a72b69df3871bbcc870af286774aee5269668b

>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cbc5466261f749fe7b84d7533dc0ff3274b6777f

>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 82143a64db163da766dcc138231b4d4174603470

>   ql/src/test/queries/clientpositive/encryption_map_join_select.q PRE-CREATION 
>   ql/src/test/results/clientpositive/encrypted/encryption_map_join_select.q.out PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/33956/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Sergio Pena
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message