hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a.@apache.org
Subject [08/19] hadoop git commit: MAPREDUCE-6257. Document encrypted spills (Bibin A Chundatt via aw)
Date Mon, 10 Aug 2015 16:47:54 GMT
MAPREDUCE-6257. Document encrypted spills (Bibin A Chundatt via aw)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/fb1be0b3
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/fb1be0b3
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/fb1be0b3

Branch: refs/heads/HADOOP-12111
Commit: fb1be0b3100cdd69f6dc1987585fcadd4e7c8a2a
Parents: e73a928
Author: Allen Wittenauer <aw@apache.org>
Authored: Thu Aug 6 10:13:18 2015 -0700
Committer: Allen Wittenauer <aw@apache.org>
Committed: Thu Aug 6 10:13:51 2015 -0700

----------------------------------------------------------------------
 hadoop-mapreduce-project/CHANGES.txt            |  2 ++
 .../src/main/resources/mapred-default.xml       | 20 ++++++++++++++++++++
 .../src/site/markdown/EncryptedShuffle.md       |  8 +++++++-
 3 files changed, 29 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/fb1be0b3/hadoop-mapreduce-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/CHANGES.txt b/hadoop-mapreduce-project/CHANGES.txt
index e910cdd..c424667 100644
--- a/hadoop-mapreduce-project/CHANGES.txt
+++ b/hadoop-mapreduce-project/CHANGES.txt
@@ -206,6 +206,8 @@ Trunk (Unreleased)
     MAPREDUCE-6406. Update FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT
     to match mapred-default.xml. (Ray Chiang via devaraj)
 
+    MAPREDUCE-6257. Document encrypted spills (Bibin A Chundatt via aw)
+
   BREAKDOWN OF MAPREDUCE-2841 (NATIVE TASK) SUBTASKS
 
     MAPREDUCE-5985. native-task: Fix build on macosx. Contributed by

http://git-wip-us.apache.org/repos/asf/hadoop/blob/fb1be0b3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
index ddcd2df..6d205c5 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
@@ -1760,4 +1760,24 @@
   </description>
 </property>
 
+<property>
+  <name>mapreduce.job.encrypted-intermediate-data</name>
+  <value>false</value>
+  <description>Encrypt intermediate MapReduce spill files or not
+  default is false</description>
+</property>
+
+<property>
+  <name>mapreduce.job.encrypted-intermediate-data-key-size-bits</name>
+  <value>128</value>
+  <description>Mapreduce encrypt data key size default is 128</description>
+</property>
+
+<property>
+  <name>mapreduce.job.encrypted-intermediate-data.buffer.kb</name>
+  <value>128</value>
+  <description>Buffer size for intermediate encrypt data in kb
+  default is 128</description>
+</property>
+
 </configuration>

http://git-wip-us.apache.org/repos/asf/hadoop/blob/fb1be0b3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md
----------------------------------------------------------------------
diff --git a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md
index c23be7a..fddd84f 100644
--- a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md
+++ b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/EncryptedShuffle.md
@@ -260,4 +260,10 @@ Encrypted Intermediate Data Spill files
 This capability allows encryption of the intermediate files generated during the merge and
shuffle phases.
 It can be enabled by setting the `mapreduce.job.encrypted-intermediate-data` job property
to `true`.
 
-**NOTE:** Currently, enabling encrypted intermediate data spills would restrict the number
of attempts of the job to 1.
\ No newline at end of file
+|               Name |  Type |                                                          
Description |
+|:---- |:---- |:---- |
+| mapreduce.job.encrypted-intermediate-data | boolean | Enable or disable encrypt intermediate
mapreduce spill files.Default is false. |
+| mapreduce.job.encrypted-intermediate-data-key-size-bits | int | The key length used by
keygenerator to encrypt data spilled to disk. |
+| mapreduce.job.encrypted-intermediate-data.buffer.kb | int | The buffer size in kb for stream
written to disk after encryption. |
+
+**NOTE:** Currently, enabling encrypted intermediate data spills would restrict the number
of attempts of the job to 1.


Mime
View raw message