hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6444) Add a checkpointable version of shuffle and reduce context supported by a checkpoint manager which uses the HDFS
Date Thu, 06 Aug 2015 02:31:04 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659365#comment-14659365
] 

Hadoop QA commented on MAPREDUCE-6444:
--------------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  21m 59s | Pre-patch trunk compilation is healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags.
|
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to include 6 new
or modified test files. |
| {color:red}-1{color} | javac |   8m 38s | The applied patch generated  1  additional warning
messages. |
| {color:green}+1{color} | javadoc |  10m 17s | There were no new javadoc warning messages.
|
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does not increase
the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m 39s | The applied patch generated  7 new checkstyle
issues (total was 294, now 296). |
| {color:green}+1{color} | whitespace |   0m  6s | The patch has no lines that end in whitespace.
|
| {color:green}+1{color} | install |   1m 26s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with eclipse:eclipse.
|
| {color:red}-1{color} | findbugs |   3m 25s | The patch appears to introduce 1 new Findbugs
(version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests |   9m 15s | Tests failed in hadoop-mapreduce-client-app.
|
| {color:red}-1{color} | mapreduce tests |   1m 44s | Tests failed in hadoop-mapreduce-client-core.
|
| {color:red}-1{color} | mapreduce tests | 149m 19s | Tests failed in hadoop-mapreduce-client-jobclient.
|
| | | 209m  0s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-mapreduce-client-core |
| Failed unit tests | hadoop.mapreduce.TestReducerPreemption |
|   | hadoop.mapreduce.task.reduce.TestMerger |
|   | hadoop.mapreduce.task.reduce.TestShufflePreemption |
|   | hadoop.mapreduce.security.ssl.TestEncryptedShuffle |
|   | hadoop.mapreduce.lib.map.TestMultithreadedMapper |
|   | hadoop.mapred.TestLineRecordReaderJobs |
|   | hadoop.mapreduce.lib.join.TestJoinDatamerge |
|   | hadoop.mapred.TestMapOutputType |
|   | hadoop.conf.TestNoDefaultsJobConf |
|   | hadoop.mapreduce.lib.input.TestMultipleInputs |
|   | hadoop.mapred.jobcontrol.TestLocalJobControl |
|   | hadoop.mapred.lib.TestChainMapReduce |
|   | hadoop.mapreduce.lib.chain.TestSingleElementChain |
|   | hadoop.fs.TestDFSIO |
|   | hadoop.mapreduce.TestMROutputFormat |
|   | hadoop.mapreduce.TestLocalRunner |
|   | hadoop.mapred.TestFieldSelection |
|   | hadoop.mapred.TestMiniMRClientCluster |
|   | hadoop.mapred.TestClusterMRNotification |
|   | hadoop.mapreduce.lib.partition.TestMRKeyFieldBasedComparator |
|   | hadoop.mapred.TestMRTimelineEventHandling |
|   | hadoop.mapreduce.TestMapCollection |
|   | hadoop.mapreduce.lib.chain.TestMapReduceChain |
|   | hadoop.mapred.TestLocalMRNotification |
|   | hadoop.mapreduce.v2.TestMRJobsWithHistoryService |
|   | hadoop.mapreduce.lib.aggregate.TestMapReduceAggregates |
|   | hadoop.mapred.TestUserDefinedCounters |
|   | hadoop.mapreduce.security.TestBinaryTokenFile |
|   | hadoop.mapred.TestReduceFetch |
|   | hadoop.mapreduce.v2.TestMiniMRProxyUser |
|   | hadoop.mapred.TestOldCombinerGrouping |
|   | hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat |
|   | hadoop.mapreduce.lib.chain.TestChainErrors |
|   | hadoop.mapreduce.TestValueIterReset |
|   | hadoop.mapreduce.v2.TestMROldApiJobs |
|   | hadoop.fs.slive.TestSlive |
|   | hadoop.mapred.TestClusterMapReduceTestCase |
|   | hadoop.mapred.TestMRIntermediateDataEncryption |
|   | hadoop.mapred.TestReduceFetchFromPartialMem |
|   | hadoop.mapreduce.TestNewCombinerGrouping |
|   | hadoop.fs.TestFileSystem |
|   | hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection |
|   | hadoop.mapred.TestJobName |
|   | hadoop.mapreduce.lib.input.TestLineRecordReaderJobs |
|   | hadoop.mapred.TestCollect |
|   | hadoop.mapreduce.v2.TestMRJobsWithProfiler |
|   | hadoop.mapred.TestLocalJobSubmission |
|   | hadoop.mapred.TestJobCounters |
|   | hadoop.mapreduce.TestMapReduceLazyOutput |
|   | hadoop.mapred.TestReduceTask |
|   | hadoop.mapred.TestJavaSerialization |
|   | hadoop.mapreduce.lib.output.TestMRMultipleOutputs |
|   | hadoop.mapred.TestSpecialCharactersInOutputPath |
|   | hadoop.mapreduce.v2.TestUberAM |
|   | hadoop.mapreduce.security.TestMRCredentials |
|   | hadoop.mapred.TestLazyOutput |
|   | hadoop.mapred.TestFileOutputFormat |
|   | hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities |
|   | hadoop.mapred.pipes.TestPipeApplication |
|   | hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
|   | hadoop.mapreduce.TestChild |
|   | hadoop.mapred.lib.TestMultithreadedMapRunner |
|   | hadoop.mapred.TestReporter |
|   | hadoop.mapred.lib.aggregate.TestAggregates |
|   | hadoop.mapred.TestMapRed |
|   | hadoop.mapred.lib.TestMultipleOutputs |
|   | hadoop.mapreduce.TestMapReduce |
|   | hadoop.mapred.TestComparators |
|   | hadoop.mapreduce.TestMRJobClient |
|   | hadoop.mapred.lib.TestKeyFieldBasedComparator |
|   | hadoop.mapred.join.TestDatamerge |
| Timed out tests | org.apache.hadoop.mapred.TestJobSysDirWithDFS |
|   | org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers |
|   | org.apache.hadoop.mapred.TestMiniMRClasspath |
|   | org.apache.hadoop.mapreduce.v2.TestMRJobs |
|   | org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12748950/MAPREDUCE-6444.001.patch
|
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / 3da0bed |
| javac | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |  https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| Findbugs warnings | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
|
| hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-core test log | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/testReport/
|
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5930/console |


This message was automatically generated.

> Add a checkpointable version of shuffle and reduce context supported by a checkpoint
manager which uses the HDFS
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6444
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6444
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Augusto Souza
>            Assignee: Augusto Souza
>         Attachments: MAPREDUCE-6444.001.patch
>
>
> Patch to add classes to provide a checkpointable version of shuffle and reduce context.
Also, add a classe which makes possible to save a checkpoint of a task in HDFS. Those classes
are respectively: CheckpointableShuffle, CheckpointableReduceContextImpl and TaskCheckpointManager.
> This improvement also helps in the splitting of patch in MAPREDUCE-5269 that is too large
for being reviewed or accepted by Jenkins CI scripts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message