hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangjiayou (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-6612) Compressing map output error
Date Thu, 21 Jan 2016 07:00:47 GMT
wangjiayou created MAPREDUCE-6612:
-------------------------------------

             Summary: Compressing map output error
                 Key: MAPREDUCE-6612
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6612
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mrv2
            Reporter: wangjiayou


I used gzip compressionor to compress map output,but there is error occurrence.my hadoop version
2.7.1
Configuration conf = getConf();
conf.setBoolean(Job.MAP_OUTPUT_COMPRESS, true);
conf.setClass(Job.MAP_OUTPUT_COMPRESS_CODEC, GzipCodec.class, CompressionCodec.class);

error info:
2016-01-21 15:02:23 [org.apache.hadoop.mapred.LocalJobRunner]-[INFO] reduce > copy
2016-01-21 15:02:23 [org.apache.hadoop.mapred.LocalJobRunner]-[WARN] job_local1329832043_0001
java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
shuffle in localfetcher#1
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle
in localfetcher#1
	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: not a gzip file
	at org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.processBasicHeader(BuiltInGzipDecompressor.java:496)
	at org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeHeaderState(BuiltInGzipDecompressor.java:257)
	at org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:186)
	at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
	at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
	at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.shuffle(InMemoryMapOutput.java:97)
	at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:157)
	at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:102)
	at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:85)
2016-01-21 15:02:23 [org.apache.hadoop.mapred.LocalJobRunner]-[INFO] reduce > copy
2016-01-21 15:02:23 [org.apache.hadoop.mapred.LocalJobRunner]-[INFO] reduce > copy
2016-01-21 15:02:24 [org.apache.hadoop.mapred.LocalJobRunner]-[INFO] reduce > copy




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message