hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Young <danoyo...@gmail.com>
Subject Re: distcp error.
Date Thu, 30 Aug 2012 13:29:00 GMT
I was just reading about this in the Hadoop definitive guide last night.
Need to be the same version. You can try hftp between versions

Regards

Dano
On Aug 28, 2012 9:44 AM, "Tao" <zta@outlook.com> wrote:

> Hi, all****
>
>          I use distcp copying data from hadoop1.0.3 to hadoop 2.0.1.****
>
>          When the file path(or file name) contain Chinese character, an
> exception will throw. Like below. I need some help about this.****
>
>          Thanks.****
>
>          ****
>
> ** **
>
> ** **
>
> ** **
>
> [hdfs@host ~]$ hadoop distcp -i -prbugp -m 14 -overwrite -log
> /tmp/distcp.log hftp://10.xx.xx.aa:50070/tmp/中文路径测试 hdfs://
> 10.xx.xx.bb:54310/tmp/distcp_test14****
>
> 12/08/28 23:32:31 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=true, maxMaps=14, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[hftp://10.xx.xx.aa:50070/tmp/中文路径测试], targetPath=hdfs://
> 10.xx.xx.bb:54310/tmp/distcp_test14}****
>
> 12/08/28 23:32:33 INFO tools.DistCp: DistCp job log path: /tmp/distcp.log*
> ***
>
> 12/08/28 23:32:34 WARN conf.Configuration: io.sort.mb is deprecated.
> Instead, use mapreduce.task.io.sort.mb****
>
> 12/08/28 23:32:34 WARN conf.Configuration: io.sort.factor is deprecated.
> Instead, use mapreduce.task.io.sort.factor****
>
> 12/08/28 23:32:34 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable**
> **
>
> 12/08/28 23:32:36 INFO mapreduce.JobSubmitter: number of splits:1****
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.jar is deprecated.
> Instead, use mapreduce.job.jar****
>
> 12/08/28 23:32:36 WARN conf.Configuration:
> mapred.map.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.map.speculative****
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.reduce.tasks is
> deprecated. Instead, use mapreduce.job.reduces****
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.mapoutput.value.class is
> deprecated. Instead, use mapreduce.map.output.value.class****
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapreduce.map.class is
> deprecated. Instead, use mapreduce.job.map.class****
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.job.name is deprecated.
> Instead, use mapreduce.job.name****
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapreduce.inputformat.class is
> deprecated. Instead, use mapreduce.job.inputformat.class****
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.output.dir is
> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir****
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapreduce.outputformat.class is
> deprecated. Instead, use mapreduce.job.outputformat.class****
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.map.tasks is deprecated.
> Instead, use mapreduce.job.maps****
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.mapoutput.key.class is
> deprecated. Instead, use mapreduce.map.output.key.class****
>
> 12/08/28 23:32:36 WARN conf.Configuration: mapred.working.dir is
> deprecated. Instead, use mapreduce.job.working.dir****
>
> 12/08/28 23:32:37 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1345831938927_0039 to ResourceManager at baby20/10.1.1.40:8040
> ****
>
> 12/08/28 23:32:37 INFO mapreduce.Job: The url to track the job:
> http://baby20:8088/proxy/application_1345831938927_0039/****
>
> 12/08/28 23:32:37 INFO tools.DistCp: DistCp job-id: job_1345831938927_0039
> ****
>
> 12/08/28 23:32:37 INFO mapreduce.Job: Running job: job_1345831938927_0039*
> ***
>
> 12/08/28 23:32:50 INFO mapreduce.Job: Job job_1345831938927_0039 running
> in uber mode : false****
>
> 12/08/28 23:32:50 INFO mapreduce.Job:  map 0% reduce 0%****
>
> 12/08/28 23:33:00 INFO mapreduce.Job:  map 100% reduce 0%****
>
> 12/08/28 23:33:00 INFO mapreduce.Job: Task Id :
> attempt_1345831938927_0039_m_000000_0, Status : FAILED****
>
> Error: java.io.IOException: File copy failed: hftp://10.1.1.26:50070/tmp/
> 中文路径测试/part-r-00017 --> hdfs://
> 10.1.1.40:54310/tmp/distcp_test14/part-r-00017****
>
>         at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262)
> ****
>
>         at
> org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229)****
>
>         at
> org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)****
>
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)****
>
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725)
> ****
>
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)****
>
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)***
> *
>
>         at java.security.AccessController.doPrivileged(Native Method)****
>
>         at javax.security.auth.Subject.doAs(Subject.java:396)****
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> ****
>
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)****
>
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hftp://10.1.1.26:50070/tmp/中文路径测试/part-r-00017 to hdfs://
> 10.1.1.40:54310/tmp/distcp_test14/part-r-00017****
>
>         at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> ****
>
>         at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> ****
>
>         ... 10 more****
>
> Caused by:
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand$CopyReadException:
> java.io.IOException: HTTP_OK expected, received 500****
>
>         at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:201)
> ****
>
>         at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyBytes(RetriableFileCopyCommand.java:167)
> ****
>
>         at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToTmpFile(RetriableFileCopyCommand.java:112)
> ****
>
>         at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:90)
> ****
>
>         at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:71)
> ****
>
>         at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ****
>
>         ... 11 more****
>
> Caused by: java.io.IOException: HTTP_OK expected, received 500****
>
>         at
> org.apache.hadoop.hdfs.HftpFileSystem$RangeHeaderInputStream.checkResponseCode(HftpFileSystem.java:381)
> ****
>
>         at
> org.apache.hadoop.hdfs.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:121)
> ****
>
>         at
> org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:103)
> ****
>
>         at
> org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:158)
> ****
>
>         at java.io.DataInputStream.read(DataInputStream.java:132)****
>
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
> ****
>
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:317)*
> ***
>
>         at java.io.FilterInputStream.read(FilterInputStream.java:90)****
>
>         at
> org.apache.hadoop.tools.util.ThrottledInputStream.read(ThrottledInputStream.java:70)
> ****
>
>         at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.readBytes(RetriableFileCopyCommand.java:198)
> ****
>
>         ... 16 more****
>
> ** **
>
> ** **
>
> ** **
>

Mime
View raw message