hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-7090) BigMapOutput example doesn't work with paths off cluster fs
Date Mon, 30 Apr 2018 14:58:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16458638#comment-16458638
] 

Steve Loughran commented on MAPREDUCE-7090:
-------------------------------------------

{code}
> hadoop jar hadoop-mapreduce-client-jobclient-tests.jar testbigmapoutput -input s3a://hwdev-steve-new/bigMapInputDir
-output s3a://hwdev-steve-new/bigMapOutputDir -create 100
java.lang.IllegalArgumentException: Wrong FS: s3a://hwdev-steve-new/bigMapOutputDir, expected:
hdfs://testcluster
	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1569)
	at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1566)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1581)
	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
	at org.apache.hadoop.mapred.BigMapOutput.run(BigMapOutput.java:139)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.mapred.BigMapOutput.main(BigMapOutput.java:162)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
	at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
	at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:139)
	at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:147)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
{code}

> BigMapOutput example doesn't work with paths off cluster fs
> -----------------------------------------------------------
>
>                 Key: MAPREDUCE-7090
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7090
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: examples
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> You can't pass an object store path to bigmapoutput, because it uses the default fs,
not the path FS, to work with the directories.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message