hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7659) fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
Date Fri, 25 May 2012 12:46:23 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283348#comment-13283348
] 

Harsh J commented on HADOOP-7659:
---------------------------------

bq. -1 javadoc. The javadoc tool appears to have generated 5 warning messages.

Was probably something else in trunk at the time. See command log below for {{mvn javadoc:javadoc}},
which I made sure to do again now before committing:

{code}

➜  trunk  svn diff
Index: hadoop-common-project/hadoop-common/CHANGES.txt
===================================================================
--- hadoop-common-project/hadoop-common/CHANGES.txt	(revision 1342586)
+++ hadoop-common-project/hadoop-common/CHANGES.txt	(working copy)
@@ -76,6 +76,9 @@
     HADOOP-8415. Add getDouble() and setDouble() in
     org.apache.hadoop.conf.Configuration (Jan van der Lugt via harsh)
 
+    HADOOP-7659. fs -getmerge isn't guaranteed to work well over non-HDFS
+    filesystems (harsh)
+
   BUG FIXES
 
     HADOOP-8177. MBeans shouldn't try to register when it fails to create MBeanName.
Index: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
===================================================================
--- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java(revision
1342586)
+++ hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java(working
copy)
@@ -307,6 +307,12 @@
     return FileUtil.fullyDelete(f);
   }
  
+  /**
+   * {@inheritDoc}
+   *
+   * (<b>Note</b>: Returned list is not sorted in any given order,
+   * due to reliance on Java's {@link File#list()} API.)
+   */
   public FileStatus[] listStatus(Path f) throws IOException {
     File localf = pathToFile(f);
     FileStatus[] results;
➜  trunk  cd hadoop-common-project/hadoop-common 
➜  hadoop-common  mvn javadoc:javadoc
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Hadoop Common 3.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] >>> maven-javadoc-plugin:2.8.1:javadoc (default-cli) @ hadoop-common >>>
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- build-helper-maven-plugin:1.5:add-source (add-source) @ hadoop-common ---
[INFO] Source directory: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/generated-sources/java
added.
[INFO] 
[INFO] --- build-helper-maven-plugin:1.5:add-test-source (add-test-source) @ hadoop-common
---
[INFO] Test Source directory: /Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/generated-test-sources/java
added.
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (compile-proto) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (save-version) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-dependency-plugin:2.1:build-classpath (build-classpath) @ hadoop-common ---
[INFO] Skipped writing classpath file '/Users/harshchouraria/Work/code/apache/root-hadoop/trunk/hadoop-common-project/hadoop-common/target/classes/mrapp-generated-classpath'.
 No changes found.
[INFO] 
[INFO] <<< maven-javadoc-plugin:2.8.1:javadoc (default-cli) @ hadoop-common <<<
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:javadoc (default-cli) @ hadoop-common ---
[INFO] 
ExcludePrivateAnnotationsStandardDoclet
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23.042s
[INFO] Finished at: Fri May 25 18:14:53 GMT+05:30 2012
[INFO] Final Memory: 11M/81M
[INFO] ------------------------------------------------------------------------
{code}
                
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
> --------------------------------------------------------------------
>
>                 Key: HADOOP-7659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20.204.0
>            Reporter: Harsh J
>            Assignee: Harsh J
>            Priority: Minor
>         Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting (part-00000,
part-00001, onwards). When you use the same with other FSes we bundle, the ordering of listing
is not guaranteed at all. This is cause of http://download.oracle.com/javase/6/docs/api/java/io/File.html#list()
which we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help pages/mans, or a
consistent ordering (similar to HDFS) must be applied atop the listing. I suspect the latter
only makes it worthy for what we include - while other FSes out there still have to deal with
this issue. Perhaps we need a recommendation doc note added to our API?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message