hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "giovanni delussu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-9084) TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed filesystem)
Date Thu, 22 Nov 2012 07:19:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

giovanni delussu updated HADOOP-9084:
-------------------------------------

    Status: Patch Available  (was: Open)

*This patch works by keeping the fs and the jobconf working directories different. In hdfs
we don't need it because the "local" fs is different from the default fs.
*potential problems of this patch: if a user creates a file ( with relative path) in the mapper/reducer
phase this file will be created in the attempt directory and then deleted on cleanup. Without
the patch the file would be created in the current working directory of the filesystem, i.e.
the directory from which the user launched hadoop.
                
> TotalOrderPartitioner fails on hadoop running on top of gpfs (or any parallel or distributed
filesystem)
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-9084
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9084
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: filecache, fs, native
>    Affects Versions: 1.0.4, 1.0.3
>            Reporter: giovanni delussu
>            Assignee: giovanni delussu
>            Priority: Critical
>             Fix For: 1.0.4, 1.0.3
>
>
> When running a job who uses TotalOrderPartitioner (like TeraSort or BulkImport of HBase)
on hadoop running on top of gpfs (instead of hdfs) the program fails to find the file _partition.lst
because is looking for it in the wrong directory. The confusion is between local fs meaning
not hdfs and local meaning distributed fs. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message