hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-18885) HFileOutputFormat2 hardcodes default FileOutputCommitter
Date Wed, 27 Sep 2017 14:02:00 GMT

     [ https://issues.apache.org/jira/browse/HBASE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ted Yu updated HBASE-18885:
---------------------------
       Resolution: Fixed
     Hadoop Flags: Reviewed
    Fix Version/s: 2.0.0-alpha-4
                   1.5.0
                   1.4.0
           Status: Resolved  (was: Patch Available)

Thanks for the patch, shaofeng

> HFileOutputFormat2 hardcodes default FileOutputCommitter
> --------------------------------------------------------
>
>                 Key: HBASE-18885
>                 URL: https://issues.apache.org/jira/browse/HBASE-18885
>             Project: HBase
>          Issue Type: Bug
>          Components: mapreduce
>            Reporter: Shaofeng SHI
>            Assignee: Shaofeng SHI
>             Fix For: 1.4.0, 1.5.0, 2.0.0-alpha-4
>
>         Attachments: HBASE-18885.branch-1.001.patch, HBASE-18885.master.001.patch
>
>
> Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. The original
reporting is in KYLIN-2788[1]. After some investigation, we found this class always uses the
default "FileOutputCommitter", see [2], regardless of the job's configuration; so it always
writing to "_temporary" folder. Since AWS EMR configured to use DirectOutputCommitter for
S3, then this problem occurs: Hadoop expects to see the file directly under output path, while
the RecordWriter generates them in "_temporary" folder. This caused no data be loaded to HTable.
> Seems this problem exists in all versions so far.
> [1] https://issues.apache.org/jira/browse/KYLIN-2788
> [2] https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message