hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaofeng SHI (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-18885) HFileOutputFormat2 hardcodes default FileOutputCommitter
Date Wed, 27 Sep 2017 01:25:00 GMT
Shaofeng SHI created HBASE-18885:

             Summary: HFileOutputFormat2 hardcodes default FileOutputCommitter
                 Key: HBASE-18885
                 URL: https://issues.apache.org/jira/browse/HBASE-18885
             Project: HBase
          Issue Type: Bug
          Components: mapreduce
            Reporter: Shaofeng SHI

Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. The original reporting
is in KYLIN-2788[1]. After some investigation, we found this class always uses the default
"FileOutputCommitter", see [2], regardless of the job's configuration; so it always writing
to "_temporary" folder. Since AWS EMR configured to use DirectOutputCommitter for S3, then
this problem occurs: Hadoop expects to see the file directly under output path, while the
RecordWriter generates them in "_temporary" folder. This caused no data be loaded to HTable.

Seems this problem exists in all versions so far.

[1] https://issues.apache.org/jira/browse/KYLIN-2788
[2] https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193

This message was sent by Atlassian JIRA

View raw message