Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
From: Fran O <franobeta@gmail.com>
Date: Thu, 16 Mar 2017 12:20:58 -0400
Message-ID: <CAGdUvLHCrKraQ=+yEjYSQiT+x9b-zpv0VqMskrHNm4DRvywGrA@mail.gmail.com>
Subject: HFILE creation to use a different committer
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=94eb2c13e2623d1642054adb72c9
archived-at: Thu, 16 Mar 2017 16:54:18 -0000

--94eb2c13e2623d1642054adb72c9
Content-Type: text/plain; charset=UTF-8

Hi folks,

I would like to hear some thoughts on the following use case:

I use a custom MR job to create HFiles . This MR writes the HFiles into S3.

I was trying to change the Outputcommitter in order to have the reducers
writing directly the HFiles to the final destination on S3. After some
tests setting the Outputcommitter to be the DirectoOutputcommitter, the
tasks are always using the FileOutputCommitter.

>> HFileOutputFormat2.configureIncrementalLoad(job, hTable);
>> FileOutputFormat.setOutputPath(job, outputPath);
>> FileOutputFormat.setCompressOutput(job, true);
>> FileOutputFormat.setOutputCompressorClass(job, >>SnappyCodec.class);

Looking at the code of the FileOutputFormat methods
<https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.html>
I see a *getOutputCommitter
<https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.html#getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext)>
method
but not a set method for the OutputCommitter. *

*Could someone bring some light on how to change the OutputCommitter for
the tasks?*

Thank you,
Fran

--94eb2c13e2623d1642054adb72c9--