hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fang Xin <nusfang...@gmail.com>
Subject Re: how to overwrite output in HDFS?
Date Tue, 03 Apr 2012 12:30:39 GMT
Hi Christoph,

Thank you for your reply.
I create such a class in the project, and build an instance of it in
main, and try to use this method included, but it didnt work.
Can you explain a little bit more about how to let this function work?

Thank you!

On Tue, Apr 3, 2012 at 6:39 PM, Christoph Schmitz
<Christoph.Schmitz@1und1.de> wrote:
> Hi Xin,
>
> you can derive your own output format class from one of the Hadoop OutputFormats and
make sure the "checkOutputSpecs" method, which usually does the checking, is empty:
>
> -----------
> public final class OverwritingTextOutputFormat<K, V> extends TextOutputFormat<K,
V> {
>    @Override
>    public void checkOutputSpecs(JobContext job) throws IOException {
>          // Nothing
>    }
> }
> -----------
>
> Regards,
> Christoph
>
> -----Ursprüngliche Nachricht-----
> Von: Fang Xin [mailto:nusfangxin@gmail.com]
> Gesendet: Dienstag, 3. April 2012 11:35
> An: mapreduce-user
> Betreff: how to overwrite output in HDFS?
>
> Hi, all
>
> I'm writing my own map-reduce code using eclipse with hadoop plug-in.
> I've specified input and output directories in the project property.
> (two folders, namely input and output)
>
> My problem is that each time when I do some modification and try to
> run it again, i have to manually delete the previous output in HDFS,
> otherwise there will be error.
> Can anyone kindly suggest how to just simply overwrite the result?
>
> Best regards,
> Xin

Mime
View raw message