hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael G. Noll" <michael.g.n...@googlemail.com>
Subject Configuring Hive to automatically create _SUCCESS files
Date Thu, 15 Sep 2011 13:59:00 GMT
Hello,

is there a way to configure Hive 0.7.1 so that it creates _SUCCESS files
similar to Hadoop MapReduce and Pig?

Hadoop has the property "mapreduce.fileoutputcommitter.marksuccessfuljobs"
(see FileOutputCommitter.java), which is as of Hadoop 0.20.203.0 enabled by
default.  In Pig 0.9.0, you can enable success files by e.g. setting the
property on the command line when running a Pig script (see PIG-1433 [1]):

    $ pig -Dmapreduce.fileoutputcommitter.marksuccessfuljobs=true ...

I have unsuccessfully tried to enable success files in Hive 0.7.1 in
combination with Hadoop 0.20.203.0:

    # has no effect
    $ hive -hiveconf mapreduce.fileoutputcommitter.marksuccessfuljobs=true
...

The job configuration XML file lists the property as being set to true, but
Hive will not create a success file.

Best,
Michael


[1] https://issues.apache.org/jira/browse/PIG-1433

Mime
View raw message