hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Kimball <aa...@cloudera.com>
Subject Re: skip setting output path for a sequential MR job..
Date Tue, 31 Mar 2009 07:29:21 GMT
You must remove the existing output directory before running the job. This
check is put in to prevent you from inadvertently destroying or muddling
your existing output data.

You can remove the output directory in advance programmatically with code
similar to:

FileSystem fs = FileSystem.get(conf); // use your JobConf here
fs.delete(new Path("/path/to/output/dir"), true);

See
http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/fs/FileSystem.htmlfor
more details.

- Aaron


On Mon, Mar 30, 2009 at 9:25 PM, some speed <speed.some@gmail.com> wrote:

> Hello everyone,
>
> Is it necessary to redirect the ouput of reduce to a file? When I am trying
> to run the same M-R job more than once, it throws an error that the output
> file already exists. I dont want to use command line args so I hard coded
> the file name into the program.
>
> So, Is there a way , I could delete a file on HDFS programatically?
> or can i skip setting a output file path n just have my output print to
> console?
> or can I just append to an existing file?
>
>
> Any help is appreciated. Thanks.
>
> -Sharath
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message