hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matouk IFTISSEN <matouk.iftis...@ysance.com>
Subject Re: How Can I store the Hive query result in one file ?
Date Thu, 04 Jul 2013 15:20:35 GMT
Thanks for your responses,
effctively  the answer of Bertrand make this possible: the set of hive
properities below froce thet job to write the hive result in one file
whithout specifing the name (0000_0) :
 set hive.exec.reducers.max = 1;

set mapred.reduce.tasks = 1;

for Nitin, I want to store the results of SELECT not the stdout (log) of
execution of the query, is this applicable for the results of SELECT?




2013/7/4 Michael Malak <michaelmalak@yahoo.com>

> I have found that for output larger than a few GB, redirecting stdout
> results in an incomplete file.  For very large output, I do CREATE TABLE
> MYTABLE AS SELECT ... and then copy the resulting HDFS files directly out
> of /user/hive/warehouse.
>
>    *From:* Bertrand Dechoux <dechouxb@gmail.com>
> *To:* user@hive.apache.org
> *Sent:* Thursday, July 4, 2013 7:09 AM
> *Subject:* Re: How Can I store the Hive query result in one file ?
>
> The question is what is the volume of your output. There is one file per
> output task (map or reduce) because that way each can write it
> independently and in parallel. That's how mapreduce work. And except by
> forcing the number of tasks to 1, there is no certain way to have one
> output file.
>
> But indeed if the volume is low enough, you could also capture the
> standard output into a local file like Nitin described.
>
> Bertrand
>
>
> On Thu, Jul 4, 2013 at 12:38 PM, Nitin Pawar <nitinpawar432@gmail.com>wrote:
>
> will hive -e "query" > filename  or hive -f query.q > filename will do ?
>
> you specially want it to write into a named file on hdfs only?
>
>
> On Thu, Jul 4, 2013 at 3:12 PM, Matouk IFTISSEN <
> matouk.iftissen@ysance.com> wrote:
>
> Hello Hive users,
> Is there a manner to store the Hive  query result (SELECT *.....) in a
> specfique and alone file (given the file name) like (INSERT OVERWRITE LOCAL
> DIRECTORY '/directory_path_name/')?
> Thanks for your answers
>
>
>
>
> --
> Nitin Pawar
>
>
>
>
> --
> Bertrand Dechoux
>
>
>

Mime
View raw message