hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Petter von Dolwitz (Hem)" <>
Subject Re: hive.query.string not reflecting the current query
Date Tue, 03 Dec 2013 19:22:54 GMT
Yes, it seems related. I think the query string is not refreshed when hive
decides to run without a map reduce job. Problem is that I try to interact
with the query string to apply an early filter in the record reader. Any
other known way to detect that a map reduce job is not spawned so that I
can work around this issue?


Den tisdagen den 3:e december 2013 skrev Adam Kawa:

> Hmmm?
> Maybe it is related to the fact, that a query:
> > select * from mytable limit 100;
> does not start any MapReduce job. It is starts a reading operation from
> HDFS (and a communication with MetaStore to know what is the schema and how
> to parse the data using InputFormat and SerDe).
> For example, If you run a query that has the same functionality (i.e. to
> show all content of the table by specifying all columns in SELECT)
> > select column1, column2, ... columnN from mytable limit 100;
> then a map-only job will be started and maybe (?) hive.query.string will
> contain this query..
> 2013/12/3 Petter von Dolwitz (Hem) <<javascript:_e({},
'cvml', '');>
> >
>> Hi,
>> I use hive 0.11 with a five machine cluster. I am reading the property
>> hive.query.string from a custom RecordReader (used for reading external
>> tables).
>> If I first invoke a query like
>> select * from mytable where mycolumn='myvalue';
>> I get the correct query string in this property.
>> If I then invoke
>> select * from mytable limit 100;
>> the property hive.query.string still contains the first query. Seems like
>> hive uses local mode for the second query. Don't know if it is related.
>> Anybody knows why the query string is not updated in the second case?
>> Thanks,
>> Petter

View raw message