hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bejoy KS" <bejoy...@yahoo.com>
Subject Re: Passing date as command line arguments
Date Sat, 04 Aug 2012 07:39:37 GMT
Yes that is the right issue. Variable substitution is not happening. I can't say much here
as I haven't tried out this on 0.6 . The code on my blog post is based on 0.7 or higher version
I guess.  

Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: Techy Teck <comptechgeeky@gmail.com>
Date: Sat, 4 Aug 2012 00:35:20 
To: <user@hive.apache.org>; <bejoy_ks@yahoo.com>
Reply-To: user@hive.apache.org
Subject: Re: Passing date as command line arguments

Hi Bejoy,

I found something very strange. When I actually viewed the Hive log file, I
found something like this- See the thing in red color. Why is it like so?

*SessionStart SESSION_ID="rjamal_201208041433" TIME="1344065620956"*
*QueryStart QUERY_STRING=" select * from lip_data_quality where dt = '
${hiveconf:start_date}'"
QUERY_ID="rjamal_20120804143333_3719afde-117d-4c6b-a322-0cff6e4b4c5d"
TIME="1344065629740"*
*Counters
plan="{"queryId":"rjamal_20120804143333_3719afde-117d-4c6b-a322-0cff6e4b4c5d","queryType":null,"queryAttributes":{"queryString":"
select * from lip_data_quality where dt =
'${hiveconf:start_date}'"},"queryCounters":"null","stageGraph":{"nodeType":"1","roots":"null","adjacencyList":"]"},"stageList":"]","done":"false","started":"true"}"
TIME="1344065629743"*
*QueryEnd QUERY_STRING=" select * from lip_data_quality where dt =
'${hiveconf:start_date}'"
QUERY_ID="rjamal_20120804143333_3719afde-117d-4c6b-a322-0cff6e4b4c5d"
QUERY_RET_CODE="0" QUERY_NUM_TASKS="0" TIME="1344065629745"*
*Counters
plan="{"queryId":"rjamal_20120804143333_3719afde-117d-4c6b-a322-0cff6e4b4c5d","queryType":null,"queryAttributes":{"queryString":"
select * from lip_data_quality where dt =
'${hiveconf:start_date}'"},"queryCounters":"null","stageGraph":{"nodeType":"1","roots":"null","adjacencyList":"]"},"stageList":"]","done":"true","started":"true"}"
TIME="1344065629745"*

On Sat, Aug 4, 2012 at 12:30 AM, Techy Teck <comptechgeeky@gmail.com> wrote:

> I cannot try that on Higher version of Hive as I am working in a company
> where they have installed only Hive 0.6 on the Ares cluster.
>
> :(
>
>
>
>
> On Sat, Aug 4, 2012 at 12:29 AM, Bejoy KS <bejoy_ks@yahoo.com> wrote:
>
>> **
>>
>> Try it on a higher version of hive and let me know if that doesn't work
>> still. 0.9 should be good.
>>
>>
>> Regards
>> Bejoy KS
>>
>> Sent from handheld, please excuse typos.
>> ------------------------------
>> *From: * Techy Teck <comptechgeeky@gmail.com>
>> *Date: *Sat, 4 Aug 2012 00:26:44 -0700
>> *To: *<user@hive.apache.org>; <bejoy_ks@yahoo.com>
>> *ReplyTo: * user@hive.apache.org
>> *Subject: *Re: Passing date as command line arguments
>>
>> Yes it is String for me here.
>>
>> This is the table structure by the way-
>>
>> *create table lip_data_quality*
>> *( buyer_id bigint,*
>> *  total_chkout bigint,*
>> *  total_errpds bigint*
>> * )*
>> * partitioned by (dt string)*
>> *row format delimited fields terminated by '\t'*
>> *stored as sequencefile*
>> *location '/apps/hdmi-technology/b_apdpds/lip-data-quality'*
>> *;*
>>
>> in which I am using dt in the where clause in my query. I am running *Hive
>> 0.6.*
>>
>>
>>
>> On Sat, Aug 4, 2012 at 12:24 AM, Bejoy KS <bejoy_ks@yahoo.com> wrote:
>>
>>> **
>>> I tried the same query on my end, It is working fine for me without any
>>> issues.
>>>
>>> By de way the data type for 'dt' is String itself right?
>>> Regards
>>> Bejoy KS
>>>
>>> Sent from handheld, please excuse typos.
>>> ------------------------------
>>> *From: * Techy Teck <comptechgeeky@gmail.com>
>>> *Date: *Sat, 4 Aug 2012 00:21:37 -0700
>>> *To: *<user@hive.apache.org>; <bejoy_ks@yahoo.com>
>>> *ReplyTo: * user@hive.apache.org
>>> *Subject: *Re: Passing date as command line arguments
>>>
>>> I was not able to understand. So what changes I need to make? This is my
>>> below *hivetest1.hql-*
>>> *
>>> *
>>> *select * from lip_data_quality where dt = '${hiveconf:start_date}';*
>>> *
>>> *
>>> And by this I am executing the above hql file-
>>>
>>> *hive -f hivetest1.hql -hiveconf start_date=20120709*
>>> *
>>> *
>>> *
>>> *
>>> Any thoughts what changes I need to make?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Aug 4, 2012 at 12:16 AM, Bejoy KS <bejoy_ks@yahoo.com> wrote:
>>>
>>>> **
>>>> Yes.
>>>>
>>>> From the logs the query being executed is
>>>> select * from lip_data_quality where dt=20120709
>>>>
>>>> But here the dt is not in quotes. It should be like
>>>> select * from lip_data_quality where dt='20120709';
>>>> Regards
>>>> Bejoy KS
>>>>
>>>> Sent from handheld, please excuse typos.
>>>> ------------------------------
>>>> *From: * Techy Teck <comptechgeeky@gmail.com>
>>>> *Date: *Sat, 4 Aug 2012 00:07:07 -0700
>>>> *To: *<user@hive.apache.org>; Bejoy Ks<bejoy_ks@yahoo.com>
>>>> *ReplyTo: * user@hive.apache.org
>>>> *Subject: *Re: Passing date as command line arguments
>>>>
>>>> I found this thing from the Hive log. This is what you are looking for?-
>>>>
>>>> *SessionStart SESSION_ID="rjamal_201208041359" TIME="1344063565655"*
>>>> *QueryStart QUERY_STRING="select * from lip_data_quality where
>>>> dt=20120709"
>>>> QUERY_ID="rjamal_20120804135959_58e245e8-086a-47c2-9ed3-6b952e4a5486"
>>>> TIME="1344063597488"*
>>>> *Counters
>>>> plan="{"queryId":"rjamal_20120804135959_58e245e8-086a-47c2-9ed3-6b952e4a5486","queryType":null,"queryAttributes":{"queryString":"select
>>>> * from lip_data_quality where dt=20120709"},"queryCo*
>>>> *unters":"null","stageGraph":{"nodeType":"1","roots":"null","adjacencyList":"]"},"stageList":"]","done":"false","started":"true"}"
>>>> TIME="1344063597501"*
>>>> *QueryEnd QUERY_STRING="select * from lip_data_quality where
>>>> dt=20120709"
>>>> QUERY_ID="rjamal_20120804135959_58e245e8-086a-47c2-9ed3-6b952e4a5486"
>>>> QUERY_RET_CODE="0" QUERY_NUM_TASKS="0" TIME="134406359*
>>>> *7503"*
>>>> *Counters
>>>> plan="{"queryId":"rjamal_20120804135959_58e245e8-086a-47c2-9ed3-6b952e4a5486","queryType":null,"queryAttributes":{"queryString":"select
>>>> * from lip_data_quality where dt=20120709"},"queryCo*
>>>> *unters":"null","stageGraph":{"nodeType":"1","roots":"null","adjacencyList":"]"},"stageList":"]","done":"true","started":"true"}"
>>>> TIME="1344063597504"*
>>>> *
>>>> *
>>>> *
>>>> *
>>>>
>>>>
>>>>
>>>> On Fri, Aug 3, 2012 at 11:58 PM, Techy Teck <comptechgeeky@gmail.com>wrote:
>>>>
>>>>> Yes that query gives me the result back correctly.
>>>>>
>>>>> How can I see the hive log Bejoy?  Basically I am not sure which hive
>>>>> log and what is the path where I can see that thing?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Aug 3, 2012 at 11:52 PM, Bejoy Ks <bejoy_ks@yahoo.com>
wrote:
>>>>>
>>>>>>  Hi Techy
>>>>>>
>>>>>> Are you getting the results for the same query directly from CLI,
I
>>>>>> mean does the below query return correct results
>>>>>> select * from lip_data_quality where dt = '20120709';
>>>>>>
>>>>>> If the above query returns results, can you check the hive logs and
>>>>>> see what is the actual query being executed?
>>>>>>
>>>>>> Regards,
>>>>>> Bejoy KS
>>>>>>
>>>>>>   ------------------------------
>>>>>> *From:* Techy Teck <comptechgeeky@gmail.com>
>>>>>> *To:* user@hive.apache.org
>>>>>> *Cc:* Vijay Dirisala <techvd@gmail.com>
>>>>>> *Sent:* Saturday, August 4, 2012 12:11 PM
>>>>>> *Subject:* Re: Passing date as command line arguments
>>>>>>
>>>>>> Thanks Vijay for the suggestion. I also tried that and it still
>>>>>> didn't worked for me.
>>>>>>
>>>>>> Any thoughts why it is not working?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 3, 2012 at 11:32 PM, Vijay <techvd@gmail.com> wrote:
>>>>>>
>>>>>> There can't be a space in the variable name. Try this:
>>>>>> ${hiveconf:start_date}
>>>>>>
>>>>>> On Fri, Aug 3, 2012 at 11:06 PM, Techy Teck <comptechgeeky@gmail.com>
>>>>>> wrote:
>>>>>> > Can anyone help me out on this?
>>>>>> >
>>>>>> > I am stuck on this and I have no clue what wrong I am doing?
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Fri, Aug 3, 2012 at 3:48 PM, Techy Teck <comptechgeeky@gmail.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> I have my below query in test1.hql file. I am trying to
pass the
>>>>>> date (dt)
>>>>>> >> as the command line argument.
>>>>>> >>
>>>>>> >>
>>>>>> >> select * from lip_data_quality where dt = '${hiveconf:
>>>>>> start_date}';
>>>>>> >>
>>>>>> >>
>>>>>> >> So whenever I try to run the above test1.hql file like this-
>>>>>> >>
>>>>>> >>  hive -f hivetest1.hql -hiveconf start_date=20120709
>>>>>> >>
>>>>>> >> I get zero records back. But the data is there in that table
for
>>>>>> that
>>>>>> >> date. Why is it so? Something wrong I am doing?
>>>>>> >>
>>>>>> >> Can anyone help me out here? I was following Bejoy's Article
here
>>>>>> >>
>>>>>> http://kickstarthadoop.blogspot.com/2011/10/include-values-during-execution-time-in.html
>>>>>> >>
>>>>>> >>
>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message