hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugene Koifman <ekoif...@hortonworks.com>
Subject Re: WebHCat MapReduce Job Syntax
Date Mon, 30 Dec 2013 22:53:14 GMT
It looks like in 0.11 it writes to stderr (limited logging anyway).

Perhaps you can try adding '*statusdir*' param to your REST call and see if
anything useful is written to that directory.


On Mon, Dec 30, 2013 at 2:22 PM, Jonathan Hodges <hodgesz@gmail.com> wrote:

> I don't see 'TrivialExecService' output in the jobtracker or tasktracker
> logs.  We are using hive 0.11 though so maybe not set to DEBUG?
>
>
> On Mon, Dec 30, 2013 at 2:11 PM, Eugene Koifman <ekoifman@hortonworks.com>wrote:
>
>> Is there any output from TrivialExecService class in any hadoop logs?
>>  (it's DEBUG level log4j output in hive 0.12).
>> It should print the command that TempletonControllerJob's launcher task
>> (LaunchMapper) is trying to launch
>>
>>
>> On Mon, Dec 30, 2013 at 12:55 PM, Jonathan Hodges <hodgesz@gmail.com>wrote:
>>
>>> I didn't try that before, but I just did.
>>>
>>> curl -s -d user.name=hadoop \
>>>
>>> >        -d
>>> jar=/tmp/camus-non-avro-consumer-1.0-SNAPSHOT-jar-with-dependencies.jar \
>>>
>>> >        -d class=com.linkedin.camus.etl.kafka.CamusJob \
>>>
>>> >        -d arg=-P \
>>>
>>> >        -d arg=/tmp/camus_non_avro.properties \
>>>
>>> >        '
>>> http://internal-daalt-hcatalog-1507773817.us-east-1.elb.amazonaws.com/templeton/v1/mapreduce/jar
>>> '
>>>
>>> {"id":"job_201312212124_0166"}
>>>
>>> DEBUG | 30 Dec 2013 20:33:43,157 | org.apache.hcatalog.templeton.Server
>>> | queued job job_201312212124_0166 in 300 ms
>>>
>>> I still the same behavior with just the TempletonControllerJob getting
>>> kicked off and ending successfully without the Camus job starting.  I
>>> didn't see any errors in the jobtracker or tasktracker logs.  It just seems
>>> to silently fail and I can't figure out why.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Dec 30, 2013 at 12:35 PM, Eugene Koifman <
>>> ekoifman@hortonworks.com> wrote:
>>>
>>>> have you tried adding
>>>> -d arg=-P
>>>> before
>>>> -d arg=/tmp/....properites
>>>>
>>>>
>>>>
>>>> On Mon, Dec 30, 2013 at 11:14 AM, Jonathan Hodges <hodgesz@gmail.com>wrote:
>>>>
>>>>> Sorry accidentally hit send before adding the lines from webhcat.log
>>>>>
>>>>> DEBUG | 30 Dec 2013 19:08:01,042 |
>>>>> org.apache.hcatalog.templeton.Server | queued job job_201312212124_0161
in
>>>>> 267 ms
>>>>>
>>>>> DEBUG | 30 Dec 2013 19:08:38,880 |
>>>>> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>>>>> /templeton-hadoop/jobs/job_201312212124_0161/notified: File does not
exist:
>>>>> /templeton-hadoop/jobs/job_201312212124_0161/notified
>>>>>
>>>>> DEBUG | 30 Dec 2013 19:08:38,881 |
>>>>> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>>>>> /templeton-hadoop/jobs/job_201312212124_0161/callback: File does not
exist:
>>>>> /templeton-hadoop/jobs/job_201312212124_0161/callback
>>>>>
>>>>>
>>>>> Any ideas?
>>>>>
>>>>>
>>>>> On Mon, Dec 30, 2013 at 12:13 PM, Jonathan Hodges <hodgesz@gmail.com>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to kick off a mapreduce job via WebHCat.  The following
>>>>>> is the hadoop jar command.
>>>>>>
>>>>>> hadoop jar
>>>>>> /home/hadoop/camus-non-avro-consumer-1.0-SNAPSHOT-jar-with-dependencies.jar
>>>>>> com.linkedin.camus.etl.kafka.CamusJob -P
>>>>>> /home/hadoop/camus_non_avro.properties
>>>>>>
>>>>>> As you can see there is an application specific parameter '-P' which
>>>>>> designates the properties file location.  How do I pass this to WebHCat?
>>>>>>
>>>>>> Referring to the docs (
>>>>>> https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference+MapReduceJar)
>>>>>> I came up with the following.
>>>>>>
>>>>>> curl -s -d user.name=hadoop \
>>>>>>        -d
>>>>>> jar=/tmp/camus-non-avro-consumer-1.0-SNAPSHOT-jar-with-dependencies.jar
\
>>>>>>        -d class=com.linkedin.camus.etl.kafka.CamusJob \
>>>>>>        -d arg=/tmp/camus_non_avro.properties \
>>>>>>        '
>>>>>> http://internal-daalt-hcatalog-1507773817.us-east-1.elb.amazonaws.com/templeton/v1/mapreduce/jar
>>>>>> '
>>>>>>
>>>>>> This command gets the following response from WebHCat
>>>>>> {"id":"job_201312212124_0161"}
>>>>>>
>>>>>> However I only see TempletonControllerJob in the jobtracker UI. 
I
>>>>>> don't see the Camus jobs that will show up if executed at the command-line.
>>>>>>
>>>>>> The following are the only things showing in webhcat.log
>>>>>>
>>>>>>
>>>>>> The jar and properties files are in the /tmp directory on HDFS.
>>>>>>
>>>>>> hadoop fs -ls /tmp
>>>>>> -rw-r--r--   2 hadoop supergroup   41456481 2013-12-27 17:45
>>>>>> /tmp/camus-non-avro-consumer-1.0-SNAPSHOT-jar-with-dependencies.jar
>>>>>> -rw-r--r--   2 hadoop supergroup       2605 2013-12-27 17:45
>>>>>> /tmp/camus_non_avro.properties
>>>>>>
>>>>>
>>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>
>>>
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message