hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remus Rusanu <rem...@microsoft.com>
Subject RE: Any way to preserve the .q unit tests MR execution output?
Date Sun, 09 Feb 2014 21:21:37 GMT
Thanks again, but no luck for me.
I started debugging this and my suspicion is it that is related to the fact that I do not
have any Hadoop deployed on this laptop. When testutil/hadoop is run there is $HADOP_HOME
to look at, no proper hadoop-env.sh or anything alike. I think this results in log4j not being
properly initialized for the "MR" task. I'll post back if I ever get to the end of it.

~Remus

________________________________________
From: Brock Noland <brock@cloudera.com>
Sent: Saturday, February 8, 2014 7:30 PM
To: dev@hive.apache.org
Cc: hive-dev@hadoop.apache.org
Subject: Re: Any way to preserve the .q unit tests MR execution output?

I just ran a test on my local machine and the MR logs went to /tmp/brock.

Brock

On Sat, Feb 8, 2014 at 6:12 AM, Remus Rusanu <remusr@microsoft.com> wrote:
> Thanks Brock,
>
> I do get a MR folder structure, like
>
> test/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache
> qtest/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache/job_local1514098477_0001
> qtest/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache/job_local1514098477_0001/attempt_local1514098477_0001_m_000000_0
> qtest/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache/job_local1514098477_0001/attempt_local1514098477_0001_r_000000_0
> qtest/target/tmp/hadoop-tmp//mapred/local/1098407621/taskTracker/rremus/jobcache/job_local1514098477_0001/attempt_local1514098477_0001_r_000000_0/output
>
> but no files in them. I'll try to dig and see whether the files get deleted or not created
to start with...
>
> thanks,
> ~Remus
> ________________________________________
> From: Brock Noland <brock@cloudera.com>
> Sent: Thursday, February 6, 2014 11:58 PM
> To: dev@hive.apache.org
> Cc: hive-dev@hadoop.apache.org
> Subject: Re: Any way to preserve the .q unit tests MR execution output?
>
> Not sure about saving the output of the jobs, but I believe the mr
> logs should be in target/ somewhere or in /tmp/$USER
>
> On Thu, Feb 6, 2014 at 3:34 PM, Remus Rusanu <remusr@microsoft.com> wrote:
>> When running a test that fails, I can get the output diff and the hive log, but when
the failure occurs in the query execution, I don't see how I can retrieve whatever was logged
by the execution.
>>
>> For example, I can see in the target\qfile-results\...q.out:
>>
>>
>> PREHOOK: query: SELECT cint,
>>       ...
>>                AND (ctinyint > cbigint)))
>> PREHOOK: type: QUERY
>> PREHOOK: Input: default@alltypesorc
>> PREHOOK: Output: file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-mr-10000
>> Execution failed with exit status: 2
>> Obtaining error information
>>
>> Task failed!
>> Task ID:
>>   Stage-1
>>
>> Logs:
>>
>> /Users/rremus/Development/hive/itests/qtest/target/tmp/log/hive.log
>> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>>
>> The hive.log is a bit more detailed:
>> 2014-02-06 12:17:05,741 INFO  ql.Driver (SessionState.java:printInfo(504)) - Launching
Job 1 out of 1
>> 2014-02-06 12:17:05,742 INFO  exec.Task (SessionState.java:printInfo(504)) - Number
of reduce tasks is set to 0 since there's no reduce operator
>> 2014-02-06 12:17:05,822 INFO  mr.ExecDriver (MapRedTask.java:execute(175)) - Generating
plan file file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-local-10003/plan.xml
>> 2014-02-06 12:17:05,822 INFO  log.PerfLogger (PerfLogger.java:PerfLogBegin(108))
- <PERFLOG method=serializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
>> 2014-02-06 12:17:05,823 INFO  exec.Utilities (Utilities.java:serializePlan(783))
- Serializing MapredWork via kryo
>> 2014-02-06 12:17:05,835 INFO  log.PerfLogger (PerfLogger.java:PerfLogEnd(135)) -
</PERFLOG method=serializePlan start=1391717825822 end=1391717825835 duration=13 from=org.apache.hadoop.hive.ql.exec.Utilities>
>> 2014-02-06 12:17:05,836 INFO  mr.ExecDriver (MapRedTask.java:execute(207)) - Executing:
/Users/rremus/Development/hive/itests/qtest/../../testutils/hadoop jar /Users/rremus/.m2/repository/org/apache/hive/hive-exec/0.13.0-SNAPSHOT/hive-exec-0.13.0-SNAPSHOT.jar
org.apache.hadoop.hive.ql.exec.mr.ExecDriver  -plan file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-local-10003/plan.xml
  -jobconffile file:/Users/rremus/Development/hive/itests/qtest/target/tmp/localscratchdir/hive_2014-02-06_12-17-05_641_8095220136958689426-1/-local-10002/jobconf.xml
>> 2014-02-06 12:17:15,431 ERROR exec.Task (SessionState.java:printError(513)) - Execution
failed with exit status: 2
>> 2014-02-06 12:17:15,432 ERROR exec.Task (SessionState.java:printError(513)) - Obtaining
error information
>> 2014-02-06 12:17:15,432 ERROR exec.Task (SessionState.java:printError(513)) -
>> Task failed!
>> Task ID:
>>   Stage-1
>>
>> Logs:
>>
>> 2014-02-06 12:17:15,433 ERROR exec.Task (SessionState.java:printError(513)) - /Users/rremus/Development/hive/itests/qtest/target/tmp/log/hive.log
>> 2014-02-06 12:17:15,433 ERROR mr.ExecDriver (MapRedTask.java:execute(280)) - Execution
failed with exit status: 2
>>
>> Still, at the end of running the test there is nothing left in target/tmp/localscratchdir,
scratchdir or in target/tmp/hadoop-tmp. In this case, I'm missing the output from the MR job
that failed.
>> Is there a way for me to get this missing output?
>>
>> Thanks,
>> ~Remus
>>
>>
>>
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org



--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Mime
View raw message