hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andre Araujo <ara...@pythian.com>
Subject Re: how to control hive log location on 0.13?
Date Sat, 19 Jul 2014 06:38:01 GMT
Hi, Yang,

you're running your mapreduce jobs in Hadoop's local mode, and in that mode
all the Hive MR logging is handled through log4j on your local machine,
which is what this log file is about.

The log location and naming is controlled by the property
log4j.appender.FA.File in the Hive log4j properties file.

Cheers,
Andre


On 19 July 2014 15:24, Yang <teddyyyy123@gmail.com> wrote:

> 2014-07-18 15:03:37,774 INFO  mr.ExecDriver
> (SessionState.java:printInfo(537)) - Execution log at:
> /tmp/myuser/myuser_2014071815030
> 3_56bf6bb0-db30-4dbc-807c-9023ce4103f4.log
> 2014-07-18 15:03:37,864 WARN  conf.Configuration
> (Configuration.java:loadProperty(2358)) -
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_
> 6799963466906099923-1/-local-10011/jobconf.xml:an attempt to override
> final parameter: mapreduce.job.end-notification.max.retry.interval;  Igno
> ring.
> 2014-07-18 15:03:37,871 WARN  conf.Configuration
> (Configuration.java:loadProperty(2358)) -
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_
> 6799963466906099923-1/-local-10011/jobconf.xml:an attempt to override
> final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
> 2014-07-18 15:03:37,951 INFO  log.PerfLogger
> (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=deserializePlan
> from=org.apache.hadoop.hive.
> ql.exec.Utilities>
> 2014-07-18 15:03:37,951 INFO  exec.Utilities
> (Utilities.java:deserializePlan(822)) - Deserializing MapredLocalWork via
> kryo
> 2014-07-18 15:03:38,237 INFO  log.PerfLogger
> (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=deserializePlan
> start=1405721017951 end=14057
> 21018237 duration=286 from=org.apache.hadoop.hive.ql.exec.Utilities>
> 2014-07-18 15:03:38,246 INFO  mr.MapredLocalTask
> (SessionState.java:printInfo(537)) - 2014-07-18 03:03:38       Starting to
> launch local task t
> o process map join;      maximum memory = 4261937152
> 2014-07-18 15:03:38,261 INFO  mr.MapredLocalTask
> (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
> null-subquery2:a-subquery2
> :dpkg_cntr:dpkg_wtransaction_p2_id_user_30m created
> 2014-07-18 15:03:38,263 INFO  mr.MapredLocalTask
> (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
> null-subquery2:a-subquery2
> :dpkg:dpkg_wtransaction_p2_id_user_30m created
> 2014-07-18 15:03:38,264 INFO  mr.MapredLocalTask
> (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
> null-subquery2:a-subquery2
> :xclick:b:wtrans_data_map_p2_30m created
> 2014-07-18 15:03:38,266 INFO  mr.MapredLocalTask
> (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
> null-subquery1:a-subquery1
> :dpkg_cntr:dpkg_wtransaction_id_user_30m created
> 2014-07-18 15:03:38,268 INFO  mr.MapredLocalTask
> (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
> null-subquery1:a-subquery1
> :dpkg:dpkg_wtransaction_id_user_30m created
> 2014-07-18 15:03:38,269 INFO  mr.MapredLocalTask
> (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for
> null-subquery1:a-subquery1
> :xclick:b:wtrans_data_map_30m created
> ---------------------------------------------
>
> whole bunch of stuff omitted here
>
> --------------------------------------
>
>
> --------------------------------------
> 2014-07-18 15:04:08,678 INFO  exec.HashTableSinkOperator
> (HashTableSinkOperator.java:flushToFile(278)) - Temp URI for side table:
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2
> 2014-07-18 15:04:08,678 INFO  exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:08       Dump the
> side-table into file:
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile11--.hashtable
> 2014-07-18 15:04:09,943 INFO  exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:09       Uploaded 1
> File to:
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile11--.hashtable
> (58010217 bytes)
> 2014-07-18 15:04:09,943 INFO  exec.HashTableSinkOperator
> (Operator.java:close(591)) - 6 Close done
> 2014-07-18 15:04:09,943 INFO  exec.SelectOperator
> (Operator.java:close(591)) - 5 Close done
> 2014-07-18 15:04:09,943 INFO  exec.TableScanOperator
> (Operator.java:close(591)) - 4 Close done
> 2014-07-18 15:04:09,951 INFO  mapred.FileInputFormat
> (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
> 2014-07-18 15:04:10,008 INFO  mapred.FileInputFormat
> (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
> 2014-07-18 15:04:11,526 INFO  exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:11       Processing
> rows:
>         200000  Hashtable size: 199999  Memory usage:   190041576
> percentage:     0.045
> 2014-07-18 15:04:11,950 INFO  exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:11       Processing
> rows:
>         300000  Hashtable size: 299999  Memory usage:   250890416
> percentage:     0.059
> 2014-07-18 15:04:12,456 INFO  exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:12       Processing
> rows:
>         400000  Hashtable size: 399999  Memory usage:   304697120
> percentage:     0.071
> 2014-07-18 15:04:12,744 INFO  exec.TableScanOperator
> (Operator.java:close(574)) - 11 finished. closing...
> 2014-07-18 15:04:12,745 INFO  exec.FilterOperator
> (Operator.java:close(574)) - 12 finished. closing...
> 2014-07-18 15:04:12,745 INFO  exec.FilterOperator
> (Operator.java:logStats(838)) - PASSED:471403
> 2014-07-18 15:04:12,745 INFO  exec.FilterOperator
> (Operator.java:logStats(838)) - FILTERED:278619
> 2014-07-18 15:04:12,745 INFO  exec.SelectOperator
> (Operator.java:close(574)) - 13 finished. closing...
> 2014-07-18 15:04:12,745 INFO  exec.HashTableSinkOperator
> (Operator.java:close(574)) - 14 finished. closing...
> 2014-07-18 15:04:12,745 INFO  exec.HashTableSinkOperator
> (HashTableSinkOperator.java:flushToFile(278)) - Temp URI for side table:
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2
> 2014-07-18 15:04:12,745 INFO  exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:12       Dump the
> side-table into file:
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile31--.hashtable
> 2014-07-18 15:04:13,836 INFO  exec.HashTableSinkOperator
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:13       Uploaded 1
> File to:
> file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile31--.hashtable
> (47251316 bytes)
> 2014-07-18 15:04:13,836 INFO  exec.HashTableSinkOperator
> (Operator.java:close(591)) - 14 Close done
> 2014-07-18 15:04:13,836 INFO  exec.SelectOperator
> (Operator.java:close(591)) - 13 Close done
> 2014-07-18 15:04:13,836 INFO  exec.FilterOperator
> (Operator.java:close(591)) - 12 Close done
> 2014-07-18 15:04:13,836 INFO  exec.TableScanOperator
> (Operator.java:close(591)) - 11 Close done
> 2014-07-18 15:04:13,837 INFO  mr.MapredLocalTask
> (SessionState.java:printInfo(537)) - 2014-07-18 03:04:13       End of local
> task; Time Taken: 35.59 sec.
>
>


-- 
André Araújo
Big Data Consultant/Solutions Architect
The Pythian Group - Australia - www.pythian.com

Office (calls from within Australia): 1300 366 021 x1270
Office (international): +61 2 8016 7000  x270 *OR* +1 613 565 8696   x1270
Mobile: +61 410 323 559
Fax: +61 2 9805 0544
IM: pythianaraujo @ AIM/MSN/Y! or araujo@pythian.com @ GTalk

“Success is not about standing at the top, it's the steps you leave behind.”
— Iker Pou (rock climber)

-- 


--




Mime
View raw message