hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryu Kobayashi (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-11353) Map env does not reflect in the Local Map Join
Date Thu, 23 Jul 2015 09:31:04 GMT

     [ https://issues.apache.org/jira/browse/HIVE-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ryu Kobayashi updated HIVE-11353:
---------------------------------
    Attachment: HIVE-11353.1.patch

> Map env does not reflect in the Local Map Join
> ----------------------------------------------
>
>                 Key: HIVE-11353
>                 URL: https://issues.apache.org/jira/browse/HIVE-11353
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Ryu Kobayashi
>         Attachments: HIVE-11353.1.patch
>
>
> mapreduce.map.env is not reflected when the Local Map Join is ran. Following a sample
query:
> {code}
> hive> set mapreduce.map.env=AAA=111,BBB=222,CCC=333;
> hive> select
>     >   reflect("java.lang.System", "getenv", "CCC") as CCC,
>     >   a.AAA,
>     >   b.BBB
>     > from (
>     >   SELECT
>     >     reflect("java.lang.System", "getenv", "AAA") as AAA
>     >   from
>     >     foo
>     > ) a
>     > join (
>     >   select
>     >     reflect("java.lang.System", "getenv", "BBB") as BBB
>     >   from
>     >     foo
>     > ) b
>     > limit 1;
> Warning: Map Join MAPJOIN[10][bigTable=?] in task 'Stage-3:MAPRED' is a cross product
> Query ID = root_20150716013643_a8ca1539-68ae-4f13-b9fa-7a8b88f01f13
> Total jobs = 1
> 15/07/16 01:36:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
> Execution log at: /tmp/root/root_20150716013643_a8ca1539-68ae-4f13-b9fa-7a8b88f01f13.log
> 2015-07-16 01:36:47     Starting to launch local task to process map join;      maximum
memory = 477102080
> 2015-07-16 01:36:48     Dump the side-table for tag: 0 with group count: 1 into file:
file:/tmp/root/9b900f85-d5e4-4632-90bc-19f4bac516ff/hive_2015-07-16_01-36-43_217_8812243019719259041-1/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> 2015-07-16 01:36:48     Uploaded 1 File to: file:/tmp/root/9b900f85-d5e4-4632-90bc-19f4bac516ff/hive_2015-07-16_01-36-43_217_8812243019719259041-1/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
(282 bytes)
> 2015-07-16 01:36:48     End of local task; Time Taken: 0.934 sec.
> Execution completed successfully
> MapredLocal task succeeded
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_1436962851556_0015, Tracking URL = http://hadoop27:8088/proxy/application_1436962851556_0015/
> Kill Command = /usr/local/hadoop/bin/hadoop job  -kill job_1436962851556_0015
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 0
> 2015-07-16 01:36:56,488 Stage-3 map = 0%,  reduce = 0%
> 2015-07-16 01:37:01,656 Stage-3 map = 100%,  reduce = 0%, Cumulative CPU 1.28 sec
> MapReduce Total cumulative CPU time: 1 seconds 280 msec
> Ended Job = job_1436962851556_0015
> MapReduce Jobs Launched:
> Stage-Stage-3: Map: 1   Cumulative CPU: 1.28 sec   HDFS Read: 5428 HDFS Write: 13 SUCCESS
> Total MapReduce CPU Time Spent: 1 seconds 280 msec
> OK
> 333     null    222
> Time taken: 19.562 seconds, Fetched: 1 row(s)
> {code}
> The attached patch will include those taken from Hadoop's code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message