hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 王锋 <wfeng1...@163.com>
Subject Re:Re: hive MR job need so large memory
Date Thu, 22 Dec 2011 01:30:54 GMT
ye, I will be looking at this question. 





在 2011-12-21 21:49:31,"北斗七" <beidou77@gmail.com> 写道:
>i aslo meet the same question,so if you know how to slove it ,please
>tell me .Ths.
>
>在 2011年12月21日 下午6:00,王锋 <wfeng1982@163.com> 写道:
>> I just create a table fool; And table data is :
>> sellerId consumerId
>> 1000,4000
>> 1000,4001
>> 1002,4002
>> 1003,4000
>> 1004,4001
>> 1005,4002
>> 1000,4000
>>
>> total 8 rows;
>>
>> then I execute sql   "select consumerId from fool".
>>
>> When the job was executing, I watch one  map  log. through
>> "http://ip:50030/"
>>
>> syslog logs
>>
>> 2011-12-21 17:57:57,013 WARN org.apache.hadoop.conf.Configuration:
>> /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/job.xml:a
>> attempt to override final parameter: mapred.hosts.exclude;  Ignoring
>>  .
>> 2011-12-21 17:57:57,074 WARN org.apache.hadoop.util.NativeCodeLoader: Unable
>> to load native-hadoop library for your platform... using builtin-java
>> classes where applicable
>> 2011-12-21 17:57:57,208 INFO org.apache.hadoop.mapred.TaskRunner: Creating
>> symlink:
>> /analyser/hdfs/mapred/local/taskTracker/distcache/-5775579569808818942_-277615576_1611537495/h-master/tmp/hive-hadoop/hive_2011-12-21_17-57-43_970_69237304724972937/-mr-10003/9bc47070-2f7c-4ad0-8a16-d5687fad2a11
>> <-
>> /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/attempt_201112161932_0283_m_000000_0/work/HIVE_PLAN9bc47070-2f7c-4ad0-8a16-d5687fad2a11
>> 2011-12-21 17:57:57,214 INFO
>> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
>> symlink:
>> /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/jars/.job.jar.crc
>> <-
>> /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/attempt_201112161932_0283_m_000000_0/work/.job.jar.crc
>> 2011-12-21 17:57:57,215 INFO
>> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
>> symlink:
>> /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/jars/job.jar
>> <-
>> /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/attempt_201112161932_0283_m_000000_0/work/job.jar
>> 2011-12-21 17:57:57,291 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Initializing JVM Metrics with processName=MAP, sessionId=
>> 2011-12-21 17:57:57,368 WARN org.apache.hadoop.conf.Configuration:
>> /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/job.xml:a
>> attempt to override final parameter: mapred.hosts.exclude;  Ignoring.
>> 2011-12-21 17:57:57,370 WARN org.apache.hadoop.conf.Configuration:
>> /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/job.xml:a
>> attempt to override final parameter: dfs.hosts.exclude;  Ignoring.
>> 2011-12-21 17:57:57,374 WARN org.apache.hadoop.conf.Configuration:
>> /analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/job.xml:a
>> attempt to override final parameter: dfs.name.dir;  Ignoring.
>> 2011-12-21 17:57:57,614 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader:
>> Loaded native gpl library
>> 2011-12-21 17:57:57,617 INFO com.hadoop.compression.lzo.LzoCodec:
>> Successfully loaded & initialized native-lzo library [hadoop-lzo rev
>> 4537f94556c9ba71ffca316514e0c0101f76b63b]
>> 2011-12-21 17:57:57,632 INFO org.apache.hadoop.mapred.MapTask:
>> numReduceTasks: 0
>> 2011-12-21 17:57:57,639 INFO ExecMapper: maximum memory = 514523136
>> 2011-12-21 17:57:57,640 INFO ExecMapper: conf classpath =
>> [file:/usr/home/hadoop/hadoop-0.20.2/conf/,
>> file:/usr/java/jdk1.6.0_23/lib/tools.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/,
>> file:/usr/home/hadoop/hadoop-0.20.2/hadoop-core-0.20.2-CDH3B4.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/aspectjrt-1.6.5.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/aspectjtools-1.6.5.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-cli-1.2.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-codec-1.4.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-daemon-1.0.1.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-el-1.0.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-httpclient-3.0.1.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-logging-1.0.4.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-logging-api-1.0.4.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-net-1.4.1.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/core-3.1.1.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib
>>  /hadoop-fairscheduler-0.20.2-CDH3B4.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/hadoop-lzo-0.4.12.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/hsqldb-1.8.0.10.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jackson-core-asl-1.5.2.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jackson-mapper-asl-1.5.2.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jasper-compiler-5.5.12.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jasper-runtime-5.5.12.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jets3t-0.6.1.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jetty-6.1.26.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jetty-servlet-tester-6.1.26.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jetty-util-6.1.26.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jsch-0.1.42.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/junit-4.5.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/kfs-0.2.2.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/log4j-1.2.15.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/mockito-all-1.8.2.jar, file:/usr/h
>>  ome/hadoop/hadoop-0.20.2/lib/oro-2.0.8.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-20081211.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-6.1.14.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/slf4j-api-1.4.3.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/slf4j-log4j12-1.4.3.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/xmlenc-0.52.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jsp-2.1/jsp-2.1.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jsp-2.1/jsp-api-2.1.jar,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/jars/classes,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/jars/job.jar,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/distcache/4011691731310986177_1050235208_1611538248/h-master/analyser/hdfs/mapred/staging/hadoop/.staging/job_201112161932_0283/libjars/logfunc.jar,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/distcache/5733595343670413510_2108
>>  507888_486173279/h-master/user/hive/hbase-0.90.3-cdh3u1.jar,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/distcache/5214408008425213222_-774945002_486208112/h-master/user/hive/hive-hbase-handler-0.7.1-cdh3u1.jar,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/distcache/9089248915585406031_-644048472_1609575637/h-master/usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u1.jar,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/attempt_201112161932_0283_m_000000_0/work/]
>> 2011-12-21 17:57:57,640 INFO ExecMapper: thread classpath =
>> [file:/usr/home/hadoop/hadoop-0.20.2/conf/,
>> file:/usr/java/jdk1.6.0_23/lib/tools.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/,
>> file:/usr/home/hadoop/hadoop-0.20.2/hadoop-core-0.20.2-CDH3B4.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/aspectjrt-1.6.5.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/aspectjtools-1.6.5.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-cli-1.2.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-codec-1.4.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-daemon-1.0.1.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-el-1.0.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-httpclient-3.0.1.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-logging-1.0.4.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-logging-api-1.0.4.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/commons-net-1.4.1.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/core-3.1.1.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/l
>>  ib/hadoop-fairscheduler-0.20.2-CDH3B4.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/hadoop-lzo-0.4.12.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/hsqldb-1.8.0.10.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jackson-core-asl-1.5.2.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jackson-mapper-asl-1.5.2.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jasper-compiler-5.5.12.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jasper-runtime-5.5.12.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jets3t-0.6.1.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jetty-6.1.26.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jetty-servlet-tester-6.1.26.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jetty-util-6.1.26.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jsch-0.1.42.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/junit-4.5.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/kfs-0.2.2.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/log4j-1.2.15.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/mockito-all-1.8.2.jar, file:/usr
>>  /home/hadoop/hadoop-0.20.2/lib/oro-2.0.8.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-20081211.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/servlet-api-2.5-6.1.14.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/slf4j-api-1.4.3.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/slf4j-log4j12-1.4.3.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/xmlenc-0.52.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jsp-2.1/jsp-2.1.jar,
>> file:/usr/home/hadoop/hadoop-0.20.2/lib/jsp-2.1/jsp-api-2.1.jar,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/jars/classes,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/jars/job.jar,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/distcache/4011691731310986177_1050235208_1611538248/h-master/analyser/hdfs/mapred/staging/hadoop/.staging/job_201112161932_0283/libjars/logfunc.jar,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/distcache/5733595343670413510_21
>>  08507888_486173279/h-master/user/hive/hbase-0.90.3-cdh3u1.jar,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/distcache/5214408008425213222_-774945002_486208112/h-master/user/hive/hive-hbase-handler-0.7.1-cdh3u1.jar,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/distcache/9089248915585406031_-644048472_1609575637/h-master/usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u1.jar,
>> file:/data1/analyser/hdfs/mapred/local/taskTracker/hadoop/jobcache/job_201112161932_0283/attempt_201112161932_0283_m_000000_0/work/]
>> 2011-12-21 17:57:57,690 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
>> Adding alias testudaf to work list for file
>> hdfs://h-master:9000/user/hadoop/wftest
>> 2011-12-21 17:57:57,693 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
>> dump TS struct<seller:string,cosumer:string>
>> 2011-12-21 17:57:57,694 INFO ExecMapper:
>> <MAP>Id =3
>>   <Children>
>>     <TS>Id =0
>>       <Children>
>>         <SEL>Id =1
>>           <Children>
>>             <FS>Id =2
>>               <Parent>Id = 1 null<\Parent>
>>             <\FS>
>>           <\Children>
>>           <Parent>Id = 0 null<\Parent>
>>         <\SEL>
>>       <\Children>
>>       <Parent>Id = 3 null<\Parent>
>>     <\TS>
>>   <\Children>
>> <\MAP>
>> 2011-12-21 17:57:57,694 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
>> Initializing Self 3 MAP
>> 2011-12-21 17:57:57,694 INFO
>> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing Self 0 TS
>> 2011-12-21 17:57:57,694 INFO
>> org.apache.hadoop.hive.ql.exec.TableScanOperator: Operator 0 TS initialized
>> 2011-12-21 17:57:57,694 INFO
>> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing children of 0
>> TS
>> 2011-12-21 17:57:57,694 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
>> Initializing child 1 SEL
>> 2011-12-21 17:57:57,694 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
>> Initializing Self 1 SEL
>> 2011-12-21 17:57:57,699 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
>> SELECT struct<seller:string,cosumer:string>
>> 2011-12-21 17:57:57,703 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
>> Operator 1 SEL initialized
>> 2011-12-21 17:57:57,703 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
>> Initializing children of 1 SEL
>> 2011-12-21 17:57:57,703 INFO
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initializing child 2 FS
>> 2011-12-21 17:57:57,703 INFO
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initializing Self 2 FS
>> 2011-12-21 17:57:57,707 INFO
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator: Operator 2 FS initialized
>> 2011-12-21 17:57:57,707 INFO
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator: Initialization Done 2 FS
>> 2011-12-21 17:57:57,707 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
>> Initialization Done 1 SEL
>> 2011-12-21 17:57:57,707 INFO
>> org.apache.hadoop.hive.ql.exec.TableScanOperator: Initialization Done 0 TS
>> 2011-12-21 17:57:57,707 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
>> Initialization Done 3 MAP
>> 2011-12-21 17:57:57,711 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
>> Processing path hdfs://h-master:9000/user/hadoop/wftest/test.txt
>> 2011-12-21 17:57:57,711 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
>> Processing alias testudaf for file hdfs://h-master:9000/user/hadoop/wftest
>> 2011-12-21 17:57:57,712 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3
>> forwarding 1 rows
>> 2011-12-21 17:57:57,712 INFO
>> org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarding 1 rows
>> 2011-12-21 17:57:57,712 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
>> 1 forwarding 1 rows
>> 2011-12-21 17:57:57,713 INFO
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS
>> hdfs://h-master:9000/tmp/hive-hadoop/hive_2011-12-21_17-57-43_970_69237304724972937/_tmp.-ext-10001/000000_0
>> 2011-12-21 17:57:57,713 INFO
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS
>> hdfs://h-master:9000/tmp/hive-hadoop/hive_2011-12-21_17-57-43_970_69237304724972937/_tmp.-ext-10001/_tmp.000000_0
>> 2011-12-21 17:57:57,714 INFO
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS
>> hdfs://h-master:9000/tmp/hive-hadoop/hive_2011-12-21_17-57-43_970_69237304724972937/_tmp.-ext-10001/000000_0
>> 2011-12-21 17:57:57,785 INFO ExecMapper: ExecMapper: processing 1 rows: used
>> memory = 59120320
>> 2011-12-21 17:57:57,786 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3
>> finished. closing...
>> 2011-12-21 17:57:57,786 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3
>> forwarded 4 rows
>> 2011-12-21 17:57:57,786 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
>> DESERIALIZE_ERRORS:0
>> 2011-12-21 17:57:57,786 INFO
>> org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 finished. closing...
>> 2011-12-21 17:57:57,786 INFO
>> org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarded 4 rows
>> 2011-12-21 17:57:57,786 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
>> 1 finished. closing...
>> 2011-12-21 17:57:57,786 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
>> 1 forwarded 4 rows
>> 2011-12-21 17:57:57,786 INFO
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator: 2 finished. closing...
>> 2011-12-21 17:57:57,786 INFO
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator: 2 forwarded 0 rows
>> 2011-12-21 17:57:57,950 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
>> 1 Close done
>> 2011-12-21 17:57:57,950 INFO
>> org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 Close done
>> 2011-12-21 17:57:57,950 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3
>> Close done
>> 2011-12-21 17:57:57,950 INFO ExecMapper: ExecMapper: processed 4 rows: used
>> memory = 64489208
>> 2011-12-21 17:57:57,953 INFO org.apache.hadoop.mapred.Task:
>> Task:attempt_201112161932_0283_m_000000_0 is done. And is in the process of
>> commiting
>> 2011-12-21 17:57:57,960 INFO org.apache.hadoop.mapred.Task: Task
>> 'attempt_201112161932_0283_m_000000_0' done.
>> 2011-12-21 17:57:57,991 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
>> Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
>>
>>
>>
>> pls see the red rows.  why map used so large memory, which parameter
>>  related  ?
>>
>> Thanks.
>>
>>
>>
>>
>>
Mime
View raw message