hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-5235) Infinite loop with ORC file and Hive 0.11
Date Wed, 25 Sep 2013 07:12:04 GMT

    [ https://issues.apache.org/jira/browse/HIVE-5235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13777215#comment-13777215
] 

Thejas M Nair commented on HIVE-5235:
-------------------------------------

Pere, We would like to get this fixed for 0.12 release if possible. Can you please give any
additional information you have for Owen ?

                
> Infinite loop with ORC file and Hive 0.11
> -----------------------------------------
>
>                 Key: HIVE-5235
>                 URL: https://issues.apache.org/jira/browse/HIVE-5235
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.11.0
>         Environment: Gentoo linux with Hortonworks Hadoop hadoop-1.1.2.23.tar.gz and
Apache Hive 0.11d
>            Reporter: Iván de Prado
>            Priority: Blocker
>
> We are using Hive 0.11 with ORC file format and we get some tasks blocked in some kind
of infinite loop. They keep working indefinitely when we set a huge task expiry timeout. If
we the expiry time to 600 second, the taks fail because of not reporting progress, and finally,
the Job fails. 
> That is not consistent, and some times between jobs executions the behavior changes.
It happen for different queries.
> We are using Hive 0.11 with Hadoop hadoop-1.1.2.23 from Hortonworks. The taks that is
blocked keeps consuming 100% of CPU usage, and the stack trace is always the same consistently.
Everything points to some kind of infinite loop. My guessing is that it has some relation
to the ORC file. Maybe some pointer is not right when writing generating some kind of infinite
loop when reading.  Or maybe there is a bug in the reading stage.
> More information below. The stack trace:
> {noformat} 
> "main" prio=10 tid=0x00007f20a000a800 nid=0x1ed2 runnable [0x00007f20a8136000]
>    java.lang.Thread.State: RUNNABLE
> 	at java.util.zip.Inflater.inflateBytes(Native Method)
> 	at java.util.zip.Inflater.inflate(Inflater.java:256)
> 	- locked <0x00000000f42a6ca0> (a java.util.zip.ZStreamRef)
> 	at org.apache.hadoop.hive.ql.io.orc.ZlibCodec.decompress(ZlibCodec.java:64)
> 	at org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.readHeader(InStream.java:128)
> 	at org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.read(InStream.java:143)
> 	at org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readVulong(SerializationUtils.java:54)
> 	at org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readVslong(SerializationUtils.java:65)
> 	at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReader.readValues(RunLengthIntegerReader.java:66)
> 	at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReader.next(RunLengthIntegerReader.java:81)
> 	at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$IntTreeReader.next(RecordReaderImpl.java:332)
> 	at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:802)
> 	at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1214)
> 	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:71)
> 	at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:46)
> 	at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:274)
> 	at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:101)
> 	at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)
> 	at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:108)
> 	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:300)
> 	at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:218)
> 	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
> 	- eliminated <0x00000000e1459700> (a org.apache.hadoop.mapred.MapTask$TrackedRecordReader)
> 	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
> 	- locked <0x00000000e1459700> (a org.apache.hadoop.mapred.MapTask$TrackedRecordReader)
> 	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat} 
> We have seen the same stack trace repeatedly for several executions of jstack.
> The log file for this kind of task is the following: 
> {noformat} 
> 2013-09-04 23:12:34,332 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop
library
> 2013-09-04 23:12:34,681 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /hd/hd6/mapred/local/taskTracker/distcache/1758137359311570022_-1138434677_1812098585/master/tmp/hive-datasalt/hive_2013-09-04_22-45-13_829_2202639092470021957/-mr-10004/22027cf8-f583-41d7-adb8-e7e74922d113
<- /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/HIVE_PLAN22027cf8-f583-41d7-adb8-e7e74922d113
> 2013-09-04 23:12:34,718 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager:
Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/com
<- /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/com
> 2013-09-04 23:12:34,733 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager:
Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/javolution
<- /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/javolution
> 2013-09-04 23:12:34,743 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager:
Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/org
<- /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/org
> 2013-09-04 23:12:34,756 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager:
Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/job.jar
<- /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/job.jar
> 2013-09-04 23:12:34,768 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager:
Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/javax
<- /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/javax
> 2013-09-04 23:12:34,779 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager:
Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/hive-exec-log4j.properties
<- /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/hive-exec-log4j.properties
> 2013-09-04 23:12:34,790 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager:
Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/META-INF
<- /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/META-INF
> 2013-09-04 23:12:34,796 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager:
Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/javaewah
<- /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/javaewah
> 2013-09-04 23:12:34,813 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager:
Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/.job.jar.crc
<- /hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/.job.jar.crc
> 2013-09-04 23:12:35,127 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source
name ugi already exists!
> 2013-09-04 23:12:35,412 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit
code 0
> 2013-09-04 23:12:35,473 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin
: org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5d0fa836
> 2013-09-04 23:12:36,927 INFO org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader:
Processing file hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-28/000798_0
> 2013-09-04 23:12:36,927 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 72
> 2013-09-04 23:12:36,936 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
> 2013-09-04 23:12:37,028 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720
> 2013-09-04 23:12:37,028 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
> 2013-09-04 23:12:37,036 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy
native library is available
> 2013-09-04 23:12:37,036 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy
native library loaded
> 2013-09-04 23:12:37,046 INFO ExecMapper: maximum memory = 559284224
> 2013-09-04 23:12:37,046 INFO ExecMapper: conf classpath = [file:/home/datasalt/sw/hadoop-1.1.2.23/conf/,
file:/opt/oracle-jdk-bin-1.7.0.17/lib/tools.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/,
file:/home/datasalt/sw/hadoop-1.1.2.23/hadoop-core-1.1.2.23.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/asm-3.2.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjrt-1.6.11.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjtools-1.6.11.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-1.7.0.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-core-1.8.0.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-cli-1.2.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-codec-1.4.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-collections-3.2.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-configuration-1.6.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-daemon-1.0.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-digester-1.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-el-1.0.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-httpclient-3.0.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-io-2.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-lang-2.4.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-1.1.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-api-1.0.4.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-math-2.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-net-3.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/core-3.1.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-capacity-scheduler-1.1.2.23.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-fairscheduler-1.1.2.23.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-thriftfs-1.1.2.23.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hsqldb-1.8.0.10.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-core-asl-1.8.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-mapper-asl-1.8.8.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-compiler-5.5.12.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-runtime-5.5.12.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jdeb-0.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-core-1.8.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-json-1.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-server-1.8.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jets3t-0.6.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-6.1.26.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-util-6.1.26.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsch-0.1.42.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/junit-4.5.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/kfs-0.2.2.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/log4j-1.2.15.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/mockito-all-1.8.5.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/oro-2.0.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/servlet-api-2.5-20081211.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-api-1.4.3.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-log4j12-1.4.3.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/xmlenc-0.52.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-2.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-api-2.1.jar,
file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/classes,
file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/, file:/hd/hd1/mapred/local/taskTracker/datasalt/distcache/-1837517190973317952_-652944854_1812099894/master/tmp/hadoop-datasalt/mapred/staging/datasalt/.staging/job_201309040511_0003/libjars/derby-10.4.2.0.jar/,
file:/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/]
> 2013-09-04 23:12:37,047 INFO ExecMapper: thread classpath = [file:/home/datasalt/sw/hadoop-1.1.2.23/conf/,
file:/opt/oracle-jdk-bin-1.7.0.17/lib/tools.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/,
file:/home/datasalt/sw/hadoop-1.1.2.23/hadoop-core-1.1.2.23.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/asm-3.2.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjrt-1.6.11.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjtools-1.6.11.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-1.7.0.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-core-1.8.0.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-cli-1.2.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-codec-1.4.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-collections-3.2.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-configuration-1.6.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-daemon-1.0.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-digester-1.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-el-1.0.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-httpclient-3.0.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-io-2.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-lang-2.4.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-1.1.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-api-1.0.4.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-math-2.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-net-3.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/core-3.1.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-capacity-scheduler-1.1.2.23.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-fairscheduler-1.1.2.23.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-thriftfs-1.1.2.23.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hsqldb-1.8.0.10.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-core-asl-1.8.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-mapper-asl-1.8.8.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-compiler-5.5.12.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-runtime-5.5.12.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jdeb-0.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-core-1.8.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-json-1.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-server-1.8.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jets3t-0.6.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-6.1.26.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-util-6.1.26.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsch-0.1.42.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/junit-4.5.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/kfs-0.2.2.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/log4j-1.2.15.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/mockito-all-1.8.5.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/oro-2.0.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/servlet-api-2.5-20081211.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-api-1.4.3.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-log4j12-1.4.3.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/xmlenc-0.52.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-2.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-api-2.1.jar,
file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/classes,
file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/, file:/hd/hd1/mapred/local/taskTracker/datasalt/distcache/-1837517190973317952_-652944854_1812099894/master/tmp/hadoop-datasalt/mapred/staging/datasalt/.staging/job_201309040511_0003/libjars/derby-10.4.2.0.jar/,
file:/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/]
> 2013-09-04 23:12:37,085 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding alias
hits to work list for file hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-28
> 2013-09-04 23:12:37,089 INFO org.apache.hadoop.hive.ql.exec.MapOperator: dump TS struct<hour:int,user_ip:string,country_iso:string,user_language:string,num_hits:int,site_hostname:string,idsite:int,num_ads:int,num_ads_wanted:int,category:int,idcampaign:int,idvariation:int,sub_id:int,cpm_value:double,publisher_share:int,publisher_currency:string,advertiser_currency:string,idzone:int,idagent:int,idadvertiser:int,idpublisher:int,ddate:string>
> 2013-09-04 23:12:37,091 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding alias
hits to work list for file hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-29
> 2013-09-04 23:12:37,092 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding alias
hits to work list for file hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-30
> 2013-09-04 23:12:37,093 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding alias
hits to work list for file hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-31
> 2013-09-04 23:12:37,094 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding alias
hits to work list for file hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-09-01
> 2013-09-04 23:12:37,094 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding alias
hits to work list for file hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-09-02
> 2013-09-04 23:12:37,095 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Adding alias
hits to work list for file hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-09-03
> 2013-09-04 23:12:37,096 INFO ExecMapper: 
> <MAP>Id =7
>   <Children>
>     <TS>Id =3
>       <Children>
>         <SEL>Id =2
>           <Children>
>             <GBY>Id =1
>               <Children>
>                 <RS>Id =0
>                   <Parent>Id = 1 null<\Parent>
>                 <\RS>
>               <\Children>
>               <Parent>Id = 2 null<\Parent>
>             <\GBY>
>           <\Children>
>           <Parent>Id = 3 null<\Parent>
>         <\SEL>
>       <\Children>
>       <Parent>Id = 7 null<\Parent>
>     <\TS>
>   <\Children>
> <\MAP>
> 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Initializing
Self 7 MAP
> 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing
Self 3 TS
> 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: Operator
3 TS initialized
> 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing
children of 3 TS
> 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing
child 2 SEL
> 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing
Self 2 SEL
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: SELECT struct<hour:int,user_ip:string,country_iso:string,user_language:string,num_hits:int,site_hostname:string,idsite:int,num_ads:int,num_ads_wanted:int,category:int,idcampaign:int,idvariation:int,sub_id:int,cpm_value:double,publisher_share:int,publisher_currency:string,advertiser_currency:string,idzone:int,idagent:int,idadvertiser:int,idpublisher:int,ddate:string>
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Operator
2 SEL initialized
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing
children of 2 SEL
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator: Initializing
child 1 GBY
> 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator: Initializing
Self 1 GBY
> 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator: Operator
1 GBY initialized
> 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator: Initializing
children of 1 GBY
> 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initializing
child 0 RS
> 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initializing
Self 0 RS
> 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Using
tag = -1
> 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Operator
0 RS initialized
> 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initialization
Done 0 RS
> 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator: Initialization
Done 1 GBY
> 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: Initialization
Done 2 SEL
> 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: Initialization
Done 3 TS
> 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Initialization
Done 7 MAP
> 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.MapOperator: Processing alias
hits for file hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-28
> 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7 forwarding
1 rows
> 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding
1 rows
> 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2 forwarding
1 rows
> 2013-09-04 23:12:37,168 INFO ExecMapper: ExecMapper: processing 1 rows: used memory =
151151936
> 2013-09-04 23:12:37,169 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7 forwarding
10 rows
> 2013-09-04 23:12:37,169 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding
10 rows
> 2013-09-04 23:12:37,169 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2 forwarding
10 rows
> 2013-09-04 23:12:37,169 INFO ExecMapper: ExecMapper: processing 10 rows: used memory
= 151151936
> 2013-09-04 23:12:37,175 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7 forwarding
100 rows
> 2013-09-04 23:12:37,175 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding
100 rows
> 2013-09-04 23:12:37,175 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2 forwarding
100 rows
> 2013-09-04 23:12:37,175 INFO ExecMapper: ExecMapper: processing 100 rows: used memory
= 151151936
> 2013-09-04 23:12:37,247 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7 forwarding
1000 rows
> 2013-09-04 23:12:37,250 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding
1000 rows
> 2013-09-04 23:12:37,250 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2 forwarding
1000 rows
> 2013-09-04 23:12:37,251 INFO ExecMapper: ExecMapper: processing 1000 rows: used memory
= 151151936
> 2013-09-04 23:12:37,642 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7 forwarding
10000 rows
> 2013-09-04 23:12:37,642 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding
10000 rows
> 2013-09-04 23:12:37,644 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2 forwarding
10000 rows
> 2013-09-04 23:12:37,644 INFO ExecMapper: ExecMapper: processing 10000 rows: used memory
= 153032176
> 2013-09-04 23:12:38,467 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7 forwarding
100000 rows
> 2013-09-04 23:12:38,468 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding
100000 rows
> 2013-09-04 23:12:38,468 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2 forwarding
100000 rows
> 2013-09-04 23:12:38,469 INFO ExecMapper: ExecMapper: processing 100000 rows: used memory
= 125346272
> 2013-09-04 23:12:40,626 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7 forwarding
1000000 rows
> 2013-09-04 23:12:40,627 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding
1000000 rows
> 2013-09-04 23:12:40,627 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 2 forwarding
1000000 rows
> 2013-09-04 23:12:40,627 INFO ExecMapper: ExecMapper: processing 1000000 rows: used memory
= 123383952
> {noformat} 
> For this particular example, the task was killed 6 hours after the last log line. 
> The strace of the process when it is in the infinite loop has the following looking:
> {noformat} 
> Process 7887 attached with 24 threads
> [pid  7930] futex(0x7f20a06df28c, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
> [pid  7922] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
> [pid  7920] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
> [pid  8333] accept(61,  <unfinished ...>
> [pid  7908] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
> [pid  7907] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
> [pid  7906] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
> [pid  7905] futex(0x7f20a00fa554, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished ...>
> [pid  7903] futex(0x7f20a00f5254, FUTEX_WAIT_PRIVATE, 417, NULL <unfinished ...>
> [pid  7904] futex(0x7f20a00f7c54, FUTEX_WAIT_PRIVATE, 387, NULL <unfinished ...>
> [pid  7902] futex(0x7f20a7513dc0, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
> [pid  7901] futex(0x7f20a00a6f54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished ...>
> [pid  7899] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
> [pid  7898] futex(0x7f20a0026554, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished ...>
> [pid  7900] futex(0x7f20a00a4b54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished ...>
> [pid  7897] futex(0x7f20a0024754, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished ...>
> [pid  7896] futex(0x7f20a0022a54, FUTEX_WAIT_PRIVATE, 41, NULL <unfinished ...>
> [pid  7895] futex(0x7f20a0020c54, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished ...>
> [pid  7894] futex(0x7f20a001ef54, FUTEX_WAIT_PRIVATE, 37, NULL <unfinished ...>
> [pid  7893] futex(0x7f20a001d154, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished ...>
> [pid  7887] futex(0x7f20a81389d0, FUTEX_WAIT, 7890, NULL <unfinished ...>
> [pid  7892] futex(0x7f20a001b454, FUTEX_WAIT_PRIVATE, 35, NULL <unfinished ...>
> [pid  7891] futex(0x7f20a0019654, FUTEX_WAIT_PRIVATE, 37, NULL <unfinished ...>
> [pid  7906] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373836, 993325000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 43598000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 93858000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 144101000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 194344000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 244659000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 294924000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 345198000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 395462000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 445768000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 495983000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 546226000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 596489000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 646798000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 697063000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 747370000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 797688000}, ffffffff <unfinished ...>
> [pid  7899] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed out)
> [pid  7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7899] futex(0x7f20a009d054, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 782077000}, ffffffff <unfinished ...>
> [pid  7906] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 848014000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 898283000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 948489000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373837, 998751000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 49086000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 99351000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 149589000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 199852000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 250180000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 300489000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 350762000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 401078000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 451258000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 501527000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 551767000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 601970000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 652161000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 702290000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 752523000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 802739000}, ffffffff <unfinished ...>
> [pid  7899] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
> [pid  7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7899] futex(0x7f20a009d054, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 782336000}, ffffffff <unfinished ...>
> [pid  7922] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed out)
> [pid  7922] futex(0x7f20a06d3a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7922] futex(0x7f20a0550554, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f20a0550550, {FUTEX_OP_SET,
0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid  7907] <... restart_syscall resumed> ) = 0
> [pid  7907] futex(0x7f20a0550528, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7922] write(56, "\0\0\0\335\0\0\2.\0\4ping\0\0\0\2\0&org.apache.h"..., 225
<unfinished ...>
> [pid  7907] read(56,  <unfinished ...>
> [pid  7922] <... write resumed> )       = 225
> [pid  7907] <... read resumed> 0x7f2098000b70, 8192) = -1 EAGAIN (Resource temporarily
unavailable)
> [pid  7922] futex(0x7f20a009d054, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f20a009d050, {FUTEX_OP_SET,
0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid  7899] <... futex resumed> )       = 0
> [pid  7922] futex(0x7f20a06d3e54, FUTEX_WAIT_PRIVATE, 53, NULL <unfinished ...>
> [pid  7907] epoll_ctl(59, EPOLL_CTL_ADD, 56, {...} <unfinished ...>
> [pid  7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid  7907] <... epoll_ctl resumed> )   = 0
> [pid  7899] <... futex resumed> )       = 0
> [pid  7907] epoll_wait(59, {?} 0x7f20a0544000, 4096, 60000) = 1
> [pid  7899] mprotect(0x7f20a8147000, 4096, PROT_READ <unfinished ...>
> [pid  7907] futex(0x7f20a0551e54, FUTEX_WAIT_PRIVATE, 11, NULL <unfinished ...>
> [pid  7899] <... mprotect resumed> )    = 0
> [pid  7890] futex(0x7f20a000b854, FUTEX_WAIT_PRIVATE, 349, NULL <unfinished ...>
> [pid  7899] mprotect(0x7f20a8147000, 4096, PROT_READ|PROT_WRITE) = 0
> [pid  7899] mprotect(0x7f20a8148000, 4096, PROT_NONE) = 0
> [pid  7899] mprotect(0x7f20a8148000, 4096, PROT_READ) = 0
> [pid  7899] futex(0x7f20a0551e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f20a0551e50, {FUTEX_OP_SET,
0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid  7899] futex(0x7f20a06d3e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f20a06d3e50, {FUTEX_OP_SET,
0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
> [pid  7922] <... futex resumed> )       = 0
> [pid  7907] <... futex resumed> )       = 0
> [pid  7922] futex(0x7f20a06d3e28, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
> [pid  7899] <... futex resumed> )       = 1
> [pid  7907] futex(0x7f20a0551e28, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid  7899] futex(0x7f20a06d3e28, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid  7922] <... futex resumed> )       = 0
> [pid  7907] <... futex resumed> )       = 0
> [pid  7922] futex(0x7f20a06d3e28, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid  7899] <... futex resumed> )       = 1
> [pid  7922] <... futex resumed> )       = 0
> [pid  7899] futex(0x7f20a009d054, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3,
{1378373839, 794194000}, ffffffff <unfinished ...>
> [pid  7922] futex(0x7f20a06d3a54, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
> [pid  7907] futex(0x7f20a000b854, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f20a000b850, {FUTEX_OP_SET,
0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid  7890] <... futex resumed> )       = 0
> [pid  7890] futex(0x7f20a000b828, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid  7907] epoll_ctl(59, EPOLL_CTL_DEL, 56, {...} <unfinished ...>
> [pid  7890] <... futex resumed> )       = 0
> [pid  7907] <... epoll_ctl resumed> )   = 0
> [pid  7907] epoll_wait(59, {}, 4096, 0) = 0
> [pid  7907] read(56, "\0\0\2.\0\0\0\0\0\7boolean\1", 8192) = 18
> [pid  7907] futex(0x7f20a06d3a54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f20a06d3a50, {FUTEX_OP_SET,
0, FUTEX_OP_CMP_GT, 1}) = 1
> [pid  7922] <... futex resumed> )       = 0
> [pid  7907] futex(0x7f20a0550554, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1107,
{1378373848, 793674000}, ffffffff <unfinished ...>
> [pid  7922] futex(0x7f20a06d3a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7922] futex(0x7f20a06d3a54, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3,
{1378373841, 794758000}, ffffffff <unfinished ...>
> [pid  7906] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 852867000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 903052000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373838, 953226000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 3436000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 53695000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 103966000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 154232000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 204466000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 254802000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 305067000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 355318000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 405623000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 455817000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 506130000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 556395000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 606661000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 656898000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> [pid  7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid  7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1,
{1378373839, 707177000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
> {noformat} 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message