Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 10F6210435 for ; Mon, 14 Oct 2013 08:47:40 +0000 (UTC) Received: (qmail 70757 invoked by uid 500); 14 Oct 2013 08:46:40 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 69604 invoked by uid 500); 14 Oct 2013 08:44:32 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 69463 invoked by uid 500); 14 Oct 2013 08:43:58 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 69384 invoked by uid 99); 14 Oct 2013 08:43:43 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Oct 2013 08:43:43 +0000 Date: Mon, 14 Oct 2013 08:43:42 +0000 (UTC) From: "Pere Ferrera Bertran (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-5235) Infinite loop with ORC file and Hive 0.11 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-5235?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D13793= 990#comment-13793990 ]=20 Pere Ferrera Bertran commented on HIVE-5235: -------------------------------------------- Hello, I used the script to generate a big file (60 GB) and ran 5 iterations of CR= EATE TABLE + SELECT without compression. One of them failed randomly with t= he same infinite loop symptoms (tasks died due to not reporting timeout). > Infinite loop with ORC file and Hive 0.11 > ----------------------------------------- > > Key: HIVE-5235 > URL: https://issues.apache.org/jira/browse/HIVE-5235 > Project: Hive > Issue Type: Bug > Affects Versions: 0.11.0 > Environment: Gentoo linux with Hortonworks Hadoop hadoop-1.1.2.23= .tar.gz and Apache Hive 0.11d > Reporter: Iv=C3=A1n de Prado > Priority: Blocker > Attachments: gendata.py > > > We are using Hive 0.11 with ORC file format and we get some tasks blocked= in some kind of infinite loop. They keep working indefinitely when we set = a huge task expiry timeout. If we the expiry time to 600 second, the taks f= ail because of not reporting progress, and finally, the Job fails.=20 > That is not consistent, and some times between jobs executions the behavi= or changes. It happen for different queries. > We are using Hive 0.11 with Hadoop hadoop-1.1.2.23 from Hortonworks. The = taks that is blocked keeps consuming 100% of CPU usage, and the stack trace= is always the same consistently. Everything points to some kind of infinit= e loop. My guessing is that it has some relation to the ORC file. Maybe som= e pointer is not right when writing generating some kind of infinite loop w= hen reading. Or maybe there is a bug in the reading stage. > More information below. The stack trace: > {noformat}=20 > "main" prio=3D10 tid=3D0x00007f20a000a800 nid=3D0x1ed2 runnable [0x00007f= 20a8136000] > java.lang.Thread.State: RUNNABLE > =09at java.util.zip.Inflater.inflateBytes(Native Method) > =09at java.util.zip.Inflater.inflate(Inflater.java:256) > =09- locked <0x00000000f42a6ca0> (a java.util.zip.ZStreamRef) > =09at org.apache.hadoop.hive.ql.io.orc.ZlibCodec.decompress(ZlibCodec.jav= a:64) > =09at org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.readHead= er(InStream.java:128) > =09at org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.read(InS= tream.java:143) > =09at org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readVulong(Seri= alizationUtils.java:54) > =09at org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readVslong(Seri= alizationUtils.java:65) > =09at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReader.readValues(= RunLengthIntegerReader.java:66) > =09at org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReader.next(RunLen= gthIntegerReader.java:81) > =09at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$IntTreeReader.nex= t(RecordReaderImpl.java:332) > =09at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.= next(RecordReaderImpl.java:802) > =09at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReader= Impl.java:1214) > =09at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.nex= t(OrcInputFormat.java:71) > =09at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.nex= t(OrcInputFormat.java:46) > =09at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(Hi= veContextAwareRecordReader.java:274) > =09at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(Combine= HiveRecordReader.java:101) > =09at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(Combine= HiveRecordReader.java:41) > =09at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(Hive= ContextAwareRecordReader.java:108) > =09at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordRea= der.doNextWithExceptionHandler(HadoopShimsSecure.java:300) > =09at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordRea= der.next(HadoopShimsSecure.java:218) > =09at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(Map= Task.java:236) > =09- eliminated <0x00000000e1459700> (a org.apache.hadoop.mapred.MapTask$= TrackedRecordReader) > =09at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.j= ava:216) > =09- locked <0x00000000e1459700> (a org.apache.hadoop.mapred.MapTask$Trac= kedRecordReader) > =09at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) > =09at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) > =09at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > =09at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > =09at java.security.AccessController.doPrivileged(Native Method) > =09at javax.security.auth.Subject.doAs(Subject.java:415) > =09at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInfor= mation.java:1178) > =09at org.apache.hadoop.mapred.Child.main(Child.java:249) > {noformat}=20 > We have seen the same stack trace repeatedly for several executions of js= tack. > The log file for this kind of task is the following:=20 > {noformat}=20 > 2013-09-04 23:12:34,332 INFO org.apache.hadoop.util.NativeCodeLoader: Loa= ded the native-hadoop library > 2013-09-04 23:12:34,681 INFO org.apache.hadoop.mapred.TaskRunner: Creatin= g symlink: /hd/hd6/mapred/local/taskTracker/distcache/1758137359311570022_-= 1138434677_1812098585/master/tmp/hive-datasalt/hive_2013-09-04_22-45-13_829= _2202639092470021957/-mr-10004/22027cf8-f583-41d7-adb8-e7e74922d113 <- /hd/= hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attemp= t_201309040511_0003_m_000803_1/work/HIVE_PLAN22027cf8-f583-41d7-adb8-e7e749= 22d113 > 2013-09-04 23:12:34,718 INFO org.apache.hadoop.filecache.TrackerDistribut= edCacheManager: Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt= /jobcache/job_201309040511_0003/jars/com <- /hd/hd1/mapred/local/taskTracke= r/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_00080= 3_1/work/com > 2013-09-04 23:12:34,733 INFO org.apache.hadoop.filecache.TrackerDistribut= edCacheManager: Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt= /jobcache/job_201309040511_0003/jars/javolution <- /hd/hd1/mapred/local/tas= kTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_= m_000803_1/work/javolution > 2013-09-04 23:12:34,743 INFO org.apache.hadoop.filecache.TrackerDistribut= edCacheManager: Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt= /jobcache/job_201309040511_0003/jars/org <- /hd/hd1/mapred/local/taskTracke= r/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_00080= 3_1/work/org > 2013-09-04 23:12:34,756 INFO org.apache.hadoop.filecache.TrackerDistribut= edCacheManager: Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt= /jobcache/job_201309040511_0003/jars/job.jar <- /hd/hd1/mapred/local/taskTr= acker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_0= 00803_1/work/job.jar > 2013-09-04 23:12:34,768 INFO org.apache.hadoop.filecache.TrackerDistribut= edCacheManager: Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt= /jobcache/job_201309040511_0003/jars/javax <- /hd/hd1/mapred/local/taskTrac= ker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000= 803_1/work/javax > 2013-09-04 23:12:34,779 INFO org.apache.hadoop.filecache.TrackerDistribut= edCacheManager: Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt= /jobcache/job_201309040511_0003/jars/hive-exec-log4j.properties <- /hd/hd1/= mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_20= 1309040511_0003_m_000803_1/work/hive-exec-log4j.properties > 2013-09-04 23:12:34,790 INFO org.apache.hadoop.filecache.TrackerDistribut= edCacheManager: Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt= /jobcache/job_201309040511_0003/jars/META-INF <- /hd/hd1/mapred/local/taskT= racker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_= 000803_1/work/META-INF > 2013-09-04 23:12:34,796 INFO org.apache.hadoop.filecache.TrackerDistribut= edCacheManager: Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt= /jobcache/job_201309040511_0003/jars/javaewah <- /hd/hd1/mapred/local/taskT= racker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_= 000803_1/work/javaewah > 2013-09-04 23:12:34,813 INFO org.apache.hadoop.filecache.TrackerDistribut= edCacheManager: Creating symlink: /hd/hd5/mapred/local/taskTracker/datasalt= /jobcache/job_201309040511_0003/jars/.job.jar.crc <- /hd/hd1/mapred/local/t= askTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_000= 3_m_000803_1/work/.job.jar.crc > 2013-09-04 23:12:35,127 WARN org.apache.hadoop.metrics2.impl.MetricsSyste= mImpl: Source name ugi already exists! > 2013-09-04 23:12:35,412 INFO org.apache.hadoop.util.ProcessTree: setsid e= xited with exit code 0 > 2013-09-04 23:12:35,473 INFO org.apache.hadoop.mapred.Task: Using Resour= ceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5= d0fa836 > 2013-09-04 23:12:36,927 INFO org.apache.hadoop.hive.ql.io.HiveContextAwar= eRecordReader: Processing file hdfs://master:54310/user/hive/warehouse/hits= /ddate=3D2013-08-28/000798_0 > 2013-09-04 23:12:36,927 INFO org.apache.hadoop.mapred.MapTask: numReduceT= asks: 72 > 2013-09-04 23:12:36,936 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb= =3D 100 > 2013-09-04 23:12:37,028 INFO org.apache.hadoop.mapred.MapTask: data buffe= r =3D 79691776/99614720 > 2013-09-04 23:12:37,028 INFO org.apache.hadoop.mapred.MapTask: record buf= fer =3D 262144/327680 > 2013-09-04 23:12:37,036 WARN org.apache.hadoop.io.compress.snappy.LoadSna= ppy: Snappy native library is available > 2013-09-04 23:12:37,036 INFO org.apache.hadoop.io.compress.snappy.LoadSna= ppy: Snappy native library loaded > 2013-09-04 23:12:37,046 INFO ExecMapper: maximum memory =3D 559284224 > 2013-09-04 23:12:37,046 INFO ExecMapper: conf classpath =3D [file:/home/d= atasalt/sw/hadoop-1.1.2.23/conf/, file:/opt/oracle-jdk-bin-1.7.0.17/lib/too= ls.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/, file:/home/datasalt/sw/had= oop-1.1.2.23/hadoop-core-1.1.2.23.jar, file:/home/datasalt/sw/hadoop-1.1.2.= 23/lib/asm-3.2.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjrt-1.= 6.11.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjtools-1.6.11.ja= r, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-1.7.0.jar, = file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-core-1.8.0.jar= , file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-cli-1.2.jar, file:/hom= e/datasalt/sw/hadoop-1.1.2.23/lib/commons-codec-1.4.jar, file:/home/datasal= t/sw/hadoop-1.1.2.23/lib/commons-collections-3.2.1.jar, file:/home/datasalt= /sw/hadoop-1.1.2.23/lib/commons-configuration-1.6.jar, file:/home/datasalt/= sw/hadoop-1.1.2.23/lib/commons-daemon-1.0.1.jar, file:/home/datasalt/sw/had= oop-1.1.2.23/lib/commons-digester-1.8.jar, file:/home/datasalt/sw/hadoop-1.= 1.2.23/lib/commons-el-1.0.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/c= ommons-httpclient-3.0.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/com= mons-io-2.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-lang-2.= 4.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-1.1.1.jar= , file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-api-1.0.4.jar,= file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-math-2.1.jar, file:/hom= e/datasalt/sw/hadoop-1.1.2.23/lib/commons-net-3.1.jar, file:/home/datasalt/= sw/hadoop-1.1.2.23/lib/core-3.1.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.= 23/lib/hadoop-capacity-scheduler-1.1.2.23.jar, file:/home/datasalt/sw/hadoo= p-1.1.2.23/lib/hadoop-fairscheduler-1.1.2.23.jar, file:/home/datasalt/sw/ha= doop-1.1.2.23/lib/hadoop-thriftfs-1.1.2.23.jar, file:/home/datasalt/sw/hado= op-1.1.2.23/lib/hsqldb-1.8.0.10.jar, file:/home/datasalt/sw/hadoop-1.1.2.23= /lib/jackson-core-asl-1.8.8.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib= /jackson-mapper-asl-1.8.8.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/j= asper-compiler-5.5.12.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jaspe= r-runtime-5.5.12.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jdeb-0.8.j= ar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-core-1.8.jar, file:/h= ome/datasalt/sw/hadoop-1.1.2.23/lib/jersey-json-1.8.jar, file:/home/datasal= t/sw/hadoop-1.1.2.23/lib/jersey-server-1.8.jar, file:/home/datasalt/sw/hado= op-1.1.2.23/lib/jets3t-0.6.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/li= b/jetty-6.1.26.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-util-6= .1.26.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsch-0.1.42.jar, file= :/home/datasalt/sw/hadoop-1.1.2.23/lib/junit-4.5.jar, file:/home/datasalt/s= w/hadoop-1.1.2.23/lib/kfs-0.2.2.jar, file:/home/datasalt/sw/hadoop-1.1.2.23= /lib/log4j-1.2.15.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/mockito-a= ll-1.8.5.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/oro-2.0.8.jar, fil= e:/home/datasalt/sw/hadoop-1.1.2.23/lib/servlet-api-2.5-20081211.jar, file:= /home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-api-1.4.3.jar, file:/home/datas= alt/sw/hadoop-1.1.2.23/lib/slf4j-log4j12-1.4.3.jar, file:/home/datasalt/sw/= hadoop-1.1.2.23/lib/xmlenc-0.52.jar, file:/home/datasalt/sw/hadoop-1.1.2.23= /lib/jsp-2.1/jsp-2.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.= 1/jsp-api-2.1.jar, file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/= job_201309040511_0003/jars/classes, file:/hd/hd5/mapred/local/taskTracker/d= atasalt/jobcache/job_201309040511_0003/jars/, file:/hd/hd1/mapred/local/tas= kTracker/datasalt/distcache/-1837517190973317952_-652944854_1812099894/mast= er/tmp/hadoop-datasalt/mapred/staging/datasalt/.staging/job_201309040511_00= 03/libjars/derby-10.4.2.0.jar/, file:/hd/hd1/mapred/local/taskTracker/datas= alt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/wor= k/] > 2013-09-04 23:12:37,047 INFO ExecMapper: thread classpath =3D [file:/home= /datasalt/sw/hadoop-1.1.2.23/conf/, file:/opt/oracle-jdk-bin-1.7.0.17/lib/t= ools.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/, file:/home/datasalt/sw/h= adoop-1.1.2.23/hadoop-core-1.1.2.23.jar, file:/home/datasalt/sw/hadoop-1.1.= 2.23/lib/asm-3.2.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjrt-= 1.6.11.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjtools-1.6.11.= jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-1.7.0.jar= , file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-core-1.8.0.j= ar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-cli-1.2.jar, file:/h= ome/datasalt/sw/hadoop-1.1.2.23/lib/commons-codec-1.4.jar, file:/home/datas= alt/sw/hadoop-1.1.2.23/lib/commons-collections-3.2.1.jar, file:/home/datasa= lt/sw/hadoop-1.1.2.23/lib/commons-configuration-1.6.jar, file:/home/datasal= t/sw/hadoop-1.1.2.23/lib/commons-daemon-1.0.1.jar, file:/home/datasalt/sw/h= adoop-1.1.2.23/lib/commons-digester-1.8.jar, file:/home/datasalt/sw/hadoop-= 1.1.2.23/lib/commons-el-1.0.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib= /commons-httpclient-3.0.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/c= ommons-io-2.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-lang-= 2.4.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-1.1.1.j= ar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-api-1.0.4.ja= r, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-math-2.1.jar, file:/h= ome/datasalt/sw/hadoop-1.1.2.23/lib/commons-net-3.1.jar, file:/home/datasal= t/sw/hadoop-1.1.2.23/lib/core-3.1.1.jar, file:/home/datasalt/sw/hadoop-1.1.= 2.23/lib/hadoop-capacity-scheduler-1.1.2.23.jar, file:/home/datasalt/sw/had= oop-1.1.2.23/lib/hadoop-fairscheduler-1.1.2.23.jar, file:/home/datasalt/sw/= hadoop-1.1.2.23/lib/hadoop-thriftfs-1.1.2.23.jar, file:/home/datasalt/sw/ha= doop-1.1.2.23/lib/hsqldb-1.8.0.10.jar, file:/home/datasalt/sw/hadoop-1.1.2.= 23/lib/jackson-core-asl-1.8.8.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/l= ib/jackson-mapper-asl-1.8.8.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib= /jasper-compiler-5.5.12.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jas= per-runtime-5.5.12.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jdeb-0.8= .jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-core-1.8.jar, file:= /home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-json-1.8.jar, file:/home/datas= alt/sw/hadoop-1.1.2.23/lib/jersey-server-1.8.jar, file:/home/datasalt/sw/ha= doop-1.1.2.23/lib/jets3t-0.6.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/= lib/jetty-6.1.26.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-util= -6.1.26.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsch-0.1.42.jar, fi= le:/home/datasalt/sw/hadoop-1.1.2.23/lib/junit-4.5.jar, file:/home/datasalt= /sw/hadoop-1.1.2.23/lib/kfs-0.2.2.jar, file:/home/datasalt/sw/hadoop-1.1.2.= 23/lib/log4j-1.2.15.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/mockito= -all-1.8.5.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/oro-2.0.8.jar, f= ile:/home/datasalt/sw/hadoop-1.1.2.23/lib/servlet-api-2.5-20081211.jar, fil= e:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-api-1.4.3.jar, file:/home/dat= asalt/sw/hadoop-1.1.2.23/lib/slf4j-log4j12-1.4.3.jar, file:/home/datasalt/s= w/hadoop-1.1.2.23/lib/xmlenc-0.52.jar, file:/home/datasalt/sw/hadoop-1.1.2.= 23/lib/jsp-2.1/jsp-2.1.jar, file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-= 2.1/jsp-api-2.1.jar, file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcach= e/job_201309040511_0003/jars/classes, file:/hd/hd5/mapred/local/taskTracker= /datasalt/jobcache/job_201309040511_0003/jars/, file:/hd/hd1/mapred/local/t= askTracker/datasalt/distcache/-1837517190973317952_-652944854_1812099894/ma= ster/tmp/hadoop-datasalt/mapred/staging/datasalt/.staging/job_201309040511_= 0003/libjars/derby-10.4.2.0.jar/, file:/hd/hd1/mapred/local/taskTracker/dat= asalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/w= ork/] > 2013-09-04 23:12:37,085 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = Adding alias hits to work list for file hdfs://master:54310/user/hive/wareh= ouse/hits/ddate=3D2013-08-28 > 2013-09-04 23:12:37,089 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = dump TS struct > 2013-09-04 23:12:37,091 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = Adding alias hits to work list for file hdfs://master:54310/user/hive/wareh= ouse/hits/ddate=3D2013-08-29 > 2013-09-04 23:12:37,092 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = Adding alias hits to work list for file hdfs://master:54310/user/hive/wareh= ouse/hits/ddate=3D2013-08-30 > 2013-09-04 23:12:37,093 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = Adding alias hits to work list for file hdfs://master:54310/user/hive/wareh= ouse/hits/ddate=3D2013-08-31 > 2013-09-04 23:12:37,094 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = Adding alias hits to work list for file hdfs://master:54310/user/hive/wareh= ouse/hits/ddate=3D2013-09-01 > 2013-09-04 23:12:37,094 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = Adding alias hits to work list for file hdfs://master:54310/user/hive/wareh= ouse/hits/ddate=3D2013-09-02 > 2013-09-04 23:12:37,095 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = Adding alias hits to work list for file hdfs://master:54310/user/hive/wareh= ouse/hits/ddate=3D2013-09-03 > 2013-09-04 23:12:37,096 INFO ExecMapper:=20 > Id =3D7 > > Id =3D3 > > Id =3D2 > > Id =3D1 > > Id =3D0 > Id =3D 1 null<\Parent> > <\RS> > <\Children> > Id =3D 2 null<\Parent> > <\GBY> > <\Children> > Id =3D 3 null<\Parent> > <\SEL> > <\Children> > Id =3D 7 null<\Parent> > <\TS> > <\Children> > <\MAP> > 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = Initializing Self 7 MAP > 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.TableScanOper= ator: Initializing Self 3 TS > 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.TableScanOper= ator: Operator 3 TS initialized > 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.TableScanOper= ator: Initializing children of 3 TS > 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: Initializing child 2 SEL > 2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: Initializing Self 2 SEL > 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: SELECT struct > 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: Operator 2 SEL initialized > 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: Initializing children of 2 SEL > 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.GroupByOperat= or: Initializing child 1 GBY > 2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.GroupByOperat= or: Initializing Self 1 GBY > 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.GroupByOperat= or: Operator 1 GBY initialized > 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.GroupByOperat= or: Initializing children of 1 GBY > 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.ReduceSinkOpe= rator: Initializing child 0 RS > 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.ReduceSinkOpe= rator: Initializing Self 0 RS > 2013-09-04 23:12:37,116 INFO org.apache.hadoop.hive.ql.exec.ReduceSinkOpe= rator: Using tag =3D -1 > 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.ReduceSinkOpe= rator: Operator 0 RS initialized > 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.ReduceSinkOpe= rator: Initialization Done 0 RS > 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.GroupByOperat= or: Initialization Done 1 GBY > 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: Initialization Done 2 SEL > 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.TableScanOper= ator: Initialization Done 3 TS > 2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = Initialization Done 7 MAP > 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = Processing alias hits for file hdfs://master:54310/user/hive/warehouse/hits= /ddate=3D2013-08-28 > 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = 7 forwarding 1 rows > 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.TableScanOper= ator: 3 forwarding 1 rows > 2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: 2 forwarding 1 rows > 2013-09-04 23:12:37,168 INFO ExecMapper: ExecMapper: processing 1 rows: u= sed memory =3D 151151936 > 2013-09-04 23:12:37,169 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = 7 forwarding 10 rows > 2013-09-04 23:12:37,169 INFO org.apache.hadoop.hive.ql.exec.TableScanOper= ator: 3 forwarding 10 rows > 2013-09-04 23:12:37,169 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: 2 forwarding 10 rows > 2013-09-04 23:12:37,169 INFO ExecMapper: ExecMapper: processing 10 rows: = used memory =3D 151151936 > 2013-09-04 23:12:37,175 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = 7 forwarding 100 rows > 2013-09-04 23:12:37,175 INFO org.apache.hadoop.hive.ql.exec.TableScanOper= ator: 3 forwarding 100 rows > 2013-09-04 23:12:37,175 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: 2 forwarding 100 rows > 2013-09-04 23:12:37,175 INFO ExecMapper: ExecMapper: processing 100 rows:= used memory =3D 151151936 > 2013-09-04 23:12:37,247 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = 7 forwarding 1000 rows > 2013-09-04 23:12:37,250 INFO org.apache.hadoop.hive.ql.exec.TableScanOper= ator: 3 forwarding 1000 rows > 2013-09-04 23:12:37,250 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: 2 forwarding 1000 rows > 2013-09-04 23:12:37,251 INFO ExecMapper: ExecMapper: processing 1000 rows= : used memory =3D 151151936 > 2013-09-04 23:12:37,642 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = 7 forwarding 10000 rows > 2013-09-04 23:12:37,642 INFO org.apache.hadoop.hive.ql.exec.TableScanOper= ator: 3 forwarding 10000 rows > 2013-09-04 23:12:37,644 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: 2 forwarding 10000 rows > 2013-09-04 23:12:37,644 INFO ExecMapper: ExecMapper: processing 10000 row= s: used memory =3D 153032176 > 2013-09-04 23:12:38,467 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = 7 forwarding 100000 rows > 2013-09-04 23:12:38,468 INFO org.apache.hadoop.hive.ql.exec.TableScanOper= ator: 3 forwarding 100000 rows > 2013-09-04 23:12:38,468 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: 2 forwarding 100000 rows > 2013-09-04 23:12:38,469 INFO ExecMapper: ExecMapper: processing 100000 ro= ws: used memory =3D 125346272 > 2013-09-04 23:12:40,626 INFO org.apache.hadoop.hive.ql.exec.MapOperator: = 7 forwarding 1000000 rows > 2013-09-04 23:12:40,627 INFO org.apache.hadoop.hive.ql.exec.TableScanOper= ator: 3 forwarding 1000000 rows > 2013-09-04 23:12:40,627 INFO org.apache.hadoop.hive.ql.exec.SelectOperato= r: 2 forwarding 1000000 rows > 2013-09-04 23:12:40,627 INFO ExecMapper: ExecMapper: processing 1000000 r= ows: used memory =3D 123383952 > {noformat}=20 > For this particular example, the task was killed 6 hours after the last l= og line.=20 > The strace of the process when it is in the infinite loop has the followi= ng looking: > {noformat}=20 > Process 7887 attached with 24 threads > [pid 7930] futex(0x7f20a06df28c, FUTEX_WAIT_PRIVATE, 1, NULL > [pid 7922] restart_syscall(<... resuming interrupted call ...> > [pid 7920] restart_syscall(<... resuming interrupted call ...> > [pid 8333] accept(61, > [pid 7908] restart_syscall(<... resuming interrupted call ...> > [pid 7907] restart_syscall(<... resuming interrupted call ...> > [pid 7906] restart_syscall(<... resuming interrupted call ...> > [pid 7905] futex(0x7f20a00fa554, FUTEX_WAIT_PRIVATE, 3, NULL > [pid 7903] futex(0x7f20a00f5254, FUTEX_WAIT_PRIVATE, 417, NULL > [pid 7904] futex(0x7f20a00f7c54, FUTEX_WAIT_PRIVATE, 387, NULL > [pid 7902] futex(0x7f20a7513dc0, FUTEX_WAIT_PRIVATE, 0, NULL > [pid 7901] futex(0x7f20a00a6f54, FUTEX_WAIT_PRIVATE, 7, NULL > [pid 7899] restart_syscall(<... resuming interrupted call ...> > [pid 7898] futex(0x7f20a0026554, FUTEX_WAIT_PRIVATE, 39, NULL > [pid 7900] futex(0x7f20a00a4b54, FUTEX_WAIT_PRIVATE, 7, NULL > [pid 7897] futex(0x7f20a0024754, FUTEX_WAIT_PRIVATE, 39, NULL > [pid 7896] futex(0x7f20a0022a54, FUTEX_WAIT_PRIVATE, 41, NULL > [pid 7895] futex(0x7f20a0020c54, FUTEX_WAIT_PRIVATE, 39, NULL > [pid 7894] futex(0x7f20a001ef54, FUTEX_WAIT_PRIVATE, 37, NULL > [pid 7893] futex(0x7f20a001d154, FUTEX_WAIT_PRIVATE, 39, NULL > [pid 7887] futex(0x7f20a81389d0, FUTEX_WAIT, 7890, NULL > [pid 7892] futex(0x7f20a001b454, FUTEX_WAIT_PRIVATE, 35, NULL > [pid 7891] futex(0x7f20a0019654, FUTEX_WAIT_PRIVATE, 37, NULL > [pid 7906] <... restart_syscall resumed> ) =3D -1 ETIMEDOUT (Connection = timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373836, 993325000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 43598000}, ffffffff) =3D -1 ETIMEDOUT (Connection = timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 93858000}, ffffffff) =3D -1 ETIMEDOUT (Connection = timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 144101000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 194344000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 244659000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 294924000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 345198000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 395462000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 445768000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 495983000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 546226000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 596489000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 646798000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 697063000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 747370000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 797688000}, ffffffff > [pid 7899] <... restart_syscall resumed> ) =3D -1 ETIMEDOUT (Connection = timed out) > [pid 7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7899] futex(0x7f20a009d054, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 782077000}, ffffffff > [pid 7906] <... futex resumed> ) =3D -1 ETIMEDOUT (Connection time= d out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 848014000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 898283000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 948489000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373837, 998751000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 49086000}, ffffffff) =3D -1 ETIMEDOUT (Connection = timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 99351000}, ffffffff) =3D -1 ETIMEDOUT (Connection = timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 149589000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 199852000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 250180000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 300489000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 350762000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 401078000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 451258000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 501527000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 551767000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 601970000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 652161000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 702290000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 752523000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 802739000}, ffffffff > [pid 7899] <... futex resumed> ) =3D -1 ETIMEDOUT (Connection time= d out) > [pid 7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7899] futex(0x7f20a009d054, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 782336000}, ffffffff > [pid 7922] <... restart_syscall resumed> ) =3D -1 ETIMEDOUT (Connection = timed out) > [pid 7922] futex(0x7f20a06d3a28, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7922] futex(0x7f20a0550554, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f20a055= 0550, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) =3D 1 > [pid 7907] <... restart_syscall resumed> ) =3D 0 > [pid 7907] futex(0x7f20a0550528, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7922] write(56, "\0\0\0\335\0\0\2.\0\4ping\0\0\0\2\0&org.apache.h".= .., 225 > [pid 7907] read(56, > [pid 7922] <... write resumed> ) =3D 225 > [pid 7907] <... read resumed> 0x7f2098000b70, 8192) =3D -1 EAGAIN (Resou= rce temporarily unavailable) > [pid 7922] futex(0x7f20a009d054, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f20a009= d050, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) =3D 1 > [pid 7899] <... futex resumed> ) =3D 0 > [pid 7922] futex(0x7f20a06d3e54, FUTEX_WAIT_PRIVATE, 53, NULL > [pid 7907] epoll_ctl(59, EPOLL_CTL_ADD, 56, {...} > [pid 7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1 > [pid 7907] <... epoll_ctl resumed> ) =3D 0 > [pid 7899] <... futex resumed> ) =3D 0 > [pid 7907] epoll_wait(59, {?} 0x7f20a0544000, 4096, 60000) =3D 1 > [pid 7899] mprotect(0x7f20a8147000, 4096, PROT_READ > [pid 7907] futex(0x7f20a0551e54, FUTEX_WAIT_PRIVATE, 11, NULL > [pid 7899] <... mprotect resumed> ) =3D 0 > [pid 7890] futex(0x7f20a000b854, FUTEX_WAIT_PRIVATE, 349, NULL > [pid 7899] mprotect(0x7f20a8147000, 4096, PROT_READ|PROT_WRITE) =3D 0 > [pid 7899] mprotect(0x7f20a8148000, 4096, PROT_NONE) =3D 0 > [pid 7899] mprotect(0x7f20a8148000, 4096, PROT_READ) =3D 0 > [pid 7899] futex(0x7f20a0551e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f20a055= 1e50, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) =3D 1 > [pid 7899] futex(0x7f20a06d3e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f20a06d= 3e50, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} > [pid 7922] <... futex resumed> ) =3D 0 > [pid 7907] <... futex resumed> ) =3D 0 > [pid 7922] futex(0x7f20a06d3e28, FUTEX_WAIT_PRIVATE, 2, NULL > [pid 7899] <... futex resumed> ) =3D 1 > [pid 7907] futex(0x7f20a0551e28, FUTEX_WAKE_PRIVATE, 1 > [pid 7899] futex(0x7f20a06d3e28, FUTEX_WAKE_PRIVATE, 1 > [pid 7922] <... futex resumed> ) =3D 0 > [pid 7907] <... futex resumed> ) =3D 0 > [pid 7922] futex(0x7f20a06d3e28, FUTEX_WAKE_PRIVATE, 1 > [pid 7899] <... futex resumed> ) =3D 1 > [pid 7922] <... futex resumed> ) =3D 0 > [pid 7899] futex(0x7f20a009d054, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 3, {1378373839, 794194000}, ffffffff > [pid 7922] futex(0x7f20a06d3a54, FUTEX_WAIT_PRIVATE, 1, NULL > [pid 7907] futex(0x7f20a000b854, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f20a000= b850, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) =3D 1 > [pid 7890] <... futex resumed> ) =3D 0 > [pid 7890] futex(0x7f20a000b828, FUTEX_WAKE_PRIVATE, 1 > [pid 7907] epoll_ctl(59, EPOLL_CTL_DEL, 56, {...} > [pid 7890] <... futex resumed> ) =3D 0 > [pid 7907] <... epoll_ctl resumed> ) =3D 0 > [pid 7907] epoll_wait(59, {}, 4096, 0) =3D 0 > [pid 7907] read(56, "\0\0\2.\0\0\0\0\0\7boolean\1", 8192) =3D 18 > [pid 7907] futex(0x7f20a06d3a54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f20a06d= 3a50, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) =3D 1 > [pid 7922] <... futex resumed> ) =3D 0 > [pid 7907] futex(0x7f20a0550554, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1107, {1378373848, 793674000}, ffffffff > [pid 7922] futex(0x7f20a06d3a28, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7922] futex(0x7f20a06d3a54, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 3, {1378373841, 794758000}, ffffffff > [pid 7906] <... futex resumed> ) =3D -1 ETIMEDOUT (Connection time= d out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 852867000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 903052000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373838, 953226000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 3436000}, ffffffff) =3D -1 ETIMEDOUT (Connection t= imed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 53695000}, ffffffff) =3D -1 ETIMEDOUT (Connection = timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 103966000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 154232000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 204466000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 254802000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 305067000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 355318000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 405623000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 455817000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 506130000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 556395000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 606661000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 656898000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > [pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) =3D 0 > [pid 7906] futex(0x7f20a0104954, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_R= EALTIME, 1, {1378373839, 707177000}, ffffffff) =3D -1 ETIMEDOUT (Connection= timed out) > {noformat}=20 -- This message was sent by Atlassian JIRA (v6.1#6144)