Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 3DA1B200C56 for ; Fri, 14 Apr 2017 23:59:00 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 3C0EB160B8C; Fri, 14 Apr 2017 21:59:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 160A8160B8A for ; Fri, 14 Apr 2017 23:58:57 +0200 (CEST) Received: (qmail 51617 invoked by uid 500); 14 Apr 2017 21:58:57 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 51608 invoked by uid 99); 14 Apr 2017 21:58:57 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Apr 2017 21:58:57 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id C15ED183A68 for ; Fri, 14 Apr 2017 21:58:56 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id igt_AN6OfJOm for ; Fri, 14 Apr 2017 21:58:43 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id D89085FAD8 for ; Fri, 14 Apr 2017 21:58:42 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id D723BE0069 for ; Fri, 14 Apr 2017 21:58:41 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 88E0D20D54 for ; Fri, 14 Apr 2017 21:58:41 +0000 (UTC) Date: Fri, 14 Apr 2017 21:58:41 +0000 (UTC) From: "Thomas Weise (JIRA)" To: commits@beam.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (BEAM-1980) Seeming deadlock using Apex with relatively small data MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 14 Apr 2017 21:59:00 -0000 [ https://issues.apache.org/jira/browse/BEAM-1980?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15969= 586#comment-15969586 ]=20 Thomas Weise commented on BEAM-1980: ------------------------------------ Did the stalled run produce the expected output and just not exit or did it= get stuck while processing? > Seeming deadlock using Apex with relatively small data > ------------------------------------------------------ > > Key: BEAM-1980 > URL: https://issues.apache.org/jira/browse/BEAM-1980 > Project: Beam > Issue Type: Bug > Components: runner-apex > Reporter: Daniel Halperin > Assignee: Thomas Weise > Fix For: First stable release > > > I'm running the "beam portability demo" at https://github.com/dhalperi/be= am-portability-demo/tree/apex > Made a very small input file: > {code} > gsutil cat gs://apache-beam-demo/data2/small-game.csv | head -n 100000 > = tiny.csv > {code} > Ran the job in embedded mode using an Apex fat-jar from the pom in that b= ranch (and adding in {{slf4j-jdk14.jar}} for debugging info): > {code} > java -cp ~/.m2/repository/org/slf4j/slf4j-jdk14/1.7.14/slf4j-jdk14-1.7.14= .jar:target/portability-demo-bundled-apex.jar demo.HourlyTeamScore --runner= =3DApexRunner --outputPrefix=3Dgs://clouddfe-dhalperi/output/apex --input= =3Dtiny.csv > {code} > A good run takes O(25 seconds): > {code} > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in [jar:file:/Users/dhalperi/.m2/repository/org/slf4= j/slf4j-jdk14/1.7.14/slf4j-jdk14-1.7.14.jar!/org/slf4j/impl/StaticLoggerBin= der.class] > SLF4J: Found binding in [jar:file:/Users/dhalperi/beam-portability-demo/t= arget/portability-demo-bundled-apex.jar!/org/slf4j/impl/StaticLoggerBinder.= class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an expla= nation. > SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory] > log4j:WARN No appenders could be found for logger (org.apache.commons.bea= nutils.converters.BooleanConverter). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for = more info. > Apr 14, 2017 1:20:55 PM com.datatorrent.common.util.AsyncFSStorageAgent s= ave > INFO: using /var/folders/7r/s0gg2qb11jz4g4pcb8n15gkc009j1y/T/chkp80748382= 77485202831 as the basepath for checkpointing. > Apr 14, 2017 1:20:56 PM com.datatorrent.bufferserver.storage.DiskStorage = > INFO: using /var/folders/7r/s0gg2qb11jz4g4pcb8n15gkc009j1y/T as the basep= ath for spooling. > Apr 14, 2017 1:20:56 PM com.datatorrent.bufferserver.server.Server regist= ered > INFO: Server started listening at /0:0:0:0:0:0:0:0:61087 > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster > INFO: Buffer server started: localhost:61087 > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-0 > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-1 > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-2 > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-3 > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-4 > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-5 > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-2 msg: [container-2] Entering heartbeat loop.. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-1 msg: [container-1] Entering heartbeat loop.. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-5 msg: [container-5] Entering heartbeat loop.. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-4 msg: [container-4] Entering heartbeat loop.. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-0 msg: [container-0] Entering heartbeat loop.. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-7 > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-7 msg: [container-7] Entering heartbeat loop.. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-6 > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-3 msg: [container-3] Entering heartbeat loop.. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-6 msg: [container-6] Entering heartbeat loop.. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-8 > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-8 msg: [container-8] Entering heartbeat loop.. > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-9 > Apr 14, 2017 1:20:56 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-9 msg: [container-9] Entering heartbeat loop.. > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D4,name=3DFixedWindows/Wind= ow.Assign,type=3DGENERIC,checkpoint=3D{ffffffffffffffff, 0, 0},inputs=3D[Op= eratorDeployInfo.InputDeployInfo[portName=3DinputPort,streamId=3Dstream1,so= urceNodeId=3D3,sourcePortName=3Doutput,locality=3D,partitionMask=3D0,= partitionKeys=3D]],outputs=3D[OperatorDeployInfo.OutputDeployInfo[por= tName=3DoutputPort,streamId=3Dstream7,bufferServer=3Dlocalhost]]]] > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D6,name=3DSumTeamScores/Com= bine.perKey(SumInteger)/GroupByKey,type=3DGENERIC,checkpoint=3D{fffffffffff= fffff, 0, 0},inputs=3D[OperatorDeployInfo.InputDeployInfo[portName=3Dinput,= streamId=3Dstream4,sourceNodeId=3D5,sourcePortName=3Doutput,locality=3D,partitionMask=3D0,partitionKeys=3D]],outputs=3D[OperatorDeployInfo= .OutputDeployInfo[portName=3Doutput,streamId=3Dstream0,bufferServer=3Dlocal= host]]]] > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D2,name=3DParseGameEvent/Pa= rMultiDo(ParseEvent),type=3DGENERIC,checkpoint=3D{ffffffffffffffff, 0, 0},i= nputs=3D[OperatorDeployInfo.InputDeployInfo[portName=3Dinput,streamId=3Dstr= eam6,sourceNodeId=3D1,sourcePortName=3Doutput,locality=3D,partitionMa= sk=3D0,partitionKeys=3D]],outputs=3D[OperatorDeployInfo.OutputDeployI= nfo[portName=3Doutput,streamId=3Dstream2,bufferServer=3Dlocalhost]]]] > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D6.output.6, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61087/5.output.5, windowId=3Dffffffffffffffff, = type=3Dstream4/6.input, upstreamIdentifier=3D5.output.5, mask=3D0, partitio= ns=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D4.outputPort.4, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D2.output.2, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61087/3.output.3, windowId=3Dffffffffffffffff, = type=3Dstream1/4.inputPort, upstreamIdentifier=3D3.output.3, mask=3D0, part= itions=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61087/1.output.1, windowId=3Dffffffffffffffff, = type=3Dstream6/2.input, upstreamIdentifier=3D1.output.1, mask=3D0, partitio= ns=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D10,name=3DSumTeamScores/Pa= rDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles),type=3DGENERIC,check= point=3D{ffffffffffffffff, 0, 0},inputs=3D[OperatorDeployInfo.InputDeployIn= fo[portName=3Dinput,streamId=3Dstream3,sourceNodeId=3D9,sourcePortName=3Dou= tput,locality=3D,partitionMask=3D0,partitionKeys=3D]],outputs= =3D[]]] > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D8,name=3DSumTeamScores/Par= Do(KeyByWindow)/ParMultiDo(KeyByWindow),type=3DGENERIC,checkpoint=3D{ffffff= ffffffffff, 0, 0},inputs=3D[OperatorDeployInfo.InputDeployInfo[portName=3Di= nput,streamId=3Dstream5,sourceNodeId=3D7,sourcePortName=3Doutput,locality= =3D,partitionMask=3D0,partitionKeys=3D]],outputs=3D[OperatorDep= loyInfo.OutputDeployInfo[portName=3Doutput,streamId=3Dstream8,bufferServer= =3Dlocalhost]]]] > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D5,name=3DSumTeamScores/Par= Do(KeyScoreByTeam)/ParMultiDo(KeyScoreByTeam),type=3DGENERIC,checkpoint=3D{= ffffffffffffffff, 0, 0},inputs=3D[OperatorDeployInfo.InputDeployInfo[portNa= me=3Dinput,streamId=3Dstream7,sourceNodeId=3D4,sourcePortName=3DoutputPort,= locality=3D,partitionMask=3D0,partitionKeys=3D]],outputs=3D[Ope= ratorDeployInfo.OutputDeployInfo[portName=3Doutput,streamId=3Dstream4,buffe= rServer=3Dlocalhost]]]] > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D3,name=3DSetTimestamps/Par= MultiDo(SetTimestamps),type=3DGENERIC,checkpoint=3D{ffffffffffffffff, 0, 0}= ,inputs=3D[OperatorDeployInfo.InputDeployInfo[portName=3Dinput,streamId=3Ds= tream2,sourceNodeId=3D2,sourcePortName=3Doutput,locality=3D,partition= Mask=3D0,partitionKeys=3D]],outputs=3D[OperatorDeployInfo.OutputDeplo= yInfo[portName=3Doutput,streamId=3Dstream1,bufferServer=3Dlocalhost]]]] > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D7,name=3DSumTeamScores/Com= bine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(A= nonymous),type=3DGENERIC,checkpoint=3D{ffffffffffffffff, 0, 0},inputs=3D[Op= eratorDeployInfo.InputDeployInfo[portName=3Dinput,streamId=3Dstream0,source= NodeId=3D6,sourcePortName=3Doutput,locality=3D,partitionMask=3D0,part= itionKeys=3D]],outputs=3D[OperatorDeployInfo.OutputDeployInfo[portNam= e=3Doutput,streamId=3Dstream5,bufferServer=3Dlocalhost]]]] > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D1,name=3DTextIO.Read/Read,= type=3DINPUT,checkpoint=3D{ffffffffffffffff, 0, 0},inputs=3D[],outputs=3D[O= peratorDeployInfo.OutputDeployInfo[portName=3Doutput,streamId=3Dstream6,buf= ferServer=3Dlocalhost]]]] > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D9,name=3DSumTeamScores/Gro= upByKey,type=3DGENERIC,checkpoint=3D{ffffffffffffffff, 0, 0},inputs=3D[Oper= atorDeployInfo.InputDeployInfo[portName=3Dinput,streamId=3Dstream8,sourceNo= deId=3D8,sourcePortName=3Doutput,locality=3D,partitionMask=3D0,partit= ionKeys=3D]],outputs=3D[OperatorDeployInfo.OutputDeployInfo[portName= =3Doutput,streamId=3Dstream3,bufferServer=3Dlocalhost]]]] > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D8.output.8, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61087/7.output.7, windowId=3Dffffffffffffffff, = type=3Dstream5/8.input, upstreamIdentifier=3D7.output.7, mask=3D0, partitio= ns=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61087/9.output.9, windowId=3Dffffffffffffffff, = type=3Dstream3/10.input, upstreamIdentifier=3D9.output.9, mask=3D0, partiti= ons=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D5.output.5, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61087/4.outputPort.4, windowId=3Dffffffffffffff= ff, type=3Dstream7/5.input, upstreamIdentifier=3D4.outputPort.4, mask=3D0, = partitions=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:20:57 PM com.datatorrent.stram.engine.WindowGenerator acti= vate > INFO: Catching up from 1492201255500 to 1492201257769 > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61087/2.output.2, windowId=3Dffffffffffffffff, = type=3Dstream2/3.input, upstreamIdentifier=3D2.output.2, mask=3D0, partitio= ns=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D3.output.3, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D9.output.9, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61087/8.output.8, windowId=3Dffffffffffffffff, = type=3Dstream8/9.input, upstreamIdentifier=3D8.output.8, mask=3D0, partitio= ns=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61087/6.output.6, windowId=3Dffffffffffffffff, = type=3Dstream0/7.input, upstreamIdentifier=3D6.output.6, mask=3D0, partitio= ns=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D7.output.7, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:20:57 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D1.output.1, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:20:57 PM org.apache.beam.sdk.io.FileBasedSource expandFile= Pattern > INFO: Matched 1 files for pattern tiny.csv > Apr 14, 2017 1:20:58 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:58 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:58 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:58 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:58 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:58 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:58 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:58 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:58 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:20:58 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:01 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Undeploy request: [1] > Apr 14, 2017 1:21:01 PM com.datatorrent.stram.engine.StreamingContainer u= ndeploy > INFO: Undeploy complete. > Apr 14, 2017 1:21:03 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Undeploy request: [2] > Apr 14, 2017 1:21:03 PM com.datatorrent.stram.engine.StreamingContainer u= ndeploy > INFO: Undeploy complete. > Apr 14, 2017 1:21:07 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Undeploy request: [5] > Apr 14, 2017 1:21:07 PM com.datatorrent.stram.engine.StreamingContainer u= ndeploy > INFO: Undeploy complete. > Apr 14, 2017 1:21:07 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Undeploy request: [3] > Apr 14, 2017 1:21:07 PM com.datatorrent.stram.engine.StreamingContainer u= ndeploy > INFO: Undeploy complete. > Apr 14, 2017 1:21:08 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Undeploy request: [4] > Apr 14, 2017 1:21:08 PM com.datatorrent.stram.engine.StreamingContainer u= ndeploy > INFO: Undeploy complete. > Apr 14, 2017 1:21:20 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Undeploy request: [6] > Apr 14, 2017 1:21:20 PM com.datatorrent.stram.engine.StreamingContainer u= ndeploy > INFO: Undeploy complete. > Apr 14, 2017 1:21:20 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Undeploy request: [8] > Apr 14, 2017 1:21:20 PM com.datatorrent.stram.engine.StreamingContainer u= ndeploy > INFO: Undeploy complete. > Apr 14, 2017 1:21:20 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Undeploy request: [9] > Apr 14, 2017 1:21:20 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Undeploy request: [7] > Apr 14, 2017 1:21:20 PM com.datatorrent.stram.engine.StreamingContainer u= ndeploy > INFO: Undeploy complete. > Apr 14, 2017 1:21:20 PM com.datatorrent.stram.engine.StreamingContainer u= ndeploy > INFO: Undeploy complete. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StreamingContainerManager p= rocessEvents > INFO: Removing inactive operators at window 58f12f270000000b [PTOperator[= id=3D1,name=3DTextIO.Read/Read]] > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StreamingContainerManager p= rocessHeartbeat > INFO: requesting idle shutdown for container container-9 > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StreamingContainerManager p= rocessHeartbeat > INFO: requesting idle shutdown for container container-0 > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Received shutdown request > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-9 msg: [container-9] Exiting heartbeat loop.. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Received shutdown request > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-0 msg: [container-0] Exiting heartbeat loop.. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Container container-9 terminating. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Container container-0 terminating. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StreamingContainerManager p= rocessHeartbeat > INFO: requesting idle shutdown for container container-7 > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Received shutdown request > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-7 msg: [container-7] Exiting heartbeat loop.. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Container container-7 terminating. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StreamingContainerManager p= rocessHeartbeat > INFO: requesting idle shutdown for container container-4 > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StreamingContainerManager p= rocessHeartbeat > INFO: requesting idle shutdown for container container-6 > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Received shutdown request > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Received shutdown request > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-6 msg: [container-6] Exiting heartbeat loop.. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-4 msg: [container-4] Exiting heartbeat loop.. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Container container-6 terminating. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Container container-4 terminating. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StreamingContainerManager p= rocessHeartbeat > INFO: requesting idle shutdown for container container-5 > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StreamingContainerManager p= rocessHeartbeat > INFO: Heartbeat for unknown operator 10 (container container-1) > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Received shutdown request > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StreamingContainerManager p= rocessHeartbeat > INFO: requesting shutdown for container container-1 > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-5 msg: [container-5] Exiting heartbeat loop.. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Undeploy request: [10] > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Container container-5 terminating. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.engine.StreamingContainer u= ndeploy > INFO: Undeploy complete. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Received shutdown request > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-1 msg: [container-1] Exiting heartbeat loop.. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Container container-1 terminating. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StreamingContainerManager p= rocessHeartbeat > INFO: requesting idle shutdown for container container-2 > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StreamingContainerManager p= rocessHeartbeat > INFO: requesting idle shutdown for container container-3 > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Received shutdown request > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Received shutdown request > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-2 msg: [container-2] Exiting heartbeat loop.. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-3 msg: [container-3] Exiting heartbeat loop.. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Container container-2 terminating. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Container container-3 terminating. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StreamingContainerManager p= rocessHeartbeat > INFO: requesting idle shutdown for container container-8 > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Received shutdown request > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-8 msg: [container-8] Exiting heartbeat loop.. > Apr 14, 2017 1:21:21 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Container container-8 terminating. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.StreamingContainerManager p= rocessEvents > INFO: Removing inactive operators at window 58f12f270000000c [PTOperator[= id=3D9,name=3DSumTeamScores/GroupByKey], PTOperator[id=3D4,name=3DFixedWind= ows/Window.Assign], PTOperator[id=3D5,name=3DSumTeamScores/ParDo(KeyScoreBy= Team)/ParMultiDo(KeyScoreByTeam)], PTOperator[id=3D3,name=3DSetTimestamps/P= arMultiDo(SetTimestamps)], PTOperator[id=3D2,name=3DParseGameEvent/ParMulti= Do(ParseEvent)], PTOperator[id=3D6,name=3DSumTeamScores/Combine.perKey(SumI= nteger)/GroupByKey], PTOperator[id=3D8,name=3DSumTeamScores/ParDo(KeyByWind= ow)/ParMultiDo(KeyByWindow)], PTOperator[id=3D7,name=3DSumTeamScores/Combin= e.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anon= ymous)]] > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:21:22 PM com.datatorrent.stram.StramLocalCluster run > INFO: Application finished. > Apr 14, 2017 1:21:22 PM com.datatorrent.bufferserver.server.Server unregi= stered > INFO: Server stopped listening at /0:0:0:0:0:0:0:0:61087 > {code} > However I got a seemingly stalled run, for which I captured a {{jstack}} = after about 3 minutes: > {code} > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in [jar:file:/Users/dhalperi/.m2/repository/org/slf4= j/slf4j-jdk14/1.7.14/slf4j-jdk14-1.7.14.jar!/org/slf4j/impl/StaticLoggerBin= der.class] > SLF4J: Found binding in [jar:file:/Users/dhalperi/beam-portability-demo/t= arget/portability-demo-bundled-apex.jar!/org/slf4j/impl/StaticLoggerBinder.= class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an expla= nation. > SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory] > log4j:WARN No appenders could be found for logger (org.apache.commons.bea= nutils.converters.BooleanConverter). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for = more info. > Apr 14, 2017 1:16:52 PM com.datatorrent.common.util.AsyncFSStorageAgent s= ave > INFO: using /var/folders/7r/s0gg2qb11jz4g4pcb8n15gkc009j1y/T/chkp49658981= 9849283689 as the basepath for checkpointing. > Apr 14, 2017 1:16:53 PM com.datatorrent.bufferserver.storage.DiskStorage = > INFO: using /var/folders/7r/s0gg2qb11jz4g4pcb8n15gkc009j1y/T as the basep= ath for spooling. > Apr 14, 2017 1:16:53 PM com.datatorrent.bufferserver.server.Server regist= ered > INFO: Server started listening at /0:0:0:0:0:0:0:0:61042 > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster > INFO: Buffer server started: localhost:61042 > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-0 > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-1 > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-2 > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-3 > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-4 > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-3 msg: [container-3] Entering heartbeat loop.. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-2 msg: [container-2] Entering heartbeat loop.. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-1 msg: [container-1] Entering heartbeat loop.. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-4 msg: [container-4] Entering heartbeat loop.. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-0 msg: [container-0] Entering heartbeat loop.. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-5 > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-5 msg: [container-5] Entering heartbeat loop.. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-6 > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-7 > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-8 > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-7 msg: [container-7] Entering heartbeat loop.. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-6 msg: [container-6] Entering heartbeat loop.. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-8 msg: [container-8] Entering heartbeat loop.. > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$LocalStre= amingContainerLauncher run > INFO: Started container container-9 > Apr 14, 2017 1:16:53 PM com.datatorrent.stram.StramLocalCluster$Umbilical= ProtocolLocalImpl log > INFO: container-9 msg: [container-9] Entering heartbeat loop.. > Apr 14, 2017 1:16:54 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:16:54 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:16:54 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:16:54 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:16:54 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:16:54 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:16:54 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:16:54 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:16:54 PM com.datatorrent.stram.engine.StreamingContainer h= eartbeatLoop > INFO: Waiting for pending request. > Apr 14, 2017 1:16:54 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D4,name=3DFixedWindows/Wind= ow.Assign,type=3DGENERIC,checkpoint=3D{ffffffffffffffff, 0, 0},inputs=3D[Op= eratorDeployInfo.InputDeployInfo[portName=3DinputPort,streamId=3Dstream1,so= urceNodeId=3D3,sourcePortName=3Doutput,locality=3D,partitionMask=3D0,= partitionKeys=3D]],outputs=3D[OperatorDeployInfo.OutputDeployInfo[por= tName=3DoutputPort,streamId=3Dstream7,bufferServer=3Dlocalhost]]]] > Apr 14, 2017 1:16:54 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D4.outputPort.4, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:16:54 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61042/3.output.3, windowId=3Dffffffffffffffff, = type=3Dstream1/4.inputPort, upstreamIdentifier=3D3.output.3, mask=3D0, part= itions=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:16:55 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D3,name=3DSetTimestamps/Par= MultiDo(SetTimestamps),type=3DGENERIC,checkpoint=3D{ffffffffffffffff, 0, 0}= ,inputs=3D[OperatorDeployInfo.InputDeployInfo[portName=3Dinput,streamId=3Ds= tream2,sourceNodeId=3D2,sourcePortName=3Doutput,locality=3D,partition= Mask=3D0,partitionKeys=3D]],outputs=3D[OperatorDeployInfo.OutputDeplo= yInfo[portName=3Doutput,streamId=3Dstream1,bufferServer=3Dlocalhost]]]] > Apr 14, 2017 1:16:55 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D9,name=3DSumTeamScores/Gro= upByKey,type=3DGENERIC,checkpoint=3D{ffffffffffffffff, 0, 0},inputs=3D[Oper= atorDeployInfo.InputDeployInfo[portName=3Dinput,streamId=3Dstream8,sourceNo= deId=3D8,sourcePortName=3Doutput,locality=3D,partitionMask=3D0,partit= ionKeys=3D]],outputs=3D[OperatorDeployInfo.OutputDeployInfo[portName= =3Doutput,streamId=3Dstream3,bufferServer=3Dlocalhost]]]] > Apr 14, 2017 1:16:55 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D5,name=3DSumTeamScores/Par= Do(KeyScoreByTeam)/ParMultiDo(KeyScoreByTeam),type=3DGENERIC,checkpoint=3D{= ffffffffffffffff, 0, 0},inputs=3D[OperatorDeployInfo.InputDeployInfo[portNa= me=3Dinput,streamId=3Dstream7,sourceNodeId=3D4,sourcePortName=3DoutputPort,= locality=3D,partitionMask=3D0,partitionKeys=3D]],outputs=3D[Ope= ratorDeployInfo.OutputDeployInfo[portName=3Doutput,streamId=3Dstream4,buffe= rServer=3Dlocalhost]]]] > Apr 14, 2017 1:16:55 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D1,name=3DTextIO.Read/Read,= type=3DINPUT,checkpoint=3D{ffffffffffffffff, 0, 0},inputs=3D[],outputs=3D[O= peratorDeployInfo.OutputDeployInfo[portName=3Doutput,streamId=3Dstream6,buf= ferServer=3Dlocalhost]]]] > Apr 14, 2017 1:16:55 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D6,name=3DSumTeamScores/Com= bine.perKey(SumInteger)/GroupByKey,type=3DGENERIC,checkpoint=3D{fffffffffff= fffff, 0, 0},inputs=3D[OperatorDeployInfo.InputDeployInfo[portName=3Dinput,= streamId=3Dstream4,sourceNodeId=3D5,sourcePortName=3Doutput,locality=3D,partitionMask=3D0,partitionKeys=3D]],outputs=3D[OperatorDeployInfo= .OutputDeployInfo[portName=3Doutput,streamId=3Dstream0,bufferServer=3Dlocal= host]]]] > Apr 14, 2017 1:16:55 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D2,name=3DParseGameEvent/Pa= rMultiDo(ParseEvent),type=3DGENERIC,checkpoint=3D{ffffffffffffffff, 0, 0},i= nputs=3D[OperatorDeployInfo.InputDeployInfo[portName=3Dinput,streamId=3Dstr= eam6,sourceNodeId=3D1,sourcePortName=3Doutput,locality=3D,partitionMa= sk=3D0,partitionKeys=3D]],outputs=3D[OperatorDeployInfo.OutputDeployI= nfo[portName=3Doutput,streamId=3Dstream2,bufferServer=3Dlocalhost]]]] > Apr 14, 2017 1:16:55 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D7,name=3DSumTeamScores/Com= bine.perKey(SumInteger)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(A= nonymous),type=3DGENERIC,checkpoint=3D{ffffffffffffffff, 0, 0},inputs=3D[Op= eratorDeployInfo.InputDeployInfo[portName=3Dinput,streamId=3Dstream0,source= NodeId=3D6,sourcePortName=3Doutput,locality=3D,partitionMask=3D0,part= itionKeys=3D]],outputs=3D[OperatorDeployInfo.OutputDeployInfo[portNam= e=3Doutput,streamId=3Dstream5,bufferServer=3Dlocalhost]]]] > Apr 14, 2017 1:16:55 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D8,name=3DSumTeamScores/Par= Do(KeyByWindow)/ParMultiDo(KeyByWindow),type=3DGENERIC,checkpoint=3D{ffffff= ffffffffff, 0, 0},inputs=3D[OperatorDeployInfo.InputDeployInfo[portName=3Di= nput,streamId=3Dstream5,sourceNodeId=3D7,sourcePortName=3Doutput,locality= =3D,partitionMask=3D0,partitionKeys=3D]],outputs=3D[OperatorDep= loyInfo.OutputDeployInfo[portName=3Doutput,streamId=3Dstream8,bufferServer= =3Dlocalhost]]]] > Apr 14, 2017 1:16:55 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Deploy request: [OperatorDeployInfo[id=3D10,name=3DSumTeamScores/Pa= rDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles),type=3DGENERIC,check= point=3D{ffffffffffffffff, 0, 0},inputs=3D[OperatorDeployInfo.InputDeployIn= fo[portName=3Dinput,streamId=3Dstream3,sourceNodeId=3D9,sourcePortName=3Dou= tput,locality=3D,partitionMask=3D0,partitionKeys=3D]],outputs= =3D[]]] > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61042/2.output.2, windowId=3Dffffffffffffffff, = type=3Dstream2/3.input, upstreamIdentifier=3D2.output.2, mask=3D0, partitio= ns=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D3.output.3, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:16:55 PM com.datatorrent.stram.engine.WindowGenerator acti= vate > INFO: Catching up from 1492201012500 to 1492201015057 > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D9.output.9, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61042/8.output.8, windowId=3Dffffffffffffffff, = type=3Dstream8/9.input, upstreamIdentifier=3D8.output.8, mask=3D0, partitio= ns=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61042/1.output.1, windowId=3Dffffffffffffffff, = type=3Dstream6/2.input, upstreamIdentifier=3D1.output.1, mask=3D0, partitio= ns=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D2.output.2, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D6.output.6, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61042/5.output.5, windowId=3Dffffffffffffffff, = type=3Dstream4/6.input, upstreamIdentifier=3D5.output.5, mask=3D0, partitio= ns=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61042/7.output.7, windowId=3Dffffffffffffffff, = type=3Dstream5/8.input, upstreamIdentifier=3D7.output.7, mask=3D0, partitio= ns=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D8.output.8, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D1.output.1, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D5.output.5, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61042/4.outputPort.4, windowId=3Dffffffffffffff= ff, type=3Dstream7/5.input, upstreamIdentifier=3D4.outputPort.4, mask=3D0, = partitions=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61042/9.output.9, windowId=3Dffffffffffffffff, = type=3Dstream3/10.input, upstreamIdentifier=3D9.output.9, mask=3D0, partiti= ons=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received subscriber request: SubscribeRequestTuple{version=3D1.0, i= dentifier=3Dtcp://localhost:61042/6.output.6, windowId=3Dffffffffffffffff, = type=3Dstream0/7.input, upstreamIdentifier=3D6.output.6, mask=3D0, partitio= ns=3Dnull, bufferSize=3D0} > Apr 14, 2017 1:16:55 PM com.datatorrent.bufferserver.server.Server$Uniden= tifiedClient onMessage > INFO: Received publisher request: PublishRequestTuple{version=3D1.0, iden= tifier=3D7.output.7, windowId=3Dffffffffffffffff} > Apr 14, 2017 1:16:55 PM org.apache.beam.sdk.io.FileBasedSource expandFile= Pattern > INFO: Matched 1 files for pattern tiny.csv > Apr 14, 2017 1:16:55 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:56 PM com.datatorrent.stram.Journal write > WARNING: Journal output stream is null. Skipping write to the WAL. > Apr 14, 2017 1:16:58 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Undeploy request: [1] > Apr 14, 2017 1:16:58 PM com.datatorrent.stram.engine.StreamingContainer u= ndeploy > INFO: Undeploy complete. > Apr 14, 2017 1:17:01 PM com.datatorrent.stram.engine.StreamingContainer p= rocessHeartbeatResponse > INFO: Undeploy request: [2] > Apr 14, 2017 1:17:01 PM com.datatorrent.stram.engine.StreamingContainer u= ndeploy > INFO: Undeploy complete. > {code} > JSTACK: > {code} > =E2=9E=9C beam-portability-demo git:(apex) =E2=9C=97 jstack 84028 > 2017-04-14 13:19:55 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.112-b16 mixed mode= ): > "Attach Listener" #121 daemon prio=3D9 os_prio=3D31 tid=3D0x00007f8a0d800= 800 nid=3D0x4f0b waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "StorageHelper-2-1" #120 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0d21a000 n= id=3D0x1207 waiting on condition [0x0000700000801000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0ddba38> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.jav= a:403) > =09at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.= java:1067) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1127) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) > =09at java.lang.Thread.run(Thread.java:745) > "7/SumTeamScores/Combine.perKey(SumInteger)/Combine.GroupedValues/ParDo(A= nonymous)/ParMultiDo(Anonymous):ApexParDoOperator" #119 prio=3D5 os_prio=3D= 31 tid=3D0x00007f8a0b413000 nid=3D0x9b03 waiting on condition [0x0000700003= d4e000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > =09at java.lang.Thread.sleep(Native Method) > =09at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:601) > =09at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingCont= ainer.java:1407) > "10/SumTeamScores/ParDo(WriteWindowedFiles)/ParMultiDo(WriteWindowedFiles= ):ApexParDoOperator" #117 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0b3f9800 ni= d=3D0x9703 waiting on condition [0x0000700003b48000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > =09at java.lang.Thread.sleep(Native Method) > =09at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:601) > =09at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingCont= ainer.java:1407) > "5/SumTeamScores/ParDo(KeyScoreByTeam)/ParMultiDo(KeyScoreByTeam):ApexPar= DoOperator" #116 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0b3f9000 nid=3D0x950= 3 waiting on condition [0x0000700003a45000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > =09at java.lang.Thread.sleep(Native Method) > =09at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:601) > =09at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingCont= ainer.java:1407) > "8/SumTeamScores/ParDo(KeyByWindow)/ParMultiDo(KeyByWindow):ApexParDoOper= ator" #114 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0a7f7000 nid=3D0x9103 wait= ing on condition [0x000070000383f000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > =09at java.lang.Thread.sleep(Native Method) > =09at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:601) > =09at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingCont= ainer.java:1407) > "6/SumTeamScores/Combine.perKey(SumInteger)/GroupByKey:ApexGroupByKeyOper= ator" #113 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0abfc800 nid=3D0x8f03 wait= ing on condition [0x000070000373c000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > =09at java.lang.Thread.sleep(Native Method) > =09at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:601) > =09at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingCont= ainer.java:1407) > "9/SumTeamScores/GroupByKey:ApexGroupByKeyOperator" #111 prio=3D5 os_prio= =3D31 tid=3D0x00007f8a0a6ee800 nid=3D0x8b03 waiting on condition [0x0000700= 003536000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > =09at java.lang.Thread.sleep(Native Method) > =09at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:601) > =09at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingCont= ainer.java:1407) > "3/SetTimestamps/ParMultiDo(SetTimestamps):ApexParDoOperator" #110 prio= =3D5 os_prio=3D31 tid=3D0x00007f8a0aaff000 nid=3D0x8903 waiting on conditio= n [0x0000700003433000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > =09at java.lang.Thread.sleep(Native Method) > =09at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:601) > =09at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingCont= ainer.java:1407) > "ServerHelper-1-1" #109 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0bd25800 ni= d=3D0x8703 waiting on condition [0x0000700003330000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0fcc600> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.= java:1067) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1127) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) > =09at java.lang.Thread.run(Thread.java:745) > "4/FixedWindows/Window.Assign:ApexProcessFnOperator" #108 prio=3D5 os_pri= o=3D31 tid=3D0x00007f8a0cc85800 nid=3D0x8503 waiting on condition [0x000070= 000322d000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > =09at java.lang.Thread.sleep(Native Method) > =09at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:601) > =09at com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingCont= ainer.java:1407) > "pool-1-thread-1" #105 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0b314000 nid= =3D0x8303 waiting on condition [0x000070000312a000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0ce0130> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.= java:1067) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1127) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) > =09at java.lang.Thread.run(Thread.java:745) > "pool-1-thread-4" #107 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0ab46000 nid= =3D0x8103 waiting on condition [0x0000700003027000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0ce0130> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.= java:1067) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1127) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) > =09at java.lang.Thread.run(Thread.java:745) > "pool-1-thread-2" #106 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0ab10000 nid= =3D0x7f03 waiting on condition [0x0000700002f24000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0ce0130> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.= java:1067) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1127) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) > =09at java.lang.Thread.run(Thread.java:745) > "pool-1-thread-3" #104 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0bc4e800 nid= =3D0x7d03 waiting on condition [0x0000700002e21000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0ce0130> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.= java:1067) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1127) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) > =09at java.lang.Thread.run(Thread.java:745) > "container-9" #103 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0bc64000 nid=3D0= x7b03 in Object.wait() [0x0000700002d1e000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > =09at java.lang.Object.wait(Native Method) > =09at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(Strea= mingContainer.java:625) > =09- locked <0x00000006c0ce92e0> (a java.lang.Object) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainer.run= (StramLocalCluster.java:184) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLaun= cher.run(StramLocalCluster.java:270) > =09at java.lang.Thread.run(Thread.java:745) > "Dispatcher-9" #102 daemon prio=3D5 os_prio=3D31 tid=3D0x00007f8a0bd0c800= nid=3D0x7903 waiting on condition [0x0000700002c1b000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0cee530> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSync= AsyncMessageBus.java:51) > =09at java.lang.Thread.run(Thread.java:745) > "container-8" #101 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0b31b000 nid=3D0= x7703 in Object.wait() [0x0000700002b18000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > =09at java.lang.Object.wait(Native Method) > =09at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(Strea= mingContainer.java:625) > =09- locked <0x00000006c0d52250> (a java.lang.Object) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainer.run= (StramLocalCluster.java:184) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLaun= cher.run(StramLocalCluster.java:270) > =09at java.lang.Thread.run(Thread.java:745) > "Dispatcher-8" #100 daemon prio=3D5 os_prio=3D31 tid=3D0x00007f8a0ab44000= nid=3D0x7503 waiting on condition [0x0000700002a15000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0d36a88> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSync= AsyncMessageBus.java:51) > =09at java.lang.Thread.run(Thread.java:745) > "container-7" #99 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0ab43800 nid=3D0x= 7303 in Object.wait() [0x0000700002912000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > =09at java.lang.Object.wait(Native Method) > =09at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(Strea= mingContainer.java:625) > =09- locked <0x00000006c0d07a98> (a java.lang.Object) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainer.run= (StramLocalCluster.java:184) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLaun= cher.run(StramLocalCluster.java:270) > =09at java.lang.Thread.run(Thread.java:745) > "Dispatcher-7" #98 daemon prio=3D5 os_prio=3D31 tid=3D0x00007f8a0ab42800 = nid=3D0x7103 waiting on condition [0x000070000280f000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0d0ab08> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSync= AsyncMessageBus.java:51) > =09at java.lang.Thread.run(Thread.java:745) > "container-6" #97 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0ab40000 nid=3D0x= 6f03 in Object.wait() [0x000070000270c000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > =09at java.lang.Object.wait(Native Method) > =09at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(Strea= mingContainer.java:625) > =09- locked <0x00000006c0d77790> (a java.lang.Object) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainer.run= (StramLocalCluster.java:184) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLaun= cher.run(StramLocalCluster.java:270) > =09at java.lang.Thread.run(Thread.java:745) > "Dispatcher-6" #96 daemon prio=3D5 os_prio=3D31 tid=3D0x00007f8a0ab3f000 = nid=3D0x6d03 waiting on condition [0x0000700002609000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0d67200> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSync= AsyncMessageBus.java:51) > =09at java.lang.Thread.run(Thread.java:745) > "container-5" #95 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0a5c5800 nid=3D0x= 6b03 in Object.wait() [0x0000700002506000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > =09at java.lang.Object.wait(Native Method) > =09at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(Strea= mingContainer.java:625) > =09- locked <0x00000006c0dd51f0> (a java.lang.Object) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainer.run= (StramLocalCluster.java:184) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLaun= cher.run(StramLocalCluster.java:270) > =09at java.lang.Thread.run(Thread.java:745) > "Dispatcher-5" #94 daemon prio=3D5 os_prio=3D31 tid=3D0x00007f8a0a565800 = nid=3D0x6903 waiting on condition [0x0000700002403000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0f21998> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSync= AsyncMessageBus.java:51) > =09at java.lang.Thread.run(Thread.java:745) > "container-4" #93 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0b40e800 nid=3D0x= 6703 in Object.wait() [0x0000700002300000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > =09at java.lang.Object.wait(Native Method) > =09at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(Strea= mingContainer.java:625) > =09- locked <0x00000006c0f5d0b8> (a java.lang.Object) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainer.run= (StramLocalCluster.java:184) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLaun= cher.run(StramLocalCluster.java:270) > =09at java.lang.Thread.run(Thread.java:745) > "Dispatcher-4" #92 daemon prio=3D5 os_prio=3D31 tid=3D0x00007f8a0b3c6000 = nid=3D0x6503 waiting on condition [0x00007000021fd000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0f60db8> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSync= AsyncMessageBus.java:51) > =09at java.lang.Thread.run(Thread.java:745) > "container-3" #91 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0ab4b800 nid=3D0x= 6303 in Object.wait() [0x00007000020fa000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > =09at java.lang.Object.wait(Native Method) > =09at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(Strea= mingContainer.java:625) > =09- locked <0x00000006c0f96a08> (a java.lang.Object) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainer.run= (StramLocalCluster.java:184) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLaun= cher.run(StramLocalCluster.java:270) > =09at java.lang.Thread.run(Thread.java:745) > "Dispatcher-3" #90 daemon prio=3D5 os_prio=3D31 tid=3D0x00007f8a0b3cc800 = nid=3D0x6103 waiting on condition [0x0000700001ff7000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0f9afe0> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSync= AsyncMessageBus.java:51) > =09at java.lang.Thread.run(Thread.java:745) > "container-2" #89 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0b3cc000 nid=3D0x= 5f03 in Object.wait() [0x0000700001ef4000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > =09at java.lang.Object.wait(Native Method) > =09at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(Strea= mingContainer.java:625) > =09- locked <0x00000006c0fc44b8> (a java.lang.Object) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainer.run= (StramLocalCluster.java:184) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLaun= cher.run(StramLocalCluster.java:270) > =09at java.lang.Thread.run(Thread.java:745) > "Dispatcher-2" #88 daemon prio=3D5 os_prio=3D31 tid=3D0x00007f8a0ab4b000 = nid=3D0x5d03 waiting on condition [0x0000700001df1000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c0fcb2d0> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSync= AsyncMessageBus.java:51) > =09at java.lang.Thread.run(Thread.java:745) > "container-1" #87 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0ab11800 nid=3D0x= 5b03 in Object.wait() [0x0000700001cee000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > =09at java.lang.Object.wait(Native Method) > =09at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(Strea= mingContainer.java:625) > =09- locked <0x00000006c12df300> (a java.lang.Object) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainer.run= (StramLocalCluster.java:184) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLaun= cher.run(StramLocalCluster.java:270) > =09at java.lang.Thread.run(Thread.java:745) > "Dispatcher-1" #86 daemon prio=3D5 os_prio=3D31 tid=3D0x00007f8a0b3cb000 = nid=3D0x5903 waiting on condition [0x0000700001beb000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c1127b08> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSync= AsyncMessageBus.java:51) > =09at java.lang.Thread.run(Thread.java:745) > "container-0" #85 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0bbda000 nid=3D0x= 5703 in Object.wait() [0x0000700001ae8000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > =09at java.lang.Object.wait(Native Method) > =09at com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(Strea= mingContainer.java:625) > =09- locked <0x00000006c06e2d20> (a java.lang.Object) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainer.run= (StramLocalCluster.java:184) > =09at com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLaun= cher.run(StramLocalCluster.java:270) > =09at java.lang.Thread.run(Thread.java:745) > "Dispatcher-0" #84 daemon prio=3D5 os_prio=3D31 tid=3D0x00007f8a0b323000 = nid=3D0x5503 waiting on condition [0x00007000019e5000] > java.lang.Thread.State: WAITING (parking) > =09at sun.misc.Unsafe.park(Native Method) > =09- parking to wait for <0x00000006c112abb8> (a java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject) > =09at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > =09at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObje= ct.await(AbstractQueuedSynchronizer.java:2039) > =09at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.j= ava:442) > =09at net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSync= AsyncMessageBus.java:51) > =09at java.lang.Thread.run(Thread.java:745) > "DestroyJavaVM" #83 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0bc3f800 nid=3D= 0x1c03 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "master" #82 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0ab4c800 nid=3D0x5303 = waiting on condition [0x00007000018e2000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > =09at java.lang.Thread.sleep(Native Method) > =09at com.datatorrent.stram.StramLocalCluster.run(StramLocalCluster.java:= 499) > =09at com.datatorrent.stram.StramLocalCluster.run(StramLocalCluster.java:= 438) > =09at java.lang.Thread.run(Thread.java:745) > "ProcessWideEventLoop" #81 prio=3D5 os_prio=3D31 tid=3D0x00007f8a0ab4f800= nid=3D0x511b runnable [0x00007000017df000] > java.lang.Thread.State: RUNNABLE > =09at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) > =09at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) > =09at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) > =09at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > =09- locked <0x00000006c0cd6f40> (a com.datatorrent.netlet.OptimizedEvent= Loop$SelectedSelectionKeySet) > =09- locked <0x00000006c09a6fb8> (a java.util.Collections$UnmodifiableSet= ) > =09- locked <0x00000006c0cd6e90> (a sun.nio.ch.KQueueSelectorImpl) > =09at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > =09at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) > =09at com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEve= ntLoop.java:185) > =09at com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEve= ntLoop.java:157) > =09at com.datatorrent.netlet.DefaultEventLoop.run(DefaultEventLoop.java:1= 56) > =09at java.lang.Thread.run(Thread.java:745) > "Service Thread" #8 daemon prio=3D9 os_prio=3D31 tid=3D0x00007f8a0a061000= nid=3D0x4803 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C1 CompilerThread2" #7 daemon prio=3D9 os_prio=3D31 tid=3D0x00007f8a0a83= 8800 nid=3D0x4603 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" #6 daemon prio=3D9 os_prio=3D31 tid=3D0x00007f8a0a05= 6000 nid=3D0x4403 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" #5 daemon prio=3D9 os_prio=3D31 tid=3D0x00007f8a0a06= c000 nid=3D0x4203 waiting on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" #4 daemon prio=3D9 os_prio=3D31 tid=3D0x00007f8a0b02c= 000 nid=3D0x4003 runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" #3 daemon prio=3D8 os_prio=3D31 tid=3D0x00007f8a0a835800 nid= =3D0x3103 in Object.wait() [0x000070000109f000] > java.lang.Thread.State: WAITING (on object monitor) > =09at java.lang.Object.wait(Native Method) > =09- waiting on <0x00000006c0005fe0> (a java.lang.ref.ReferenceQueue$Lock= ) > =09at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) > =09- locked <0x00000006c0005fe0> (a java.lang.ref.ReferenceQueue$Lock) > =09at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) > =09at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" #2 daemon prio=3D10 os_prio=3D31 tid=3D0x00007f8a0b80= 4000 nid=3D0x2f03 in Object.wait() [0x0000700000f9c000] > java.lang.Thread.State: WAITING (on object monitor) > =09at java.lang.Object.wait(Native Method) > =09- waiting on <0x00000006c000d548> (a java.lang.ref.Reference$Lock) > =09at java.lang.Object.wait(Object.java:502) > =09at java.lang.ref.Reference.tryHandlePending(Reference.java:191) > =09- locked <0x00000006c000d548> (a java.lang.ref.Reference$Lock) > =09at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > "VM Thread" os_prio=3D31 tid=3D0x00007f8a0b801800 nid=3D0x2d03 runnable= =20 > "GC task thread#0 (ParallelGC)" os_prio=3D31 tid=3D0x00007f8a0b000800 nid= =3D0x2503 runnable=20 > "GC task thread#1 (ParallelGC)" os_prio=3D31 tid=3D0x00007f8a0b800800 nid= =3D0x2703 runnable=20 > "GC task thread#2 (ParallelGC)" os_prio=3D31 tid=3D0x00007f8a0a015000 nid= =3D0x2903 runnable=20 > "GC task thread#3 (ParallelGC)" os_prio=3D31 tid=3D0x00007f8a0a015800 nid= =3D0x2b03 runnable=20 > "VM Periodic Task Thread" os_prio=3D31 tid=3D0x00007f8a0a09e000 nid=3D0x4= a03 waiting on condition=20 > JNI global references: 332 > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)