Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id AA4CC200AF7 for ; Tue, 14 Jun 2016 18:36:42 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id A8CAE160A47; Tue, 14 Jun 2016 16:36:42 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7CAFA160A06 for ; Tue, 14 Jun 2016 18:36:41 +0200 (CEST) Received: (qmail 14558 invoked by uid 500); 14 Jun 2016 16:36:40 -0000 Mailing-List: contact users-help@apex.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@apex.apache.org Delivered-To: mailing list users@apex.apache.org Received: (qmail 14549 invoked by uid 99); 14 Jun 2016 16:36:40 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2016 16:36:40 +0000 Received: from mail-io0-f175.google.com (mail-io0-f175.google.com [209.85.223.175]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 629D51A0247 for ; Tue, 14 Jun 2016 16:36:40 +0000 (UTC) Received: by mail-io0-f175.google.com with SMTP id o127so106497116iod.2 for ; Tue, 14 Jun 2016 09:36:40 -0700 (PDT) X-Gm-Message-State: ALyK8tIgneUCfvOeqUsxWPG3xphjVYOJiagVSSAM1S6U7ZIaqtcMxeN88dRzHymxIqp6/BXdO/5fKcYQDiTZ0Z3+ X-Received: by 10.107.4.78 with SMTP id 75mr35796197ioe.52.1465922198560; Tue, 14 Jun 2016 09:36:38 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.12.215 with HTTP; Tue, 14 Jun 2016 09:36:37 -0700 (PDT) In-Reply-To: References: From: "Pradeep A. Dalvi" Date: Tue, 14 Jun 2016 09:36:37 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Containers Not getting Allocated. To: users@apex.apache.org Content-Type: multipart/alternative; boundary=001a113edf60ea136f05353f9b5d archived-at: Tue, 14 Jun 2016 16:36:42 -0000 --001a113edf60ea136f05353f9b5d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable From logs, it appears that container requests are being resubmitted. Doubt that it is related to APEXCORE-471. How much Total Memory is shown in DT or YARN UI? On Tue, Jun 14, 2016 at 8:22 AM, Sandesh Hegde wrote: > There is one known issue, > > https://issues.apache.org/jira/browse/APEXCORE-471 > > > On Tue, Jun 14, 2016 at 7:58 AM Munagala Ramanath > wrote: > >> Could you take a look at the YARN logs ? >> >> Ram >> >> On Tue, Jun 14, 2016 at 5:43 AM, Shubham Pathak >> wrote: >> >>> Hello, >>> >>> I have setup a 3 node hadoop cluster using Cloudera Express 5.7.0. and >>> installed Dt RTS community edition datatorrent-rts-3.3.0.bin. >>> Total memory available is 24 GB. >>> >>> However when I launch PiDemo, containers are not getting allocated. Jus= t >>> 2 / 3 containers are being allocated. This is error in Stram logs. >>> >>> 2016-06-14 17:41:48,762 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>> Capability[]Priority[2] >>> 2016-06-14 17:41:48,762 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containe= rs: >>> [Capability[]Priority[2]] >>> 2016-06-14 17:41:48,762 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Requested container: >>> Capability[]Priority[2] >>> 2016-06-14 17:42:20,185 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removing container >>> request: [Capability[]Priority[2]] >>> 2016-06-14 17:42:20,185 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>> Capability[]Priority[2] >>> 2016-06-14 17:42:20,185 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containe= rs: >>> [Capability[]Priority[2]] >>> 2016-06-14 17:42:20,185 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Requested container: >>> Capability[]Priority[2] >>> 2016-06-14 17:42:35,822 WARN >>> com.datatorrent.common.util.PubSubWebSocketClient: Ignoring unparseable >>> JSON message: =EF=BF=BDIdle for 300289ms > 300000ms >>> org.codehaus.jackson.JsonParseException: Illegal character ((CTRL-CHAR, >>> code 3)): only regular white space (\r, \n, \t) is allowed between toke= ns >>> at [Source: java.io.StringReader@7ab8225e; line: 1, column: 2] >>> at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1432= ) >>> at >>> org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParser= MinimalBase.java:385) >>> at >>> org.codehaus.jackson.impl.JsonParserMinimalBase._throwInvalidSpace(Json= ParserMinimalBase.java:331) >>> at >>> org.codehaus.jackson.impl.ReaderBasedParser._skipWSOrEnd(ReaderBasedPar= ser.java:1485) >>> at >>> org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser= .java:362) >>> at >>> org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java= :2761) >>> at >>> org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.jav= a:2709) >>> at >>> org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1854) >>> at >>> com.datatorrent.common.util.PubSubMessageCodec.parseMessage(PubSubMessa= geCodec.java:66) >>> at >>> com.datatorrent.common.util.PubSubWebSocketClient$PubSubWebSocket.onMes= sage(PubSubWebSocketClient.java:79) >>> at >>> com.ning.http.client.providers.netty.NettyWebSocket.onTextFragment(Nett= yWebSocket.java:183) >>> at >>> com.ning.http.client.providers.netty.NettyAsyncHttpProvider$WebSocketPr= otocol.handle(NettyAsyncHttpProvider.java:2369) >>> at >>> com.ning.http.client.providers.netty.NettyAsyncHttpProvider.messageRece= ived(NettyAsyncHttpProvider.java:1142) >>> at >>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(Sim= pleChannelUpstreamHandler.java:70) >>> at >>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChan= nelPipeline.java:560) >>> at >>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerCon= text.sendUpstream(DefaultChannelPipeline.java:787) >>> at >>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) >>> at >>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageRe= ceived(FrameDecoder.java:462) >>> at >>> org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(Replay= ingDecoder.java:536) >>> at >>> org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(R= eplayingDecoder.java:435) >>> at >>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(Sim= pleChannelUpstreamHandler.java:70) >>> at >>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChan= nelPipeline.java:560) >>> at >>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChan= nelPipeline.java:555) >>> at >>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) >>> at >>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) >>> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) >>> at >>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNi= oWorker.java:107) >>> at >>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioS= elector.java:312) >>> at >>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWor= ker.java:88) >>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) >>> at >>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.= java:108) >>> at >>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWo= rker.java:42) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja= va:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j= ava:615) >>> at java.lang.Thread.run(Thread.java:745) >>> 2016-06-14 17:42:51,522 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removing container >>> request: [Capability[]Priority[2]] >>> 2016-06-14 17:42:51,523 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>> Capability[]Priority[2] >>> 2016-06-14 17:42:51,523 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containe= rs: >>> [Capability[]Priority[2]] >>> 2016-06-14 17:42:51,523 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Requested container: >>> Capability[]Priority[2] >>> 2016-06-14 17:43:22,829 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removing container >>> request: [Capability[]Priority[2]] >>> 2016-06-14 17:43:22,830 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>> Capability[]Priority[2] >>> 2016-06-14 17:43:22,830 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containe= rs: >>> [Capability[]Priority[2]] >>> 2016-06-14 17:43:22,830 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Requested container: >>> Capability[]Priority[2] >>> 2016-06-14 17:43:54,218 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removing container >>> request: [Capability[]Priority[2]] >>> 2016-06-14 17:43:54,219 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>> Capability[]Priority[2] >>> 2016-06-14 17:43:54,219 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containe= rs: >>> [Capability[]Priority[2]] >>> 2016-06-14 17:43:54,219 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Requested container: >>> Capability[]Priority[2] >>> 2016-06-14 17:44:25,543 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removing container >>> request: [Capability[]Priority[2]] >>> 2016-06-14 17:44:25,543 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>> Capability[]Priority[2] >>> 2016-06-14 17:44:25,543 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containe= rs: >>> [Capability[]Priority[2]] >>> 2016-06-14 17:44:25,543 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Requested container: >>> Capability[]Priority[2] >>> 2016-06-14 17:44:56,885 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removing container >>> request: [Capability[]Priority[2]] >>> 2016-06-14 17:44:56,886 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Removed container: >>> Capability[]Priority[2] >>> 2016-06-14 17:44:56,886 INFO >>> com.datatorrent.stram.StreamingAppMasterService: Asking RM for containe= rs: >>> [Capability[]Priority[2]] >>> >>> >>> Any idea what could be the issue ? >>> >>> Thanks, >>> Shubham >>> >> >> --001a113edf60ea136f05353f9b5d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
From logs, it appears that container requests are being re= submitted. Doubt that it is related to APEXCORE-471.

How= much Total Memory is shown in DT or YARN UI?

On Tue, Jun 14, 2016 at 8:22 AM, Sa= ndesh Hegde <sandesh@datatorrent.com> wrote:
There is one known issue,=C2=A0

<= br>
Could you take a look at the YARN logs ?

<= /div>
Ram

On Tue, Jun 14, 2016 at 5:43 AM, Shubham Pathak &= lt;shubham@dat= atorrent.com> wrote:
Hello,

I have setup a 3 node hadoop cluster u= sing Cloudera Express 5.7.0. and installed Dt RTS community edition datator= rent-rts-3.3.0.bin.=C2=A0
Total memory available is 24 GB.
<= div>
However when I launch PiDemo, containers are not getting= allocated. Just 2 / 3 containers are being allocated. This is error in Str= am logs.

2016-06-14 17:41:48,762 INFO com.dat= atorrent.stram.StreamingAppMasterService: Removed container: Capability[<= ;memory:1024, vCores:1>]Priority[2]
2016-06-14 17:41:48,762 IN= FO com.datatorrent.stram.StreamingAppMasterService: Asking RM for container= s: [Capability[<memory:1024, vCores:1>]Priority[2]]
2016-06= -14 17:41:48,762 INFO com.datatorrent.stram.StreamingAppMasterService: Requ= ested container: Capability[<memory:1024, vCores:1>]Priority[2]
=
2016-06-14 17:42:20,185 INFO com.datatorrent.stram.StreamingAppMasterS= ervice: =C2=A0Removing container request: [Capability[<memory:1024, vCor= es:1>]Priority[2]]
2016-06-14 17:42:20,185 INFO com.datatorren= t.stram.StreamingAppMasterService: Removed container: Capability[<memory= :1024, vCores:1>]Priority[2]
2016-06-14 17:42:20,185 INFO com.= datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Cap= ability[<memory:1024, vCores:1>]Priority[2]]
2016-06-14 17:= 42:20,185 INFO com.datatorrent.stram.StreamingAppMasterService: Requested c= ontainer: Capability[<memory:1024, vCores:1>]Priority[2]
20= 16-06-14 17:42:35,822 WARN com.datatorrent.common.util.PubSubWebSocketClien= t: Ignoring unparseable JSON message: =EF=BF=BDIdle for 300289ms > 3000= 00ms
org.codehaus.jackson.JsonParseException: Illegal character (= (CTRL-CHAR, code 3)): only regular white space (\r, \n, \t) is allowed betw= een tokens
=C2=A0at [Source: java.io.StringReader@7ab8225e; line:= 1, column: 2]
at or= g.codehaus.jackson.JsonParser._constructError(JsonParser.java:1432)
at org.codehaus.jackson.imp= l.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
<= div> at org.codehaus.jackson.im= pl.JsonParserMinimalBase._throwInvalidSpace(JsonParserMinimalBase.java:331)=
at org.codehaus.jac= kson.impl.ReaderBasedParser._skipWSOrEnd(ReaderBasedParser.java:1485)
=
at org.codehaus.jackson.i= mpl.ReaderBasedParser.nextToken(ReaderBasedParser.java:362)
at org.codehaus.jackson.map.ObjectM= apper._initForReading(ObjectMapper.java:2761)
at org.codehaus.jackson.map.ObjectMapper._readMap= AndClose(ObjectMapper.java:2709)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMappe= r.java:1854)
at com.= datatorrent.common.util.PubSubMessageCodec.parseMessage(PubSubMessageCodec.= java:66)
at com.data= torrent.common.util.PubSubWebSocketClient$PubSubWebSocket.onMessage(PubSubW= ebSocketClient.java:79)
at com.ning.http.client.providers.netty.NettyWebSocket.onTextFragment(N= ettyWebSocket.java:183)
at com.ning.http.client.providers.netty.NettyAsyncHttpProvider$WebSocke= tProtocol.handle(NettyAsyncHttpProvider.java:2369)
at com.ning.http.client.providers.netty.Nett= yAsyncHttpProvider.messageReceived(NettyAsyncHttpProvider.java:1142)
<= div> at org.jboss.netty.channel= .SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.j= ava:70)
at org.jboss= .netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.j= ava:560)
at org.jbos= s.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUps= tream(DefaultChannelPipeline.java:787)
at org.jboss.netty.channel.Channels.fireMessageReceived(= Channels.java:296)
a= t org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageRece= ived(FrameDecoder.java:462)
= at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode= (ReplayingDecoder.java:536)
= at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageRec= eived(ReplayingDecoder.java:435)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handle= Upstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipelin= e.sendUpstream(DefaultChannelPipeline.java:560)
at org.jboss.netty.channel.DefaultChannelPipeli= ne.sendUpstream(DefaultChannelPipeline.java:555)
at org.jboss.netty.channel.Channels.fireMessag= eReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.j= ava:255)
at org.jbos= s.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.= nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at org.jboss.netty.channel.socket.nio= .AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.= run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnab= le.java:108)
at org.= jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.jav= a:42)
at java.util.c= oncurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
<= div> at java.util.concurrent.Th= readPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:74= 5)
2016-06-14 17:42:51,522 INFO com.datatorrent.stram.StreamingAp= pMasterService: =C2=A0Removing container request: [Capability[<memory:10= 24, vCores:1>]Priority[2]]
2016-06-14 17:42:51,523 INFO com.da= tatorrent.stram.StreamingAppMasterService: Removed container: Capability[&l= t;memory:1024, vCores:1>]Priority[2]
2016-06-14 17:42:51,523 I= NFO com.datatorrent.stram.StreamingAppMasterService: Asking RM for containe= rs: [Capability[<memory:1024, vCores:1>]Priority[2]]
2016-0= 6-14 17:42:51,523 INFO com.datatorrent.stram.StreamingAppMasterService: Req= uested container: Capability[<memory:1024, vCores:1>]Priority[2]
2016-06-14 17:43:22,829 INFO com.datatorrent.stram.StreamingAppMaster= Service: =C2=A0Removing container request: [Capability[<memory:1024, vCo= res:1>]Priority[2]]
2016-06-14 17:43:22,830 INFO com.datatorre= nt.stram.StreamingAppMasterService: Removed container: Capability[<memor= y:1024, vCores:1>]Priority[2]
2016-06-14 17:43:22,830 INFO com= .datatorrent.stram.StreamingAppMasterService: Asking RM for containers: [Ca= pability[<memory:1024, vCores:1>]Priority[2]]
2016-06-14 17= :43:22,830 INFO com.datatorrent.stram.StreamingAppMasterService: Requested = container: Capability[<memory:1024, vCores:1>]Priority[2]
2= 016-06-14 17:43:54,218 INFO com.datatorrent.stram.StreamingAppMasterService= : =C2=A0Removing container request: [Capability[<memory:1024, vCores:1&g= t;]Priority[2]]
2016-06-14 17:43:54,219 INFO com.datatorrent.stra= m.StreamingAppMasterService: Removed container: Capability[<memory:1024,= vCores:1>]Priority[2]
2016-06-14 17:43:54,219 INFO com.datato= rrent.stram.StreamingAppMasterService: Asking RM for containers: [Capabilit= y[<memory:1024, vCores:1>]Priority[2]]
2016-06-14 17:43:54,= 219 INFO com.datatorrent.stram.StreamingAppMasterService: Requested contain= er: Capability[<memory:1024, vCores:1>]Priority[2]
2016-06-= 14 17:44:25,543 INFO com.datatorrent.stram.StreamingAppMasterService: =C2= =A0Removing container request: [Capability[<memory:1024, vCores:1>]Pr= iority[2]]
2016-06-14 17:44:25,543 INFO com.datatorrent.stram.Str= eamingAppMasterService: Removed container: Capability[<memory:1024, vCor= es:1>]Priority[2]
2016-06-14 17:44:25,543 INFO com.datatorrent= .stram.StreamingAppMasterService: Asking RM for containers: [Capability[<= ;memory:1024, vCores:1>]Priority[2]]
2016-06-14 17:44:25,543 I= NFO com.datatorrent.stram.StreamingAppMasterService: Requested container: C= apability[<memory:1024, vCores:1>]Priority[2]
2016-06-14 17= :44:56,885 INFO com.datatorrent.stram.StreamingAppMasterService: =C2=A0Remo= ving container request: [Capability[<memory:1024, vCores:1>]Priority[= 2]]
2016-06-14 17:44:56,886 INFO com.datatorrent.stram.StreamingA= ppMasterService: Removed container: Capability[<memory:1024, vCores:1>= ;]Priority[2]
2016-06-14 17:44:56,886 INFO com.datatorrent.stram.= StreamingAppMasterService: Asking RM for containers: [Capability[<memory= :1024, vCores:1>]Priority[2]]


<= div>Any idea what could be the issue ?=C2=A0

Thank= s,
Shubham


--001a113edf60ea136f05353f9b5d--