Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 1713C200BB6 for ; Fri, 21 Oct 2016 03:48:16 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 0B122160AF2; Fri, 21 Oct 2016 01:48:16 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 82F10160AE0 for ; Fri, 21 Oct 2016 03:48:14 +0200 (CEST) Received: (qmail 19381 invoked by uid 500); 21 Oct 2016 01:48:13 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 19371 invoked by uid 99); 21 Oct 2016 01:48:13 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Oct 2016 01:48:13 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 1B903C1B01 for ; Fri, 21 Oct 2016 01:48:13 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.629 X-Spam-Level: ** X-Spam-Status: No, score=2.629 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id PyTswpZhZckK for ; Fri, 21 Oct 2016 01:48:11 +0000 (UTC) Received: from mail-vk0-f51.google.com (mail-vk0-f51.google.com [209.85.213.51]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 690645F59E for ; Fri, 21 Oct 2016 01:48:11 +0000 (UTC) Received: by mail-vk0-f51.google.com with SMTP id b186so98297492vkb.1 for ; Thu, 20 Oct 2016 18:48:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=uJi87dQ/sD7PoOVJNt3UwYiNpnNEUnrCiwUo37lWF28=; b=K4UXZ4UNF7Ag+rjuFWxY630D/I6iBtNO7Mf485Xxe+GqEKQyt7Yqj0ObkJglVjgN6E b2/EJmEoK+wBjmkXJoQNFccVEb5AaUAGFagTVmDXI7VZoesM9CvqmeQnybIOtHhkup70 QZ16S1yi3+fj06B3nX4FFbi8dOMn9HIQnvTVUPHMnl1adpqqSJT185sYRONZ1jZ/sgor Pp5luqLr5Ci9gIeylf1zuzMIeRYPKbsu9qrQeD2z68QEV4Dw/RLIKd+yxTmbArInvmz5 rZenaN8QCw4QESI2UimPUVoP9itzFqGwMK/WTzVVaUG/VQtaicNKlzHyaiE+TORAOr1J h+HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=uJi87dQ/sD7PoOVJNt3UwYiNpnNEUnrCiwUo37lWF28=; b=SVDDELScp+7W+nUJlJle43WdJdHCXnURvx7r8cc7C+f26DaTgBGgw7Tk7yvR32Fb/1 un9zbmRnOh0fl4C0N/uybKSTXRxPczrEG49mvOWVvnTsMa2QGDPyLz799GalkgqsVdht FtY/ZpSBc+mfA0U2Kq+aSHQfli+yB1ZXj+a4z51ojelY+IIHQmNVk9f/8GLOQDxeDMFC TnR/0PHn0q58dtDP7FRp2Aub1QA2ITHEMAe2OWvaEgf8QlonXumsSMTsMbYkdK4fPobj wPywneIDBfTal8bMLUyyOvCezMhuvcAx6useSooonAW9dzrRK+wkUtTUosycKP4gpvBu kc1g== X-Gm-Message-State: AA6/9RnrTBGdWxZsAgEb/Vq3huCcGffjIq/PHyWojVg/TE/edWoe/fv2xb8f1Tl1H5U4xL7RKCcTB1S6r57oUw== X-Received: by 10.31.152.195 with SMTP id a186mr4879478vke.79.1477014490544; Thu, 20 Oct 2016 18:48:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.103.119.12 with HTTP; Thu, 20 Oct 2016 18:48:10 -0700 (PDT) From: Hai Lan Date: Thu, 20 Oct 2016 21:48:10 -0400 Message-ID: Subject: OutOfMemoryError: Java heap space during Large graph running To: user@giraph.apache.org Content-Type: multipart/alternative; boundary=001a1141bf42098127053f563c48 archived-at: Fri, 21 Oct 2016 01:48:16 -0000 --001a1141bf42098127053f563c48 Content-Type: text/plain; charset=UTF-8 Dear all, I'm facing a problem when I run large graph job (currently 1.6T, will be 16T then), it always shows java.lang.OutOfMemoryError: Java heap space error when loaded specific numbers of vertex(near 59000000). I tried to add like: -Dgiraph.useOutOfCoreGraph=true -Dmapred.child.java.opts="-XX:-UseGCOverheadLimit" OR -Dmapred.child.java.opts="-Xmx16384" -Dgiraph.yarn.task.heap.mb=36570 but the problem remain though I can see those value are shown in Metadata. I'm not sure the max value of memory in this VertexInputSplitsCallable info is related to java heap size. INFO [load-0] org.apache.giraph.worker.VertexInputSplitsCallable: readVertexInputSplit: Loaded 46975802 vertices at 68977.49310291892 vertices/sec 0 edges at 0.0 edges/sec Memory (free/total/max) = 475.08M / 2759.00M / 2759.00M But I am noticed in main log, it *always* shows: INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx2868m *no matter what arguments I added*. Even when I run normal Hadoop jobs. Any ideas about this? Following is the log. 2016-10-20 21:25:49,008 ERROR [netty-client-worker-2] org.apache.giraph.comm.netty.NettyClient: Request failed java.lang.OutOfMemoryError: Java heap space at io.netty.buffer.UnpooledHeapByteBuf.(UnpooledHeapByteBuf.java:45) at io.netty.buffer.UnpooledByteBufAllocator.newHeapBuffer(UnpooledByteBufAllocator.java:43) at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:136) at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:127) at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:85) at org.apache.giraph.comm.netty.handler.RequestEncoder.write(RequestEncoder.java:81) at io.netty.channel.DefaultChannelHandlerContext.invokeWrite(DefaultChannelHandlerContext.java:645) at io.netty.channel.DefaultChannelHandlerContext.access$2000(DefaultChannelHandlerContext.java:29) at io.netty.channel.DefaultChannelHandlerContext$WriteTask.run(DefaultChannelHandlerContext.java:906) at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:36) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) at java.lang.Thread.run(Thread.java:745) 2016-10-20 21:25:55,299 ERROR [netty-client-worker-1] org.apache.giraph.comm.netty.NettyClient: Request failed java.lang.OutOfMemoryError: Java heap space at io.netty.buffer.UnpooledHeapByteBuf.(UnpooledHeapByteBuf.java:45) at io.netty.buffer.UnpooledByteBufAllocator.newHeapBuffer(UnpooledByteBufAllocator.java:43) at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:136) at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:127) at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:85) at org.apache.giraph.comm.netty.handler.RequestEncoder.write(RequestEncoder.java:81) at io.netty.channel.DefaultChannelHandlerContext.invokeWrite(DefaultChannelHandlerContext.java:645) at io.netty.channel.DefaultChannelHandlerContext.access$2000(DefaultChannelHandlerContext.java:29) at io.netty.channel.DefaultChannelHandlerContext$WriteTask.run(DefaultChannelHandlerContext.java:906) at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:36) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) at java.lang.Thread.run(Thread.java:745) 2016-10-20 21:26:06,731 ERROR [main] org.apache.giraph.graph.GraphMapper: Caught an unrecoverable exception waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@6737a445 java.lang.IllegalStateException: waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@6737a445 at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:193) at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:151) at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:136) at org.apache.giraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:99) at org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:233) at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(BspServiceWorker.java:316) at org.apache.giraph.worker.BspServiceWorker.loadVertices(BspServiceWorker.java:409) at org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:629) at org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:284) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:93) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:202) at org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:312) at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:185) ... 16 more Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.giraph.utils.UnsafeByteArrayOutputStream.(UnsafeByteArrayOutputStream.java:81) at org.apache.giraph.conf.ImmutableClassesGiraphConfiguration.createExtendedDataOutput(ImmutableClassesGiraphConfiguration.java:1161) at org.apache.giraph.comm.SendPartitionCache.addVertex(SendPartitionCache.java:77) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.sendVertexRequest(NettyWorkerClientRequestProcessor.java:248) at org.apache.giraph.worker.VertexInputSplitsCallable.readInputSplit(VertexInputSplitsCallable.java:231) at org.apache.giraph.worker.InputSplitsCallable.loadInputSplit(InputSplitsCallable.java:267) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:211) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:60) at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-10-20 21:26:06,737 ERROR [main] org.apache.giraph.worker.BspServiceWorker: unregisterHealth: Got failure, unregistering health on /_hadoopBsp/job_1476386340018_0175/_applicationAttemptsDir/0/_superstepDir/-1/_workerHealthyDir/hadoop18.umd.com_23 on superstep -1 2016-10-20 21:26:06,746 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalStateException: run: Caught an unrecoverable exception waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@6737a445 at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:104) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.IllegalStateException: waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@6737a445 at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:193) at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:151) at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:136) at org.apache.giraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:99) at org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:233) at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(BspServiceWorker.java:316) at org.apache.giraph.worker.BspServiceWorker.loadVertices(BspServiceWorker.java:409) at org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:629) at org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:284) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:93) ... 7 more Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:202) at org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:312) at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:185) ... 16 more Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.giraph.utils.UnsafeByteArrayOutputStream.(UnsafeByteArrayOutputStream.java:81) at org.apache.giraph.conf.ImmutableClassesGiraphConfiguration.createExtendedDataOutput(ImmutableClassesGiraphConfiguration.java:1161) at org.apache.giraph.comm.SendPartitionCache.addVertex(SendPartitionCache.java:77) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.sendVertexRequest(NettyWorkerClientRequestProcessor.java:248) at org.apache.giraph.worker.VertexInputSplitsCallable.readInputSplit(VertexInputSplitsCallable.java:231) at org.apache.giraph.worker.InputSplitsCallable.loadInputSplit(InputSplitsCallable.java:267) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:211) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:60) at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Thank you so much! Best, Hai --001a1141bf42098127053f563c48 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Dear all,

I'm facing a p= roblem when I run large graph job (currently 1.6T, will be 16T then), it al= ways shows java.lang.OutOfMemoryError: Java heap space=C2=A0error when load= ed specific numbers of vertex(near 59000000). I tried to add like:
-Dgiraph.useOutOfCoreGraph=3Dtrue
=C2=A0-Dmapred.child.java.opt= s=3D"-XX:-UseGCOverheadLimit" OR -Dmapred.child.java.opts=3D"= ;-Xmx16384"
=C2=A0-Dgiraph.yarn.task.heap.mb=3D36570

but the problem remain though I can see those value a= re shown in Metadata.

I'm not sure the max val= ue of memory in this VertexInputSplitsCallable info is related to java heap= size.=C2=A0
INFO [load-0] org.apache.giraph.worker.VertexInputSp= litsCallable: readVertexInputSplit: Loaded 46975802 vertices at 68977.49310= 291892 vertices/sec 0 edges at 0.0 edges/sec Memory (free/total/max) =3D 47= 5.08M / 2759.00M / 2759.00M

But I am noticed in ma= in log, it always shows:
INFO [AsyncDispatcher event handl= er] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap si= ze. Setting task attempt jvm max heap size to -Xmx2868m
no= matter what arguments I added. Even when I run normal Hadoop jobs.=C2= =A0

Any ideas about this? Following is the log.

2016-10-20 21:25:49,008 ERROR [netty-client-wor= ker-2] org.apache.giraph.comm.netty.NettyClient: Request failed
j= ava.lang.OutOfMemoryError: Java heap space
at io.netty.buffer.Unpoole= dHeapByteBuf.<init>(UnpooledHeapByteBuf.java:45)
at io.netty.bu= ffer.UnpooledByteBufAllocator.newHeapBuffer(UnpooledByteBufAllocator.java:4= 3)
at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByt= eBufAllocator.java:136)
at io.netty.buffer.AbstractByteBufAllocator.h= eapBuffer(AbstractByteBufAllocator.java:127)
at io.netty.buffer.Abstr= actByteBufAllocator.buffer(AbstractByteBufAllocator.java:85)
at org.a= pache.giraph.comm.netty.handler.RequestEncoder.write(RequestEncoder.java:81= )
= at io.netty.channel.DefaultChannelHandlerContext.invokeWrite(Defaul= tChannelHandlerContext.java:645)
at io.netty.channel.DefaultChannelHa= ndlerContext.access$2000(DefaultChannelHandlerContext.java:29)
at io.= netty.channel.DefaultChannelHandlerContext$WriteTask.run(DefaultChannelHand= lerContext.java:906)
at io.netty.util.concurrent.DefaultEventExecut= or.run(DefaultEventExecutor.java:36)
at io.netty.util.concurrent.Sing= leThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
<= span class=3D"gmail-Apple-tab-span" style=3D"white-space:pre"> at ja= va.lang.Thread.run(Thread.java:745)
2016-10-20 21:25:55,299 ERROR= [netty-client-worker-1] org.apache.giraph.comm.netty.NettyClient: Request = failed
java.lang.OutOfMemoryError: Java heap space
at io.ne= tty.buffer.UnpooledHeapByteBuf.<init>(UnpooledHeapByteBuf.java:45)
at io.netty.buffer.UnpooledByteBufAllocator.newHeapBuffer(UnpooledByteB= ufAllocator.java:43)
at io.netty.buffer.AbstractByteBufAllocator.he= apBuffer(AbstractByteBufAllocator.java:136)
at io.netty.buffer.Abstra= ctByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:127)
= at i= o.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.jav= a:85)
at org.apache.giraph.comm.netty.handler.RequestEncoder.write(Re= questEncoder.java:81)
at io.netty.channel.DefaultChannelHandlerConte= xt.invokeWrite(DefaultChannelHandlerContext.java:645)
at io.netty.cha= nnel.DefaultChannelHandlerContext.access$2000(DefaultChannelHandlerContext.= java:29)
at io.netty.channel.DefaultChannelHandlerContext$WriteTask.r= un(DefaultChannelHandlerContext.java:906)
at io.netty.util.concurrent= .DefaultEventExecutor.run(DefaultEventExecutor.java:36)
at io.netty.u= til.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.ja= va:101)
at java.lang.Thread.run(Thread.java:745)
2016-10-20= 21:26:06,731 ERROR [main] org.apache.giraph.graph.GraphMapper: Caught an u= nrecoverable exception waitFor: ExecutionException occurred while waiting f= or org.apache.giraph.utils.ProgressableUtils$FutureWaitable@6737a445
<= div>java.lang.IllegalStateException: waitFor: ExecutionException occurred w= hile waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@6= 737a445
at org.apache.giraph.utils.ProgressableUtils.waitFor(Progress= ableUtils.java:193)
at org.apache.giraph.utils.ProgressableUtils.wait= Forever(ProgressableUtils.java:151)
at org.apache.giraph.utils.Progre= ssableUtils.waitForever(ProgressableUtils.java:136)
at org.apache.g= iraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:99)
at org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(P= rogressableUtils.java:233)
at org.apache.giraph.worker.BspServiceWork= er.loadInputSplits(BspServiceWorker.java:316)
at org.apache.giraph.wo= rker.BspServiceWorker.loadVertices(BspServiceWorker.java:409)
at org.= apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:629)
at org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java= :284)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:93)=
= at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org= .apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.sec= urity.AccessController.doPrivileged(Native Method)
at javax.security.= auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.Use= rGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.= hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.= util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap s= pace
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:202)
= at o= rg.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(Progressabl= eUtils.java:312)
at org.apache.giraph.utils.ProgressableUtils.waitFor= (ProgressableUtils.java:185)
... 16 more
Caused by: java.la= ng.OutOfMemoryError: Java heap space
at org.apache.giraph.utils.Unsaf= eByteArrayOutputStream.<init>(UnsafeByteArrayOutputStream.java:81)
at org.apache.giraph.conf.ImmutableClassesGiraphConfiguration.createExt= endedDataOutput(ImmutableClassesGiraphConfiguration.java:1161)
at org= .apache.giraph.comm.SendPartitionCache.addVertex(SendPartitionCache.java:77= )
= at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.s= endVertexRequest(NettyWorkerClientRequestProcessor.java:248)
at org.a= pache.giraph.worker.VertexInputSplitsCallable.readInputSplit(VertexInputSpl= itsCallable.java:231)
at org.apache.giraph.worker.InputSplitsCallabl= e.loadInputSplit(InputSplitsCallable.java:267)
at org.apache.giraph.w= orker.InputSplitsCallable.call(InputSplitsCallable.java:211)
at org.a= pache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:60)
at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCal= lable.java:51)
at java.util.concurrent.FutureTask.run(FutureTask.java= :262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool= Executor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker= .run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java= :745)
2016-10-20 21:26:06,737 ERROR [main] org.apache.giraph.work= er.BspServiceWorker: unregisterHealth: Got failure, unregistering health on= /_hadoopBsp/job_1476386340018_0175/_applicationAttemptsDir/0/_superstepDir= /-1/_workerHealthyDir/hadoop18.umd.com_23 on superstep -1
2016-10= -20 21:26:06,746 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception = running child : java.lang.IllegalStateException: run: Caught an unrecoverab= le exception waitFor: ExecutionException occurred while waiting for org.apa= che.giraph.utils.ProgressableUtils$FutureWaitable@6737a445
at org.apa= che.giraph.graph.GraphMapper.run(GraphMapper.java:104)
at org.apache.= hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.h= adoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapre= d.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessControll= er.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Su= bject.java:415)
at org.apache.hadoop.security.UserGroupInformation.do= As(UserGroupInformation.java:1693)
at org.apache.hadoop.mapred.YarnCh= ild.main(YarnChild.java:158)
Caused by: java.lang.IllegalStateExc= eption: waitFor: ExecutionException occurred while waiting for org.apache.g= iraph.utils.ProgressableUtils$FutureWaitable@6737a445
at org.apache.g= iraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:193)
at = org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.jav= a:151)
at org.apache.giraph.utils.ProgressableUtils.waitForever(Progr= essableUtils.java:136)
at org.apache.giraph.utils.ProgressableUtils.g= etFutureResult(ProgressableUtils.java:99)
at org.apache.giraph.utils.= ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:233)
at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(BspServiceWo= rker.java:316)
at org.apache.giraph.worker.BspServiceWorker.loadVerti= ces(BspServiceWorker.java:409)
at org.apache.giraph.worker.BspService= Worker.setup(BspServiceWorker.java:629)
at org.apache.giraph.graph.Gr= aphTaskManager.execute(GraphTaskManager.java:284)
at org.apache.girap= h.graph.GraphMapper.run(GraphMapper.java:93)
... 7 more
Cau= sed by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError= : Java heap space
at java.util.concurrent.FutureTask.report(FutureTas= k.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:20= 2)
at org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFo= r(ProgressableUtils.java:312)
at org.apache.giraph.utils.Progressable= Utils.waitFor(ProgressableUtils.java:185)
... 16 more
Cause= d by: java.lang.OutOfMemoryError: Java heap space
at org.apache.girap= h.utils.UnsafeByteArrayOutputStream.<init>(UnsafeByteArrayOutputStrea= m.java:81)
at org.apache.giraph.conf.ImmutableClassesGiraphConfigurat= ion.createExtendedDataOutput(ImmutableClassesGiraphConfiguration.java:1161)=
= at org.apache.giraph.comm.SendPartitionCache.addVertex(SendPartition= Cache.java:77)
at org.apache.giraph.comm.netty.NettyWorkerClientReque= stProcessor.sendVertexRequest(NettyWorkerClientRequestProcessor.java:248)
at org.apache.giraph.worker.VertexInputSplitsCallable.readInputSplit(V= ertexInputSplitsCallable.java:231)
at org.apache.giraph.worker.InputS= plitsCallable.loadInputSplit(InputSplitsCallable.java:267)
at org.apa= che.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:211)
at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallabl= e.java:60)
at org.apache.giraph.utils.LogStacktraceCallable.call(LogS= tacktraceCallable.java:51)
at java.util.concurrent.FutureTask.run(Fut= ureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorke= r(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExe= cutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run= (Thread.java:745)


Thank you s= o much!

Best,

Hai
--001a1141bf42098127053f563c48--