Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@apache.org
Date: Tue, 30 Apr 2013 17:52:16 +0000 (UTC)
From: "Nasheb Ismaily (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.12644921.1367006991897.249334.1367344336140@arcas>
In-Reply-To: <JIRA.12644921.1367006991897@arcas>
References: <JIRA.12644921.1367006991897@arcas>
Subject: [jira] [Commented] (ACCUMULO-1355) Accumulo ingestion hangs/fails
 for 300 MB+ file, local machine memory usage grows to 10GB+
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/ACCUMULO-1355?page=3Dcom.atlass=
ian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1=
3645775#comment-13645775 ]=20

Nasheb Ismaily commented on ACCUMULO-1355:
------------------------------------------

hmm...
I've attempted to increase the TServers memory: export ACCUMULO_TSERVER_OPT=
S=3D"-Xmx2G"
And ran the ingest, but I get the same issue.=20
I'm running on my local machine (Mac) with Accumulo 1.4.3 and zookeeper 3.4=
.5

How much RAM are you burning for the ingest?
Which version of Zookeeper are you running?

Im using a pig script to ingest so I can get a complete stack trace, when I=
 force interrupt the process I get

=09at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.waitRTE(=
TabletServerBatchWriter.java:396)
=09at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.close(Ta=
bletServerBatchWriter.java:296)
=09at org.apache.accumulo.core.client.impl.BatchWriterImpl.close(BatchWrite=
rImpl.java:51)
=09..
=09..=09
=09at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutpu=
tFormat$PigRecordWriter.write(PigOutputFormat.java:139)
=09at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutpu=
tFormat$PigRecordWriter.write(PigOutputFormat.java:98)
=09at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTa=
sk.java:639)
=09at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOut=
putContext.java:80)
=09at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOn=
ly$Map.collect(PigMapOnly.java:48)
=09at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGener=
icMapBase.runPipeline(PigGenericMapBase.java:273)
=09at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGener=
icMapBase.map(PigGenericMapBase.java:266)
=09at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGener=
icMapBase.map(PigGenericMapBase.java:64)
=09at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
=09at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
=09at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
=09at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:2=
12)

               =20
> Accumulo ingestion hangs/fails for 300 MB+ file, local machine memory usa=
ge grows to 10GB+
> -------------------------------------------------------------------------=
-----------------
>
>                 Key: ACCUMULO-1355
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1355
>             Project: Accumulo
>          Issue Type: Bug
>          Components: client, thrift, tserver
>    Affects Versions: 1.4.2, 1.4.3
>            Reporter: Nasheb Ismaily
>            Assignee: Billie Rinaldi
>            Priority: Blocker
>
> *Bug:*
> I am attempting to ingest a 300+ MB file into Accumulo, however the inges=
tion process hangs and my local machines memory consumption grows to 10GB+.
> The mutation is never put into accumulo.
> *The file I am ingesting can be found here (needs to be unzipped):* http:=
//www.epa.gov/ttn/atw/nata2005/emissions_mdbzip/2005natav3_ei_ca.zip
> =20
> *I=E2=80=99ve attached a snippet of code used to ingest the data here:* h=
ttp://pastebin.com/Yh4V6nng
> *Initial Investigation:*
> While attempting to upload the file, the thread which sends the mutation =
to the tablet =E2=80=98waits=E2=80=99 in the waitRTE() method which is call=
ed by synchronized void flush() in TabletServerBatchWriter.java and is neve=
r =E2=80=98woken up=E2=80=99 by a notify call. While =E2=80=98waiting=E2=80=
=99 my local machines memory allocation grows to 10 GB+.=20
> *Thread Stack Trace (from java profiler):*
> java.nio.Bits.copyFromArray(Bits.java:699)
> java.nio.DirectByteBuffer.put(DirectByteBuffer.java:360)
> java.nio.DirectByteBuffer.put(DirectByteBuffer.java:331)
> sun.nio.ch.IOUtil.write(IOUtil.java:35)
> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:336)
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStr=
eam.java:55)
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:1=
42)
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:14=
6)
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:10=
7)
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.j=
ava:145)
> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:=
157)
> org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport=
.flush(ThriftTransportPool.java:299)
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.s=
end_applyUpdates(TabletClientService.java:449)
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.a=
pplyUpdates(TabletClientService.java:436)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java=
:39)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI=
mpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
> org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$2.invoke(Trace=
Wrap.java:84)
> com.sun.proxy.$Proxy7.applyUpdates(Unknown Source)
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWrit=
er.sendMutationsToTabletServer(TabletServerBatchWriter.java:768)
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWrit=
er.access$1400(TabletServerBatchWriter.java:536)
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWrit=
er$SendTask.send(TabletServerBatchWriter.java:700)
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWrit=
er$SendTask.run(TabletServerBatchWriter.java:671)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor=
.java:895)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav=
a:918)
> java.lang.Thread.run(Thread.java:680)
> *Tserver log error:*
> 2013-04-26 15:36:25,253 [server.TNonblockingServer] ERROR: Unexpected exc=
eption while invoking!
> java.lang.RuntimeException: No Such SessionID
> =09at org.apache.accumulo.server.tabletserver.TabletServer$ThriftClientHa=
ndler.applyUpdates(TabletServer.java:1433)
> =09at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc=
essorImpl.java:43)
> =09at java.lang.reflect.Method.invoke(Method.java:601)
> =09at org.apache.accumulo.cloudtrace.instrument.thrift.TraceWrap$1.invoke=
(TraceWrap.java:59)
> =09at com.sun.proxy.$Proxy2.applyUpdates(Unknown Source)
> =09at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Pr=
ocessor$applyUpdates.process(TabletClientService.java:2315)
> =09at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Pr=
ocessor.process(TabletClientService.java:2037)
> =09at org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process=
(TServerUtils.java:154)
> =09at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNon=
blockingServer.java:631)
> =09at org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation=
.run(TServerUtils.java:202)
> =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto=
r.java:1145)
> =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut=
or.java:615)
> =09at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.j=
ava:34)
> =09at java.lang.Thread.run(Thread.java:722)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira