accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jianshi Huang <jianshi.hu...@gmail.com>
Subject Re: MutationsRejectedException and TApplicationException
Date Wed, 18 Jun 2014 16:51:35 GMT
Oh, this memory size:

tserver.memory.maps.max
1G -> 20G (looks like this is an overkill, is it?)

tserver.cache.data.size
128M? -> 1024M

tserver.cache.index.size
128M? -> 1024M





On Thu, Jun 19, 2014 at 12:48 AM, Josh Elser <josh.elser@gmail.com> wrote:

> Which memory size? :)
>
> JVM max heap size? In-memory map size? Either (really, both) would be good
> to increase.
>
> If the situation is as Eric posited, it's very likely that increasing the
> resources that Accumulo has available would make it more responsive and
> able to keep up with your ingest load.
>
> - Josh
>
>
> On 6/18/14, 9:44 AM, Jianshi Huang wrote:
>
>> Does memory size important to reduce this sort of errors? I used the
>> default settings (1GB/per tserver), this might be too small.
>>
>> I increased it to 20GB, and I saw no errors after that.
>>
>> Jianshi
>>
>>
>> On Thu, Jun 19, 2014 at 12:39 AM, Eric Newton <eric.newton@gmail.com
>> <mailto:eric.newton@gmail.com>> wrote:
>>
>>     This error is often a result of overwhelming your server resources.
>>
>>     It basically says "an update came in that was so old, the id used to
>>     identify the sender has already aged off."
>>
>>     What is your expected ingest rate during the job?  What sort of
>>     resources does accumulo have?
>>
>>
>>     On Wed, Jun 18, 2014 at 7:09 AM, Jianshi Huang
>>     <jianshi.huang@gmail.com <mailto:jianshi.huang@gmail.com>> wrote:
>>
>>         Here's the error message I got from the tserver_xxx.log
>>
>>         2014-06-18 01:06:06,816 [tserver.TabletServer] INFO : Adding 1
>>         logs for extent g;cust:2072821;cust:20700111 as alias 37
>>         2014-06-18 01:06:16,286 [thrift.ProcessFunction] ERROR: Internal
>>         error processing applyUpdates
>>         java.lang.RuntimeException: No Such SessionID
>>                  at
>>         org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.
>> applyUpdates(TabletServer.java:1522)
>>                  at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown
>>         Source)
>>                  at
>>         sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>>                  at java.lang.reflect.Method.invoke(Method.java:606)
>>                  at
>>         org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.
>> invoke(TraceWrap.java:63)
>>                  at com.sun.proxy.$Proxy23.applyUpdates(Unknown Source)
>>                  at
>>         org.apache.accumulo.core.tabletserver.thrift.
>> TabletClientService$Processor$applyUpdates.getResult(
>> TabletClientService.java:2347\
>>         )
>>                  at
>>         org.apache.accumulo.core.tabletserver.thrift.
>> TabletClientService$Processor$applyUpdates.getResult(
>> TabletClientService.java:2333\
>>         )
>>                  at
>>         org.apache.thrift.ProcessFunction.process(
>> ProcessFunction.java:39)
>>                  at
>>         org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>>                  at
>>         org.apache.accumulo.server.util.TServerUtils$
>> TimedProcessor.process(TServerUtils.java:171)
>>                  at
>>         org.apache.thrift.server.AbstractNonblockingServer$
>> FrameBuffer.invoke(AbstractNonblockingServer.java:478)
>>                  at
>>         org.apache.accumulo.server.util.TServerUtils$THsHaServer$
>> Invocation.run(TServerUtils.java:231)
>>                  at
>>         java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>                  at
>>         java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>                  at
>>         org.apache.accumulo.trace.instrument.TraceRunnable.run(
>> TraceRunnable.java:47)
>>                  at
>>         org.apache.accumulo.core.util.LoggingRunnable.run(
>> LoggingRunnable.java:34)
>>                  at java.lang.Thread.run(Thread.java:724)
>>         2014-06-18 01:06:16,287 [thrift.ProcessFunction] ERROR: Internal
>>         error processing applyUpdates
>>         java.lang.RuntimeException: No Such SessionID
>>                  at
>>         org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.
>> applyUpdates(TabletServer.java:1522)
>>                  at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown
>>         Source)
>>                  at
>>         sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>>                  at java.lang.reflect.Method.invoke(Method.java:606)
>>                  at
>>         org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.
>> invoke(TraceWrap.java:63)
>>                  at com.sun.proxy.$Proxy23.applyUpdates(Unknown Source)
>>                  at
>>         org.apache.accumulo.core.tabletserver.thrift.
>> TabletClientService$Processor$applyUpdates.getResult(
>> TabletClientService.java:2347\
>>         )
>>                  at
>>         org.apache.accumulo.core.tabletserver.thrift.
>> TabletClientService$Processor$applyUpdates.getResult(
>> TabletClientService.java:2333\
>>         )
>>                  at
>>         org.apache.thrift.ProcessFunction.process(
>> ProcessFunction.java:39)
>>                  at
>>         org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>>                  at
>>         org.apache.accumulo.server.util.TServerUtils$
>> TimedProcessor.process(TServerUtils.java:171)
>>                  at
>>         org.apache.thrift.server.AbstractNonblockingServer$
>> FrameBuffer.invoke(AbstractNonblockingServer.java:478)
>>                  at
>>         org.apache.accumulo.server.util.TServerUtils$THsHaServer$
>> Invocation.run(TServerUtils.java:231)
>>                  at
>>         java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>>                  at
>>         java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>>                  at
>>         org.apache.accumulo.trace.instrument.TraceRunnable.run(
>> TraceRunnable.java:47)
>>                  at
>>         org.apache.accumulo.core.util.LoggingRunnable.run(
>> LoggingRunnable.java:34)
>>                  at java.lang.Thread.run(Thread.java:724)
>>         2014-06-18 01:06:16,287 [util.TServerUtils$THsHaServer] WARN :
>>         Got an IOException during write!
>>         java.io.IOException: Connection reset by peer
>>                  at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>                  at
>>         sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>                  at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.
>> java:93)
>>                  at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>>                  at
>>         sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
>>                  at
>>         org.apache.thrift.transport.TNonblockingSocket.write(
>> TNonblockingSocket.java:164)
>>                  at
>>         org.apache.thrift.server.AbstractNonblockingServer$
>> FrameBuffer.write(AbstractNonblockingServer.java:381)
>>                  at
>>         org.apache.thrift.server.AbstractNonblockingServer$
>> AbstractSelectThread.handleWrite(AbstractNonblockingServer.java:220)
>>                  at
>>         org.apache.thrift.server.TNonblockingServer$
>> SelectAcceptThread.select(TNonblockingServer.java:201)
>>
>>
>>         Jianshi
>>
>>
>>         On Wed, Jun 18, 2014 at 2:54 PM, Jianshi Huang
>>         <jianshi.huang@gmail.com <mailto:jianshi.huang@gmail.com>> wrote:
>>
>>             I see. I'll check the tablet server log and paste the error
>>             message in later thread.
>>
>>             BTW, looks like the AccumuloOutputFormat is the cause, I'm
>>             currently using BatchWriter and it works well.
>>
>>             My code looks like this (it's in Scala as I'm using Spark):
>>
>>                  AccumuloOutputFormat.setZooKeeperInstance(job,
>>             Conf.getString("accumulo.instance"),
>>             Conf.getString("accumulo.zookeeper.servers"))
>>                  AccumuloOutputFormat.setConnectorInfo(job,
>>             Conf.getString("accumulo.user"), new
>>             PasswordToken(Conf.getString("accumulo.password")))
>>                  AccumuloOutputFormat.setDefaultTableName(job,
>>             Conf.getString("accumulo.table"))
>>
>>                  val paymentRDD: RDD[(Text, Mutation)] = payment.flatMap
>>             { payment =>
>>             //      val key = new Text(Conf.getString("accumulo.table"))
>>                    paymentMutations(payment).map((null, _))
>>                  }
>>
>>
>>             paymentRDD.saveAsNewAPIHadoopFile(Conf.
>> getString("accumulo.instance"),
>>             classOf[Void], classOf[Mutation],
>>             classOf[AccumuloOutputFormat], job.getConfiguration)
>>
>>
>>             It's also possible that saveAsNewAPIHadoopFile doesn't work
>>             well with AccumuloOutputFormat.
>>
>>
>>
>>             Jianshi
>>
>>
>>
>>
>>             On Wed, Jun 18, 2014 at 12:45 PM, Josh Elser
>>             <josh.elser@gmail.com <mailto:josh.elser@gmail.com>> wrote:
>>
>>                 Check the TabletServer logs. This Exception is telling
>>                 you that there was an error on the server. You should
>>                 look there for what the real problem was. You can do
>>                 this one of two ways.
>>
>>                 1) Use the "Recent Logs" page on the Accumulo monitor
>>                 (http://accumulo_monitor_host:__50095
>>                 <http://accumulo_monitor_host:50095>). Unless you
>>
>>                 cleared the logs, or restarted the monitor process since
>>                 you got this error, you should be able to see a nice
>>                 HTML view of any errors
>>
>>                 2) Check the debug log, e.g.
>>                 $ACCUMULO_HOME/logs/tserver_$__host.debug.log. If you're
>>
>>                 running tservers on more than one node, be sure that you
>>                 check the log files on all nodes.
>>
>>                 - Josh
>>
>>
>>                 On 6/17/14, 9:33 PM, Jianshi Huang wrote:
>>
>>                     Hi,
>>
>>                     I got the following errors during MapReduce
>>                     ingestion, are they serious
>>                     errors?
>>
>>                     java.io.IOException:
>>                     org.apache.accumulo.core.__client.__
>> MutationsRejectedException:
>>
>>                     # constraint
>>                     violations : 0  security codes: {}  # server\ errors
>>                     1 # exceptions 0
>>                               at
>>                     org.apache.accumulo.core.__client.mapreduce.__
>> AccumuloOutputFormat$__AccumuloRecordWriter.write(__
>> AccumuloOutputFormat.java:437)
>>                               at
>>                     org.apache.accumulo.core.__client.mapreduce.__
>> AccumuloOutputFormat$__AccumuloRecordWriter.write(__
>> AccumuloOutputFormat.java:373)
>>                               at
>>                     org.apache.spark.rdd.__PairRDDFunctions.org
>>                     <http://org.apache.spark.rdd.PairRDDFunctions.org>
>>                     <http://org.apache.spark.rdd.__PairRDDFunctions.org
>>                     <http://org.apache.spark.rdd.PairRDDFunctions.org
>> >>$apache$__spark$rdd$PairRDDFunctions$$__writeShard$1(
>> PairRDDFunctions.__scala:716)
>>
>>                               at
>>                     org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__
>> saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730)
>>                               at
>>                     org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__
>> saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730)
>>                               at
>>                     org.apache.spark.scheduler.__
>> ResultTask.runTask(ResultTask.__scala:111)
>>                               at
>>                     org.apache.spark.scheduler.__Task.run(Task.scala:51)
>>                               at
>>                     org.apache.spark.executor.__
>> Executor$TaskRunner.run(__Executor.scala:187)
>>                               at
>>                     java.util.concurrent.__ThreadPoolExecutor.runWorker(_
>> _ThreadPoolExecutor.java:1145)
>>                               at
>>                     java.util.concurrent.__ThreadPoolExecutor$Worker.run(
>> __ThreadPoolExecutor.java:615)
>>                               at java.lang.Thread.run(Thread.__java:724)
>>
>>
>>                     And
>>
>>                     java.io.IOException:
>>                     org.apache.accumulo.core.__client.AccumuloException:
>>                     org.apache.thrift.__TApplicationException: Internal
>>                     error processing\
>>                        applyUpdates
>>                               at
>>                     org.apache.accumulo.core.__client.mapreduce.__
>> AccumuloOutputFormat.__getRecordWriter(__AccumuloOutputFormat.java:558)
>>                               at
>>                     org.apache.spark.rdd.__PairRDDFunctions.org
>>                     <http://org.apache.spark.rdd.PairRDDFunctions.org>
>>                     <http://org.apache.spark.rdd.__PairRDDFunctions.org
>>                     <http://org.apache.spark.rdd.PairRDDFunctions.org
>> >>$apache$__spark$rdd$PairRDDFunctions$$__writeShard$1(
>> PairRDDFunctions.__scala:712)
>>
>>                               at
>>                     org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__
>> saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730)
>>                               at
>>                     org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__
>> saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730)
>>                               at
>>                     org.apache.spark.scheduler.__
>> ResultTask.runTask(ResultTask.__scala:111)
>>                               at
>>                     org.apache.spark.scheduler.__Task.run(Task.scala:51)
>>                               at
>>                     org.apache.spark.executor.__
>> Executor$TaskRunner.run(__Executor.scala:187)
>>                               at
>>                     java.util.concurrent.__ThreadPoolExecutor.runWorker(_
>> _ThreadPoolExecutor.java:1145)
>>                               at
>>                     java.util.concurrent.__ThreadPoolExecutor$Worker.run(
>> __ThreadPoolExecutor.java:615)
>>                               at java.lang.Thread.run(Thread.__java:724)
>>
>>
>>
>>                     Cheers,
>>                     --
>>                     Jianshi Huang
>>
>>                     LinkedIn: jianshi
>>                     Twitter: @jshuang
>>                     Github & Blog: http://huangjs.github.com/
>>
>>
>>
>>
>>             --
>>             Jianshi Huang
>>
>>             LinkedIn: jianshi
>>             Twitter: @jshuang
>>             Github & Blog: http://huangjs.github.com/
>>
>>
>>
>>
>>         --
>>         Jianshi Huang
>>
>>         LinkedIn: jianshi
>>         Twitter: @jshuang
>>         Github & Blog: http://huangjs.github.com/
>>
>>
>>
>>
>>
>> --
>> Jianshi Huang
>>
>> LinkedIn: jianshi
>> Twitter: @jshuang
>> Github & Blog: http://huangjs.github.com/
>>
>


-- 
Jianshi Huang

LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/

Mime
View raw message