accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: MutationsRejectedException and TApplicationException
Date Wed, 18 Jun 2014 16:48:47 GMT
Which memory size? :)

JVM max heap size? In-memory map size? Either (really, both) would be 
good to increase.

If the situation is as Eric posited, it's very likely that increasing 
the resources that Accumulo has available would make it more responsive 
and able to keep up with your ingest load.

- Josh

On 6/18/14, 9:44 AM, Jianshi Huang wrote:
> Does memory size important to reduce this sort of errors? I used the
> default settings (1GB/per tserver), this might be too small.
>
> I increased it to 20GB, and I saw no errors after that.
>
> Jianshi
>
>
> On Thu, Jun 19, 2014 at 12:39 AM, Eric Newton <eric.newton@gmail.com
> <mailto:eric.newton@gmail.com>> wrote:
>
>     This error is often a result of overwhelming your server resources.
>
>     It basically says "an update came in that was so old, the id used to
>     identify the sender has already aged off."
>
>     What is your expected ingest rate during the job?  What sort of
>     resources does accumulo have?
>
>
>     On Wed, Jun 18, 2014 at 7:09 AM, Jianshi Huang
>     <jianshi.huang@gmail.com <mailto:jianshi.huang@gmail.com>> wrote:
>
>         Here's the error message I got from the tserver_xxx.log
>
>         2014-06-18 01:06:06,816 [tserver.TabletServer] INFO : Adding 1
>         logs for extent g;cust:2072821;cust:20700111 as alias 37
>         2014-06-18 01:06:16,286 [thrift.ProcessFunction] ERROR: Internal
>         error processing applyUpdates
>         java.lang.RuntimeException: No Such SessionID
>                  at
>         org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.applyUpdates(TabletServer.java:1522)
>                  at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown
>         Source)
>                  at
>         sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>                  at java.lang.reflect.Method.invoke(Method.java:606)
>                  at
>         org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
>                  at com.sun.proxy.$Proxy23.applyUpdates(Unknown Source)
>                  at
>         org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$applyUpdates.getResult(TabletClientService.java:2347\
>         )
>                  at
>         org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$applyUpdates.getResult(TabletClientService.java:2333\
>         )
>                  at
>         org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>                  at
>         org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>                  at
>         org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:171)
>                  at
>         org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
>                  at
>         org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:231)
>                  at
>         java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>                  at
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>                  at
>         org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>                  at
>         org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>                  at java.lang.Thread.run(Thread.java:724)
>         2014-06-18 01:06:16,287 [thrift.ProcessFunction] ERROR: Internal
>         error processing applyUpdates
>         java.lang.RuntimeException: No Such SessionID
>                  at
>         org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.applyUpdates(TabletServer.java:1522)
>                  at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown
>         Source)
>                  at
>         sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>                  at java.lang.reflect.Method.invoke(Method.java:606)
>                  at
>         org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
>                  at com.sun.proxy.$Proxy23.applyUpdates(Unknown Source)
>                  at
>         org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$applyUpdates.getResult(TabletClientService.java:2347\
>         )
>                  at
>         org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$applyUpdates.getResult(TabletClientService.java:2333\
>         )
>                  at
>         org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>                  at
>         org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>                  at
>         org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:171)
>                  at
>         org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
>                  at
>         org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:231)
>                  at
>         java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>                  at
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>                  at
>         org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
>                  at
>         org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
>                  at java.lang.Thread.run(Thread.java:724)
>         2014-06-18 01:06:16,287 [util.TServerUtils$THsHaServer] WARN :
>         Got an IOException during write!
>         java.io.IOException: Connection reset by peer
>                  at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>                  at
>         sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>                  at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>                  at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>                  at
>         sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
>                  at
>         org.apache.thrift.transport.TNonblockingSocket.write(TNonblockingSocket.java:164)
>                  at
>         org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.write(AbstractNonblockingServer.java:381)
>                  at
>         org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleWrite(AbstractNonblockingServer.java:220)
>                  at
>         org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:201)
>
>
>         Jianshi
>
>
>         On Wed, Jun 18, 2014 at 2:54 PM, Jianshi Huang
>         <jianshi.huang@gmail.com <mailto:jianshi.huang@gmail.com>> wrote:
>
>             I see. I'll check the tablet server log and paste the error
>             message in later thread.
>
>             BTW, looks like the AccumuloOutputFormat is the cause, I'm
>             currently using BatchWriter and it works well.
>
>             My code looks like this (it's in Scala as I'm using Spark):
>
>                  AccumuloOutputFormat.setZooKeeperInstance(job,
>             Conf.getString("accumulo.instance"),
>             Conf.getString("accumulo.zookeeper.servers"))
>                  AccumuloOutputFormat.setConnectorInfo(job,
>             Conf.getString("accumulo.user"), new
>             PasswordToken(Conf.getString("accumulo.password")))
>                  AccumuloOutputFormat.setDefaultTableName(job,
>             Conf.getString("accumulo.table"))
>
>                  val paymentRDD: RDD[(Text, Mutation)] = payment.flatMap
>             { payment =>
>             //      val key = new Text(Conf.getString("accumulo.table"))
>                    paymentMutations(payment).map((null, _))
>                  }
>
>
>             paymentRDD.saveAsNewAPIHadoopFile(Conf.getString("accumulo.instance"),
>             classOf[Void], classOf[Mutation],
>             classOf[AccumuloOutputFormat], job.getConfiguration)
>
>
>             It's also possible that saveAsNewAPIHadoopFile doesn't work
>             well with AccumuloOutputFormat.
>
>
>
>             Jianshi
>
>
>
>
>             On Wed, Jun 18, 2014 at 12:45 PM, Josh Elser
>             <josh.elser@gmail.com <mailto:josh.elser@gmail.com>> wrote:
>
>                 Check the TabletServer logs. This Exception is telling
>                 you that there was an error on the server. You should
>                 look there for what the real problem was. You can do
>                 this one of two ways.
>
>                 1) Use the "Recent Logs" page on the Accumulo monitor
>                 (http://accumulo_monitor_host:__50095
>                 <http://accumulo_monitor_host:50095>). Unless you
>                 cleared the logs, or restarted the monitor process since
>                 you got this error, you should be able to see a nice
>                 HTML view of any errors
>
>                 2) Check the debug log, e.g.
>                 $ACCUMULO_HOME/logs/tserver_$__host.debug.log. If you're
>                 running tservers on more than one node, be sure that you
>                 check the log files on all nodes.
>
>                 - Josh
>
>
>                 On 6/17/14, 9:33 PM, Jianshi Huang wrote:
>
>                     Hi,
>
>                     I got the following errors during MapReduce
>                     ingestion, are they serious
>                     errors?
>
>                     java.io.IOException:
>                     org.apache.accumulo.core.__client.__MutationsRejectedException:
>                     # constraint
>                     violations : 0  security codes: {}  # server\ errors
>                     1 # exceptions 0
>                               at
>                     org.apache.accumulo.core.__client.mapreduce.__AccumuloOutputFormat$__AccumuloRecordWriter.write(__AccumuloOutputFormat.java:437)
>                               at
>                     org.apache.accumulo.core.__client.mapreduce.__AccumuloOutputFormat$__AccumuloRecordWriter.write(__AccumuloOutputFormat.java:373)
>                               at
>                     org.apache.spark.rdd.__PairRDDFunctions.org
>                     <http://org.apache.spark.rdd.PairRDDFunctions.org>
>                     <http://org.apache.spark.rdd.__PairRDDFunctions.org
>                     <http://org.apache.spark.rdd.PairRDDFunctions.org>>$apache$__spark$rdd$PairRDDFunctions$$__writeShard$1(PairRDDFunctions.__scala:716)
>
>                               at
>                     org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730)
>                               at
>                     org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730)
>                               at
>                     org.apache.spark.scheduler.__ResultTask.runTask(ResultTask.__scala:111)
>                               at
>                     org.apache.spark.scheduler.__Task.run(Task.scala:51)
>                               at
>                     org.apache.spark.executor.__Executor$TaskRunner.run(__Executor.scala:187)
>                               at
>                     java.util.concurrent.__ThreadPoolExecutor.runWorker(__ThreadPoolExecutor.java:1145)
>                               at
>                     java.util.concurrent.__ThreadPoolExecutor$Worker.run(__ThreadPoolExecutor.java:615)
>                               at java.lang.Thread.run(Thread.__java:724)
>
>
>                     And
>
>                     java.io.IOException:
>                     org.apache.accumulo.core.__client.AccumuloException:
>                     org.apache.thrift.__TApplicationException: Internal
>                     error processing\
>                        applyUpdates
>                               at
>                     org.apache.accumulo.core.__client.mapreduce.__AccumuloOutputFormat.__getRecordWriter(__AccumuloOutputFormat.java:558)
>                               at
>                     org.apache.spark.rdd.__PairRDDFunctions.org
>                     <http://org.apache.spark.rdd.PairRDDFunctions.org>
>                     <http://org.apache.spark.rdd.__PairRDDFunctions.org
>                     <http://org.apache.spark.rdd.PairRDDFunctions.org>>$apache$__spark$rdd$PairRDDFunctions$$__writeShard$1(PairRDDFunctions.__scala:712)
>
>                               at
>                     org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730)
>                               at
>                     org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730)
>                               at
>                     org.apache.spark.scheduler.__ResultTask.runTask(ResultTask.__scala:111)
>                               at
>                     org.apache.spark.scheduler.__Task.run(Task.scala:51)
>                               at
>                     org.apache.spark.executor.__Executor$TaskRunner.run(__Executor.scala:187)
>                               at
>                     java.util.concurrent.__ThreadPoolExecutor.runWorker(__ThreadPoolExecutor.java:1145)
>                               at
>                     java.util.concurrent.__ThreadPoolExecutor$Worker.run(__ThreadPoolExecutor.java:615)
>                               at java.lang.Thread.run(Thread.__java:724)
>
>
>                     Cheers,
>                     --
>                     Jianshi Huang
>
>                     LinkedIn: jianshi
>                     Twitter: @jshuang
>                     Github & Blog: http://huangjs.github.com/
>
>
>
>
>             --
>             Jianshi Huang
>
>             LinkedIn: jianshi
>             Twitter: @jshuang
>             Github & Blog: http://huangjs.github.com/
>
>
>
>
>         --
>         Jianshi Huang
>
>         LinkedIn: jianshi
>         Twitter: @jshuang
>         Github & Blog: http://huangjs.github.com/
>
>
>
>
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/

Mime
View raw message