accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Kapoor <a...@koanect.com>
Subject Log exception in unresponsive tablet server
Date Thu, 06 Mar 2014 20:14:33 GMT
Hi,

  We have been using Accumulo 1.5.0 for a little while now and bugging Eric.
  Moving the discussion to public mailing list hopefully will benefit others.

  Some background:

  1. We are running 100 tablet servers with 3 zookeepers
  2. About 7 tables with 203M entries (so far)
  3. We are running these on Amazon EC2

  Noticed one of the tablet servers being unresponsive. Here are the exceptions
  in the log on the tablet server:

  A. 2014-03-06 11:02:38,701 [file.BloomFilterLayer] ERROR: Thread "bloom-loader 3194" died
File /accumulo/tables/8/t-00001tv/F0003cul.rf is closed
java.lang.IllegalStateException: File /accumulo/tables/8/t-00001tv/F0003cul.rf is closed
        at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getBCFile(CachableBlockFile.java:251)
        at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.access$000(CachableBlockFile.java:143)
        at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader$MetaBlockLoader.get(CachableBlockFile.java:212)
        at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getBlock(CachableBlockFile.java:313)
        at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getMetaBlock(CachableBlockFile.java:367)
        at org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getMetaBlock(CachableBlockFile.java:143)
        at org.apache.accumulo.core.file.rfile.RFile$Reader.getMetaStore(RFile.java:964)
        at org.apache.accumulo.core.file.BloomFilterLayer$BloomFilterLoader$1.run(BloomFilterLayer.java:198)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:42)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
        at java.lang.Thread.run(Thread.java:744)

 B. 2014-03-06 09:18:08,165 [util.TServerUtils$THsHaServer] WARN : Got an IOException in internalRead!
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
        at org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:141)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:515)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:305)
        at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:202)
        at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:198)
        at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:154)
2014-03-06 09:18:08,165 [util.TServerUtils$THsHaServer] WARN : Got an IOException in internalRead!

  (B) is multiple times over the last few days.

  Any insight whether we should be be concerned about these and what the fix
  should be much appreciated.

  regards

Amit

Mime
View raw message