accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "mohit.kaushik" <mohit.kaus...@orkash.com>
Subject Slow ReadProcessor (Degrade in performance)
Date Mon, 09 Nov 2015 12:12:25 GMT

I have a application which query/inserts data in to Accumulo 
continuously. But the ingestion performance degrades after a day and 
continuously degrades by a considerable factor. I scanned all the hadoop 
and Accumulo logs but only found a warning in my two tservers logs after 
some inserts regularly.

[hdfs.DFSClient] WARN : Slow ReadProcessor read fields took 30783ms 
(threshold=30000ms); ack: seqno: 427381 reply: SUCCESS reply: SUCCESS 
reply: SUCCESS downstreamAckTimeNanos: 66816204 flag: 0 flag: 0 flag: 0, 
targets: 
[DatanodeInfoWithStorage[192.168.10.122:50010,DS-188489f9-89d3-40bd-9d20-9db358d644c9,DISK],

DatanodeInfoWithStorage[192.168.10.123:50010,DS-804290f9-edd0-45a4-ad2d-bb1bd336a348,DISK],

DatanodeInfoWithStorage[192.168.10.121:50010,DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11,DISK]]



After monitoring the query time, It seems to be increased(for ex from 30 
ms to 3 sec)after a day and continues. I also get the connection reset 
exception regularly in Tserver logs in both read and write operations. 
Which I read that it comes when a connection is closed before scan or 
write completes. But how does it affect?

015-11-08 11:25:23,272 [util.CustomNonBlockingServer$CustomFrameBuffer] 
WARN : Got an IOException in internalRead!
java.io.IOException: Connection reset by peer
         at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
         at 
org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:141)
         at 
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:537)
         at 
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
         at 
org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
         at 
org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:227)
         at 
org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:183)



  Please provide your inputs.

-Mohit Kaushik



Mime
View raw message