accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: Slow ReadProcessor (Degrade in performance)
Date Mon, 16 Nov 2015 21:05:58 GMT
Mohit,

Were you able to figure this problem out?   Do you have enough info to know
if its a hdfs, Accumulo, network, or OS problem?

Keith

On Mon, Nov 9, 2015 at 7:12 AM, mohit.kaushik <mohit.kaushik@orkash.com>
wrote:

>
> I have a application which query/inserts data in to Accumulo continuously.
> But the ingestion performance degrades after a day and continuously
> degrades by a considerable factor. I scanned all the hadoop and Accumulo
> logs but only found a warning in my two tservers logs after some inserts
> regularly.
>
> [hdfs.DFSClient] WARN : Slow ReadProcessor read fields took 30783ms
> (threshold=30000ms); ack: seqno: 427381 reply: SUCCESS reply: SUCCESS
> reply: SUCCESS downstreamAckTimeNanos: 66816204 flag: 0 flag: 0 flag: 0,
> targets: [DatanodeInfoWithStorage[192.168.10.122:50010,DS-188489f9-89d3-40bd-9d20-9db358d644c9,DISK],
> DatanodeInfoWithStorage[192.168.10.123:50010,DS-804290f9-edd0-45a4-ad2d-bb1bd336a348,DISK],
> DatanodeInfoWithStorage[192.168.10.121:50010,DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11,DISK]]
>
>
> After monitoring the query time, It seems to be increased(for ex from 30
> ms to 3 sec)after a day and continues. I also get the connection reset
> exception regularly in Tserver logs in both read and write operations.
> Which I read that it comes when a connection is closed before scan or write
> completes. But how does it affect?
>
> 015-11-08 11:25:23,272 [util.CustomNonBlockingServer$CustomFrameBuffer]
> WARN : Got an IOException in internalRead!
> java.io.IOException: Connection reset by peer
>         at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
>         at
> org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:141)
>         at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:537)
>         at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
>         at
> org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
>         at
> org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:227)
>         at
> org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:183)
>
>
>  Please provide your inputs.
>
> -Mohit Kaushik
>
>
>

Mime
View raw message