Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
MIME-Version: 1.0
In-Reply-To: <56408DA9.3060700@orkash.com>
References: <56408DA9.3060700@orkash.com>
Date: Mon, 16 Nov 2015 16:05:58 -0500
Message-ID: 
 <CAGUtCHoEeeOCfLg4XBc1BmW-=_gcUaGBX=fTJ2BPH-2xsF8kKQ@mail.gmail.com>
Subject: Re: Slow ReadProcessor (Degrade in performance)
From: Keith Turner <keith@deenlo.com>
To: user@accumulo.apache.org
Content-Type: multipart/alternative; boundary=e89a8f6431309a9ba30524aec6bf

--e89a8f6431309a9ba30524aec6bf
Content-Type: text/plain; charset=UTF-8

Mohit,

Were you able to figure this problem out?   Do you have enough info to know
if its a hdfs, Accumulo, network, or OS problem?

Keith

On Mon, Nov 9, 2015 at 7:12 AM, mohit.kaushik <mohit.kaushik@orkash.com>
wrote:

>
> I have a application which query/inserts data in to Accumulo continuously.
> But the ingestion performance degrades after a day and continuously
> degrades by a considerable factor. I scanned all the hadoop and Accumulo
> logs but only found a warning in my two tservers logs after some inserts
> regularly.
>
> [hdfs.DFSClient] WARN : Slow ReadProcessor read fields took 30783ms
> (threshold=30000ms); ack: seqno: 427381 reply: SUCCESS reply: SUCCESS
> reply: SUCCESS downstreamAckTimeNanos: 66816204 flag: 0 flag: 0 flag: 0,
> targets: [DatanodeInfoWithStorage[192.168.10.122:50010,DS-188489f9-89d3-40bd-9d20-9db358d644c9,DISK],
> DatanodeInfoWithStorage[192.168.10.123:50010,DS-804290f9-edd0-45a4-ad2d-bb1bd336a348,DISK],
> DatanodeInfoWithStorage[192.168.10.121:50010,DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11,DISK]]
>
>
> After monitoring the query time, It seems to be increased(for ex from 30
> ms to 3 sec)after a day and continues. I also get the connection reset
> exception regularly in Tserver logs in both read and write operations.
> Which I read that it comes when a connection is closed before scan or write
> completes. But how does it affect?
>
> 015-11-08 11:25:23,272 [util.CustomNonBlockingServer$CustomFrameBuffer]
> WARN : Got an IOException in internalRead!
> java.io.IOException: Connection reset by peer
>         at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
>         at
> org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:141)
>         at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:537)
>         at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
>         at
> org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
>         at
> org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:227)
>         at
> org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:183)
>
>
>  Please provide your inputs.
>
> -Mohit Kaushik
>
>
>

--e89a8f6431309a9ba30524aec6bf
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Mohit,<br><br></div>Were you able to figure this=
 problem out?=C2=A0=C2=A0 Do you have enough info to know if its a hdfs, Ac=
cumulo, network, or OS problem?<br><br></div>Keith<br><div><div><div><div c=
lass=3D"gmail_extra"><br><div class=3D"gmail_quote">On Mon, Nov 9, 2015 at =
7:12 AM, mohit.kaushik <span dir=3D"ltr">&lt;<a href=3D"mailto:mohit.kaushi=
k@orkash.com" target=3D"_blank">mohit.kaushik@orkash.com</a>&gt;</span> wro=
te:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-=
left:1px #ccc solid;padding-left:1ex">
 =20

   =20
 =20
  <div text=3D"#000000" bgcolor=3D"#FFFFFF">
    <br>
    I have a application which query/inserts data in to Accumulo
    continuously. But the ingestion performance degrades after a day and
    continuously degrades by a considerable factor. I scanned all the
    hadoop and Accumulo logs but only found a warning in my two tservers
    logs after some inserts regularly.<br>
    <br>
    [hdfs.DFSClient] WARN : Slow ReadProcessor read fields took 30783ms
    (threshold=3D30000ms); ack: seqno: 427381 reply: SUCCESS reply:
    SUCCESS reply: SUCCESS downstreamAckTimeNanos: 66816204 flag: 0
    flag: 0 flag: 0, targets:
    [DatanodeInfoWithStorage[<a href=3D"http://192.168.10.122:50010" target=
=3D"_blank">192.168.10.122:50010</a>,DS-188489f9-89d3-40bd-9d20-9db358d644c=
9,DISK],
    DatanodeInfoWithStorage[<a href=3D"http://192.168.10.123:50010" target=
=3D"_blank">192.168.10.123:50010</a>,DS-804290f9-edd0-45a4-ad2d-bb1bd336a34=
8,DISK],
    DatanodeInfoWithStorage[<a href=3D"http://192.168.10.121:50010" target=
=3D"_blank">192.168.10.121:50010</a>,DS-dd6d6a25-122f-4958-a20b-4ccb82f49f1=
1,DISK]]=C2=A0=C2=A0
    <br>
    <br>
    After monitoring the query time, It seems to be increased(for ex
    from 30 ms to 3 sec)after a day and continues. I also get the
    connection reset exception regularly in Tserver logs in both read
    and write operations. Which I read that it comes when a connection
    is closed before scan or write completes. But how does it affect?<br>
    <br>
    015-11-08 11:25:23,272
    [util.CustomNonBlockingServer$CustomFrameBuffer] WARN : Got an
    IOException in internalRead!<br>
    java.io.IOException: Connection reset by peer<br>
    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.nio.ch.FileDispatcher=
Impl.read0(Native Method)<br>
    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at
    sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)<br>
    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.nio.ch.IOUtil.readInt=
oNativeBuffer(IOUtil.java:223)<br>
    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.nio.ch.IOUtil.read(IO=
Util.java:197)<br>
    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at
    sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)<br>
    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at
org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java=
:141)<br>
    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead=
(AbstractNonblockingServer.java:537)<br>
    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(Abstrac=
tNonblockingServer.java:338)<br>
    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at
org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.han=
dleRead(AbstractNonblockingServer.java:203)<br>
    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at
org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.=
select(CustomNonBlockingServer.java:227)<br>
    =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at
org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.=
run(CustomNonBlockingServer.java:183)
    <br>
    <br>
    =C2=A0Please provide your inputs. <br><span class=3D"HOEnZb"><font colo=
r=3D"#888888">
    <br>
    -Mohit Kaushik<br>
    <div><br>
      <br>
      <font face=3D"Arial">
        </font></div>
  </font></span></div>

</blockquote></div><br></div></div></div></div></div>

--e89a8f6431309a9ba30524aec6bf--