Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of azuryyyu@gmail.com designates
 209.85.215.50 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAMVC6RN2_pRCyXK5kxhogB7QeO-cewTFwiJ7x3p1Nz9Npwy7YA@mail.gmail.com>
References: 
 <CAOT3TWom29tMOaBsMbCnEWv9HQuv1ovHqXKz1AojtARZS5hXeg@mail.gmail.com>
	<CAMVC6RN_B4bQFQRvqoVPy6M0FkWwaz-hEhHHh6yysudbPuZDXw@mail.gmail.com>
	<CAMVC6RO6QtF=wPr-oF-t3wFzX4pMO0x4rWB57-7X2XtFVwSx3g@mail.gmail.com>
	<CAOT3TWqYi1pcvXQYT4MWP_F7wn+nkMd1B25zC-t59iE5Jm4vag@mail.gmail.com>
	<CAMVC6RPJj1oGzXAxKcY8A_1RRn8W6NUy07d78wZfmZSzaCrayw@mail.gmail.com>
	<CAOT3TWqbmtAFgwA=xfcyq8c=vskyKRe_S6F3tWSRE+4W9h7MFQ@mail.gmail.com>
	<CAMVC6RP6cPLGsF2wHpFOSwwCLcnVuncMHK7nfH_tWfKqRb_hCg@mail.gmail.com>
	<CAOT3TWpiQC680X6PeXChzN3eoGVjqex1ZuNmi6WSzjSRt92R7A@mail.gmail.com>
	<CAMVC6RN2_pRCyXK5kxhogB7QeO-cewTFwiJ7x3p1Nz9Npwy7YA@mail.gmail.com>
Date: Sun, 23 Dec 2012 09:50:12 +0800
Message-ID: 
 <CALr1C9qNyKo2rwGxWWixv=3VAonaHRuPo6JON26k=-8Lh55Y9w@mail.gmail.com>
Subject: Re: responseTooSlow
From: Azuryy Yu <azuryyyu@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=bcaec54ee77624c7fb04d17b4d1c

--bcaec54ee77624c7fb04d17b4d1c
Content-Type: text/plain; charset=ISO-8859-1

I am sure you have a long gc, please monitor your gc log.
On Dec 22, 2012 8:14 PM, "Mohammad Tariq" <dontariq@gmail.com> wrote:

> yeah
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>
>
> On Sat, Dec 22, 2012 at 6:57 AM, Mohit Anchlia <mohitanchlia@gmail.com
> >wrote:
>
> > You mean batch multiple put?
> >
> > On Fri, Dec 21, 2012 at 4:16 PM, Mohammad Tariq <dontariq@gmail.com>
> > wrote:
> >
> > > It might be the RS which could not complete the operation in time. The
> > > appropriate way to find out is to monitor that RS's metrics and see if
> > > anything unusual is happening there. What type of keys are you using?
> It
> > is
> > > time-series data?You might be a victim of RS hotspotting in that case
> or
> > > perhaps some other processes are eating up resources there. Try using
> > > "put(List<Put> puts)" instead of "put(Put put)" and see if it makes any
> > > difference.
> > >
> > > I'm afraid, I can't say anything with 100% confidence as there could be
> > 'n'
> > > reasons which are not traceable from here. Some of the possible reasons
> > > could be :
> > > hotspotting region
> > > too much I/O wait due to Swapping
> > > overloaded disk
> > > slowness due to high cpu consumption
> > >
> > > Best Regards,
> > > Tariq
> > > +91-9741563634
> > > https://mtariq.jux.com/
> > >
> > >
> > > On Sat, Dec 22, 2012 at 5:23 AM, Mohit Anchlia <mohitanchlia@gmail.com
> > > >wrote:
> > >
> > > > I am just doing a put. This operation generally takes 10ms but in
> this
> > > case
> > > > it took more than 10sec. Nothing out of ordinary in the logs
> > > >
> > > > On Fri, Dec 21, 2012 at 3:26 PM, Mohammad Tariq <dontariq@gmail.com>
> > > > wrote:
> > > >
> > > > > what exactly is the operation your trying to do?how is your
> network's
> > > > > health?is swapping too high at RS side?anything odd in your RS
> logs?
> > > > >
> > > > > Best Regards,
> > > > > Tariq
> > > > > +91-9741563634
> > > > > https://mtariq.jux.com/
> > > > >
> > > > >
> > > > > On Sat, Dec 22, 2012 at 4:36 AM, Mohit Anchlia <
> > mohitanchlia@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > I looked at that link, but couldn't find anything useful. How do
> I
> > > > check
> > > > > if
> > > > > > it was client who didn't write data within that time or if it was
> > > > region
> > > > > > server that didn't finish operation in time.
> > > > > >
> > > > > > On Fri, Dec 21, 2012 at 2:54 PM, Mohammad Tariq <
> > dontariq@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > The socket through which your client is communicating is
> getting
> > > > closed
> > > > > > > before the operation could get finished. May be it is taking
> > longer
> > > > > than
> > > > > > > usual or something.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Tariq
> > > > > > > +91-9741563634
> > > > > > > https://mtariq.jux.com/
> > > > > > >
> > > > > > >
> > > > > > >  On Sat, Dec 22, 2012 at 4:08 AM, Mohammad Tariq <
> > > dontariq@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Mohit,
> > > > > > > >
> > > > > > > >       You might this link<
> > > > > > > http://hbase.apache.org/book/ops.monitoring.html>useful.
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Tariq
> > > > > > > > +91-9741563634
> > > > > > > > https://mtariq.jux.com/
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Dec 22, 2012 at 2:09 AM, Mohit Anchlia <
> > > > > mohitanchlia@gmail.com
> > > > > > > >wrote:
> > > > > > > >
> > > > > > > >> Could someone help me understand what this really means. Is
> > this
> > > > the
> > > > > > > >> network transfer taking long from client -> server or region
> > > > server
> > > > > > > taking
> > > > > > > >> long time writing to the memory?
> > > > > > > >>
> > > > > > > >> 2012-12-21 10:54:21,980 WARN
> > org.apache.hadoop.ipc.HBaseServer:
> > > > > > > >> (responseTooSlow): {"processingtimems":135652,"call":"multi(
> > > > > > > >> org.apache.hadoop.hbase.client.MultiAction@28338472), rpc
> > > > > version=1,
> > > > > > > >> client
> > > > > > > >> version=29, methodsFingerPrint=54742778","client":"
> > > > 10.18.3.80:48218
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ","starttimems":1356115926326,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
> > > > > > > >> 2012-12-21 10:54:21,985 WARN
> > org.apache.hadoop.ipc.HBaseServer:
> > > > IPC
> > > > > > > Server
> > > > > > > >> handler 26 on 60020 caught:
> > > > java.nio.channels.ClosedChannelException
> > > > > > > >>         at
> > > > > > > >>
> > > > > >
> > > >
> > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
> > > > > > > >>         at
> > > > > > > sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> > > > > > > >>         at
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
> > > > > > > >>         at
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
> > > > > > > >>         at
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
> > > > > > > >>         at
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
> > > > > > > >>         at
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--bcaec54ee77624c7fb04d17b4d1c--