hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
Subject Re: responseTooSlow
Date Sat, 22 Dec 2012 14:33:57 GMT
How many threads are doing the puts?
Is there some flushes happening parallely?  Are there too many scans going
on in paralle.

Increasing the Xceivers could solve this problem if they are of shortage.

Regards
Ram

On Sat, Dec 22, 2012 at 5:42 PM, Mohammad Tariq <dontariq@gmail.com> wrote:

> yeah
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>
>
> On Sat, Dec 22, 2012 at 6:57 AM, Mohit Anchlia <mohitanchlia@gmail.com
> >wrote:
>
> > You mean batch multiple put?
> >
> > On Fri, Dec 21, 2012 at 4:16 PM, Mohammad Tariq <dontariq@gmail.com>
> > wrote:
> >
> > > It might be the RS which could not complete the operation in time. The
> > > appropriate way to find out is to monitor that RS's metrics and see if
> > > anything unusual is happening there. What type of keys are you using?
> It
> > is
> > > time-series data?You might be a victim of RS hotspotting in that case
> or
> > > perhaps some other processes are eating up resources there. Try using
> > > "put(List<Put> puts)" instead of "put(Put put)" and see if it makes any
> > > difference.
> > >
> > > I'm afraid, I can't say anything with 100% confidence as there could be
> > 'n'
> > > reasons which are not traceable from here. Some of the possible reasons
> > > could be :
> > > hotspotting region
> > > too much I/O wait due to Swapping
> > > overloaded disk
> > > slowness due to high cpu consumption
> > >
> > > Best Regards,
> > > Tariq
> > > +91-9741563634
> > > https://mtariq.jux.com/
> > >
> > >
> > > On Sat, Dec 22, 2012 at 5:23 AM, Mohit Anchlia <mohitanchlia@gmail.com
> > > >wrote:
> > >
> > > > I am just doing a put. This operation generally takes 10ms but in
> this
> > > case
> > > > it took more than 10sec. Nothing out of ordinary in the logs
> > > >
> > > > On Fri, Dec 21, 2012 at 3:26 PM, Mohammad Tariq <dontariq@gmail.com>
> > > > wrote:
> > > >
> > > > > what exactly is the operation your trying to do?how is your
> network's
> > > > > health?is swapping too high at RS side?anything odd in your RS
> logs?
> > > > >
> > > > > Best Regards,
> > > > > Tariq
> > > > > +91-9741563634
> > > > > https://mtariq.jux.com/
> > > > >
> > > > >
> > > > > On Sat, Dec 22, 2012 at 4:36 AM, Mohit Anchlia <
> > mohitanchlia@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > I looked at that link, but couldn't find anything useful. How
do
> I
> > > > check
> > > > > if
> > > > > > it was client who didn't write data within that time or if it
was
> > > > region
> > > > > > server that didn't finish operation in time.
> > > > > >
> > > > > > On Fri, Dec 21, 2012 at 2:54 PM, Mohammad Tariq <
> > dontariq@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > The socket through which your client is communicating is
> getting
> > > > closed
> > > > > > > before the operation could get finished. May be it is taking
> > longer
> > > > > than
> > > > > > > usual or something.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Tariq
> > > > > > > +91-9741563634
> > > > > > > https://mtariq.jux.com/
> > > > > > >
> > > > > > >
> > > > > > >  On Sat, Dec 22, 2012 at 4:08 AM, Mohammad Tariq <
> > > dontariq@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Mohit,
> > > > > > > >
> > > > > > > >       You might this link<
> > > > > > > http://hbase.apache.org/book/ops.monitoring.html>useful.
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Tariq
> > > > > > > > +91-9741563634
> > > > > > > > https://mtariq.jux.com/
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Dec 22, 2012 at 2:09 AM, Mohit Anchlia <
> > > > > mohitanchlia@gmail.com
> > > > > > > >wrote:
> > > > > > > >
> > > > > > > >> Could someone help me understand what this really
means. Is
> > this
> > > > the
> > > > > > > >> network transfer taking long from client ->
server or region
> > > > server
> > > > > > > taking
> > > > > > > >> long time writing to the memory?
> > > > > > > >>
> > > > > > > >> 2012-12-21 10:54:21,980 WARN
> > org.apache.hadoop.ipc.HBaseServer:
> > > > > > > >> (responseTooSlow): {"processingtimems":135652,"call":"multi(
> > > > > > > >> org.apache.hadoop.hbase.client.MultiAction@28338472),
rpc
> > > > > version=1,
> > > > > > > >> client
> > > > > > > >> version=29, methodsFingerPrint=54742778","client":"
> > > > 10.18.3.80:48218
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ","starttimems":1356115926326,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
> > > > > > > >> 2012-12-21 10:54:21,985 WARN
> > org.apache.hadoop.ipc.HBaseServer:
> > > > IPC
> > > > > > > Server
> > > > > > > >> handler 26 on 60020 caught:
> > > > java.nio.channels.ClosedChannelException
> > > > > > > >>         at
> > > > > > > >>
> > > > > >
> > > >
> > sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
> > > > > > > >>         at
> > > > > > > sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> > > > > > > >>         at
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
> > > > > > > >>         at
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
> > > > > > > >>         at
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
> > > > > > > >>         at
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
> > > > > > > >>         at
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message