hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
Subject Re: responseTooSlow
Date Sat, 22 Dec 2012 16:47:55 GMT
For seeing if flushing is happening you need to see the logs.
Scans you need to check your app.

Regards
Ram

On Sat, Dec 22, 2012 at 9:41 PM, Mohit Anchlia <mohitanchlia@gmail.com>wrote:

> What would be the best way to check? Region Logs?
>
> Sent from my iPhone
>
> On Dec 22, 2012, at 6:33 AM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > How many threads are doing the puts?
> > Is there some flushes happening parallely?  Are there too many scans
> going
> > on in paralle.
> >
> > Increasing the Xceivers could solve this problem if they are of shortage.
> >
> > Regards
> > Ram
> >
> > On Sat, Dec 22, 2012 at 5:42 PM, Mohammad Tariq <dontariq@gmail.com>
> wrote:
> >
> >> yeah
> >>
> >> Best Regards,
> >> Tariq
> >> +91-9741563634
> >> https://mtariq.jux.com/
> >>
> >>
> >> On Sat, Dec 22, 2012 at 6:57 AM, Mohit Anchlia <mohitanchlia@gmail.com
> >>> wrote:
> >>
> >>> You mean batch multiple put?
> >>>
> >>> On Fri, Dec 21, 2012 at 4:16 PM, Mohammad Tariq <dontariq@gmail.com>
> >>> wrote:
> >>>
> >>>> It might be the RS which could not complete the operation in time. The
> >>>> appropriate way to find out is to monitor that RS's metrics and see
if
> >>>> anything unusual is happening there. What type of keys are you using?
> >> It
> >>> is
> >>>> time-series data?You might be a victim of RS hotspotting in that case
> >> or
> >>>> perhaps some other processes are eating up resources there. Try using
> >>>> "put(List<Put> puts)" instead of "put(Put put)" and see if it
makes
> any
> >>>> difference.
> >>>>
> >>>> I'm afraid, I can't say anything with 100% confidence as there could
> be
> >>> 'n'
> >>>> reasons which are not traceable from here. Some of the possible
> reasons
> >>>> could be :
> >>>> hotspotting region
> >>>> too much I/O wait due to Swapping
> >>>> overloaded disk
> >>>> slowness due to high cpu consumption
> >>>>
> >>>> Best Regards,
> >>>> Tariq
> >>>> +91-9741563634
> >>>> https://mtariq.jux.com/
> >>>>
> >>>>
> >>>> On Sat, Dec 22, 2012 at 5:23 AM, Mohit Anchlia <
> mohitanchlia@gmail.com
> >>>>> wrote:
> >>>>
> >>>>> I am just doing a put. This operation generally takes 10ms but in
> >> this
> >>>> case
> >>>>> it took more than 10sec. Nothing out of ordinary in the logs
> >>>>>
> >>>>> On Fri, Dec 21, 2012 at 3:26 PM, Mohammad Tariq <dontariq@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> what exactly is the operation your trying to do?how is your
> >> network's
> >>>>>> health?is swapping too high at RS side?anything odd in your
RS
> >> logs?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Tariq
> >>>>>> +91-9741563634
> >>>>>> https://mtariq.jux.com/
> >>>>>>
> >>>>>>
> >>>>>> On Sat, Dec 22, 2012 at 4:36 AM, Mohit Anchlia <
> >>> mohitanchlia@gmail.com
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> I looked at that link, but couldn't find anything useful.
How do
> >> I
> >>>>> check
> >>>>>> if
> >>>>>>> it was client who didn't write data within that time or
if it was
> >>>>> region
> >>>>>>> server that didn't finish operation in time.
> >>>>>>>
> >>>>>>> On Fri, Dec 21, 2012 at 2:54 PM, Mohammad Tariq <
> >>> dontariq@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> The socket through which your client is communicating
is
> >> getting
> >>>>> closed
> >>>>>>>> before the operation could get finished. May be it is
taking
> >>> longer
> >>>>>> than
> >>>>>>>> usual or something.
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>> Tariq
> >>>>>>>> +91-9741563634
> >>>>>>>> https://mtariq.jux.com/
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Sat, Dec 22, 2012 at 4:08 AM, Mohammad Tariq <
> >>>> dontariq@gmail.com
> >>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hello Mohit,
> >>>>>>>>>
> >>>>>>>>>      You might this link<
> >>>>>>>> http://hbase.apache.org/book/ops.monitoring.html>useful.
> >>>>>>>>>
> >>>>>>>>> Best Regards,
> >>>>>>>>> Tariq
> >>>>>>>>> +91-9741563634
> >>>>>>>>> https://mtariq.jux.com/
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Sat, Dec 22, 2012 at 2:09 AM, Mohit Anchlia <
> >>>>>> mohitanchlia@gmail.com
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Could someone help me understand what this really
means. Is
> >>> this
> >>>>> the
> >>>>>>>>>> network transfer taking long from client ->
server or region
> >>>>> server
> >>>>>>>> taking
> >>>>>>>>>> long time writing to the memory?
> >>>>>>>>>>
> >>>>>>>>>> 2012-12-21 10:54:21,980 WARN
> >>> org.apache.hadoop.ipc.HBaseServer:
> >>>>>>>>>> (responseTooSlow): {"processingtimems":135652,"call":"multi(
> >>>>>>>>>> org.apache.hadoop.hbase.client.MultiAction@28338472),
rpc
> >>>>>> version=1,
> >>>>>>>>>> client
> >>>>>>>>>> version=29, methodsFingerPrint=54742778","client":"
> >>>>> 10.18.3.80:48218
> >>
> ","starttimems":1356115926326,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
> >>>>>>>>>> 2012-12-21 10:54:21,985 WARN
> >>> org.apache.hadoop.ipc.HBaseServer:
> >>>>> IPC
> >>>>>>>> Server
> >>>>>>>>>> handler 26 on 60020 caught:
> >>>>> java.nio.channels.ClosedChannelException
> >>>>>>>>>>        at
> >>>
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
> >>>>>>>>>>        at
> >>>>>>>> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> >>>>>>>>>>        at
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
> >>>>>>>>>>        at
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
> >>>>>>>>>>        at
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
> >>>>>>>>>>        at
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
> >>>>>>>>>>        at
> >>
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message