hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nanheng Wu <nanhen...@gmail.com>
Subject Re: What's the region server doing?
Date Wed, 02 Mar 2011 01:22:35 GMT
Thanks man I'll try that and post back when I find something. BTW, I
ran the script to set the memstore flush size on .META., now I am
seeing a lot less writing to HDFS from the .META RS and less
compaction, unfortunately it's still low. :(

On Tue, Mar 1, 2011 at 5:15 PM, Jean-Daniel Cryans <jdcryans@apache.org> wrote:
> In that specific jstack it's doing nothing at all, but keep in mind
> that it's only a snapshot of a precise moment in time. Try jstack'ing
> a few times and at some point you should see the threads named like
> "IPC Server handler xx on 60020" (where xx is a number) showing bigger
> stack traces with HRegionServer doing stuff like get, next, put, etc
>
> You should also try scanning '.META.' from the shell and if it's slow,
> do the jstack'ing at the same time.
>
> J-D
>
> On Tue, Mar 1, 2011 at 5:07 PM, Nanheng Wu <nanhengwu@gmail.com> wrote:
>> My cluster (10 nodes, hbase-0.20.6 + hadoop 0.20.2) is very very slow
>> for any operation like disable table or delete. Master's thread dump
>> says they are blocked by the metaScanner thread. When I looked at the
>> log file on the .META RS there are no outputs at all! (INFO debug
>> level). J-D has been helping me on this, we pretty much figured out
>> that RegionManager.metaScanner is the culprit, because it's taking
>> around 25 minutes to scan 8K rows. What I don't get is what the region
>> server is actually doing during this time. There's no request at all
>> on the cluster, no RS splits either because we just use a MR job to
>> output HFiles and never write again.
>> J-D has been really really helpful, but I feel like I took too much of
>> his time. Below is the thread dump of the .META RS during the time
>> when disables command are blocked on meta scanner, can someone help me
>> figure out what the server is doing, is it running any thread at all?
>> Thank you!
>>
>> http://pastebin.com/CZQAywq3
>>
>

Mime
View raw message