hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com>
Subject Re: Confusion with Actual Number of Regions in HBase
Date Thu, 21 Feb 2013 04:18:48 GMT
The two RS could have died due to GC or some connectivity issues with ZK.
 Checking the logs would help us get a clear idea.

REgards
Ram

On Thu, Feb 21, 2013 at 7:09 AM, Joarder KAMAL <joarderm@gmail.com> wrote:

> Hi Ted,
>
> After login into the HBase UI I found two of my region servers were dead.
> Don't have any clue why. Master is still up and running for the past 4
> days.
> Memory is still running pretty low without any workload and two region
> servers!!
>
> --------------------------------------------------
>
> root@hdb1-01:/home/ubuntu# jps
> 1064 NameNode
> 1578 JobTracker
> 10770 Jps
> 2481 HRegionServer
> 2261 HMaster
> 1280 DataNode
> 1769 TaskTracker
> 2197 HQuorumPeer
> 1480 SecondaryNameNode
> --------------------------------------------------
> root@hdb1-02:/home/ubuntu# jps
> 1492 HRegionServer
> 1315 TaskTracker
> 1137 DataNode
> 3290 Jps
> --------------------------------------------------
> root@hdb1-03:/home/ubuntu# jps
> 1527 HRegionServer
> 1171 DataNode
> 7475 Jps
> 1349 TaskTracker
>
> --------------------------------------------------
> PID found 2197, 2261 and 2481 for 'hbase'
> --------------------------------------------------
> root@hdb1-01:/home/ubuntu# ps axo pid,ppid,rss,vsz | grep 2197
>  2197     1 52248 1938732
> --------------------------------------------------
> root@hdb1-01:/home/ubuntu# ps axo pid,ppid,rss,vsz | grep 2261
>  2261     1 108724 2007912
> --------------------------------------------------
> root@hdb1-01:/home/ubuntu# ps axo pid,ppid,rss,vsz | grep 2481
>  2481     1 686252 2031376
> --------------------------------------------------
> root@hdb1-01:/home/ubuntu# free -m
>              total       used       free     shared    buffers     cached
> Mem:     1652       1616         36          0         51         305
>
> -/+ buffers/cache:       1259        393
> Swap:          895          8        887
>
>
> Thanks a lot again ...
>
>
> Regards,
> Joarder Kamal
>
>
>
> On 21 February 2013 11:08, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > What do you see when you use 'ps aux | grep <pid>' command ?
> > A sample is shown below:
> >
> > USER              PID  %CPU %MEM      VSZ    RSS   TT  STAT STARTED
> >  TIME COMMAND
> > tyu             16468 232.2  1.4  4974136 240528 s002  R+    4:05PM
> > 2:11.15
> >
> >
> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/javadoc
> > -J-Xmx2048m @options @packages
> >
> > Please give us the VSZ and RSS readings.
> >
> > Thanks
> >
> > On Wed, Feb 20, 2013 at 3:56 PM, Joarder KAMAL <joarderm@gmail.com>
> wrote:
> >
> > > Sorry may be I didn't clarify the question properly.
> > >
> > > Q2. As per the default setting maxHeapMB size is 991, so in saturation
> > > condition HBase can't use more RAM (even if it is available, for
> example
> > > the system memory size is 4GB) than this defined size, is it right?
> > >
> > >
> > > Many thanks again...
> > >
> > > On 21 February 2013 10:48, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > I am not familiar with monitoring tools on EC2.
> > > >
> > > > You can use Ganglia to monitor hadoop / HBase.
> > > >
> > > > Cheers
> > > >
> > > > On Wed, Feb 20, 2013 at 3:42 PM, Joarder KAMAL <joarderm@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Ted,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > Q1. Cleared now. Sorry I didn't count the -ROOT- and .META. tables
> > > > > Q2. Not clear. Could explain a bit more? I am using the default
> > > setting.
> > > > >
> > > > > Thanks again.
> > > > >
> > > > >
> > > > > On 21 February 2013 10:36, Ted Yu <yuzhihong@gmail.com> wrote:
> > > > >
> > > > > > bq. but when I click to view the table details it shows there
> are 5
> > > > > > regions.
> > > > > >
> > > > > > Did you include -ROOT- and .META. tables above ?
> > > > > >
> > > > > > bq. as you can see the maxHeapMB size is 991
> > > > > >
> > > > > > Attachment is automatically stripped from your email. FYI
> > > > > >
> > > > > > On Wed, Feb 20, 2013 at 3:26 PM, Joarder KAMAL <
> joarderm@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Dear All,
> > > > > > >
> > > > > > > I am currently running a 3 node HBase cluster with Amazon
EC2
> > small
> > > > > > > instance (1CPU, 1.7GB Mem). After running a simple workload
> (type
> > > A)
> > > > > > using
> > > > > > > YCSB benchmarking tool, I noticed in the Web UI that the
number
> > of
> > > > > > regions
> > > > > > > in the 3 region servers are 7 but when I click to view
the
> table
> > > > > details
> > > > > > it
> > > > > > > shows there are 5 regions. I am a bit confuse. Does the
Web UI
> > > > showing
> > > > > > > wrong results? How can I check region details from the
shell/is
> > > there
> > > > > any
> > > > > > > other command line tools for that? I am using Hbase 0.94.3
(as
> it
> > > is
> > > > > the
> > > > > > > latest version compatible with YCSB at this moment) with
> > > > Hadoop-1.0.4.
> > > > > > >
> > > > > > > Another thing, as you can see the maxHeapMB size is 991
> (default
> > > > > setting)
> > > > > > > then whenever I am running workloads the server free memory
> > status
> > > is
> > > > > > > terrible - almost 0MB for 1M/10M transactions
> (YCSB/Workload-A).
> > My
> > > > > > > question is if the maxHeapMB is fixed at 1GB then why I
am
> > loosing
> > > > > extra
> > > > > > > 700MB memory? How can I monitor cpu/memory/io usage for
> > individual
> > > > > > cluster
> > > > > > > components (HBase/Hadoop/MapRed/ZK, etc.)? Is there any
tool
> > > > available
> > > > > > with
> > > > > > > HBase?
> > > > > > >
> > > > > > > Thanks a lot.
> > > > > > >
> > > > > > >
> > > > > > > [image: Inline images 1]
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > > Joarder Kamal
> > > > > > >
> > > > > > > ------------------------------------------
> > > > > > > PhD Student
> > > > > > > Gippsland School of Information Technology,
> > > > > > > Faculty of Information Technology,
> > > > > > > MONASH University
> > > > > > > Churchill VIC, Australia. 3842
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message