hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From syed kather <in.ab...@gmail.com>
Subject Re: Cluster load
Date Fri, 27 Jul 2012 18:21:48 GMT
Thank you so much for your valuable information. I had not yet used any
monitoring tool .. can please suggest me a good monitor tool .

Syed Abdul kather
send from Samsung S3
On Jul 27, 2012 11:37 PM, "Alex Baranau" <alex.baranov.v@gmail.com> wrote:

> -rwxr-xr-x 3 root root 1993369 2012-07-26 13:59
>
> /hbase/SESSION_TIMELINE1/0a5f6fadd0435898c6f4cf11daa9895a/S_T_MTX/1566523617482885717
>
> "1993369" is the size. Oh sorry. It is 2MB, not 2GB. Yeah, that doesn't
> tell a lot. Looks like all data is in Memstore. As I said, you should try
> flushing the table, so that you can see where data was written.
>
> Of course it is always great to setup monitoring and see what is going on
> ;)
>
> Anyhow, the piece pasted above, means:
>
> table:SESSION_TIMELINE1, region: 0a5f6fadd0435898c6f4cf11daa9895a,
>  columnFamily: S_T_MTX, hfile(created by memstore flush):
> 1566523617482885717,
> size: 1993369 bytes.
>
> btw, 2MB looks weird: very small flush size (in this case, in other cases
> this may happen - long story). May be compression does very well :)
>
> Alex Baranau
> ------
> Sematext :: http://blog.sematext.com/ :: Hadoop - HBase - ElasticSearch -
> Solr
>
> On Fri, Jul 27, 2012 at 10:52 AM, syed kather <in.abdul@gmail.com> wrote:
>
> > Alex Baranau,
> >
> >     Can please tell how did you found it has 2GB of data from
> > "0a5f6fadd0435898c6f4cf11daa9895a" . I am pretty much intrested to know
> it
> > .
> >             Thanks and Regards,
> >         S SYED ABDUL KATHER
> >
> >
> >
> > On Fri, Jul 27, 2012 at 7:51 PM, Alex Baranau <alex.baranov.v@gmail.com
> > >wrote:
> >
> > > From what you posted above, I guess one of the regions
> > > (0a5f6fadd0435898c6f4cf11daa9895a,
> > > note that it has 2 files 2GB each [1], while others regions are
> "empty")
> > is
> > > getting hit with writes. You may want to run "flush 'mytable'" command
> > from
> > > hbase shell before looking at hdfs - this way you make sure your data
> is
> > > flushed to hdfs (and not hanged in Memstores).
> > >
> > > You may want to check the START/END keys of this region (via master web
> > ui
> > > or in .META.). Then you can compare with the keys generated by your
> app.
> > > This should give you some info about what's going on.
> > >
> > > Alex Baranau
> > > ------
> > > Sematext :: http://blog.sematext.com/ :: Hadoop - HBase -
> ElasticSearch
> > -
> > > Solr
> > >
> > > [1]
> > >
> > > -rwxr-xr-x 3 root root 1993369 2012-07-26 13:59
> > >
> > >
> >
> /hbase/SESSION_TIMELINE1/0a5f6fadd0435898c6f4cf11daa9895a/S_T_MTX/1566523617482885717
> > >
> > > -rwxr-xr-x 3 root root 2003372 2012-07-26 13:57
> > >
> > >
> >
> /hbase/SESSION_TIMELINE1/0a5f6fadd0435898c6f4cf11daa9895a/S_T_MTX/7665015246030620502
> > >
> > > On Fri, Jul 27, 2012 at 3:16 AM, Khang Pham <khangich@gmail.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > by node do you mean regionserver node ?
> > > >
> > > > if you referring to RegionServer node: you can go to the hbase master
> > web
> > > > interface master:65510/master.jsp to see load for each regionserver.
> > > That's
> > > > the overall load. If you want to see load per node per table, you
> will
> > > need
> > > > to query on .META. table (column: info:server)
> > > >
> > > >
> > > >
> > > > --K
> > > > On Fri, Jul 27, 2012 at 9:07 AM, Mohit Anchlia <
> mohitanchlia@gmail.com
> > > > >wrote:
> > > >
> > > > > Is there a way to see how much data does each node have per Hbase
> > > table?
> > > > >
> > > > > On Thu, Jul 26, 2012 at 5:53 PM, syed kather <in.abdul@gmail.com>
> > > wrote:
> > > > >
> > > > > > First check whether the data in hbase is consistent ... check
> this
> > by
> > > > > > running hbck (bin/hbase hbck ) If all the region is consistent
.
> > > > > > Now check no of splits in localhost:60010 for the table mention
> ..
> > > > > >  On Jul 27, 2012 4:02 AM, "Mohit Anchlia" <
> mohitanchlia@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > I added new regions and the performance didn't improve.
I think
> > it
> > > > > still
> > > > > > is
> > > > > > > the load balancing issue. I want to ensure that my rows
are
> > getting
> > > > > > > distrbuted accross cluster. What I see is this:
> > > > > > >
> > > > > > > Could you please tell me what's the best way to see the
load?
> > > > > > >
> > > > > > >
> > > > > > > [root@dsdb4 ~]# hadoop fs -lsr /hbase/SESSION_TIMELINE1/
> > > > > > >
> > > > > > > drwxr-xr-x - root root 3 2012-07-26 13:32
> > > > > > > /hbase/SESSION_TIMELINE1/8c02c8ed87e1a023ece8d8090a364641
> > > > > > >
> > > > > > > drwxr-xr-x - root root 1 2012-07-26 13:32
> > > > > > >
> > /hbase/SESSION_TIMELINE1/8c02c8ed87e1a023ece8d8090a364641/.oldlogs
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 124 2012-07-26 13:32
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/SESSION_TIMELINE1/8c02c8ed87e1a023ece8d8090a364641/.oldlogs/hlog.1343334723359
> > > > > > >
> > > > > > > drwxr-xr-x - root root 0 2012-07-26 13:32
> > > > > > >
> /hbase/SESSION_TIMELINE1/8c02c8ed87e1a023ece8d8090a364641/S_T_MTX
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 764 2012-07-26 13:32
> > > > > > >
> > > /hbase/SESSION_TIMELINE1/8c02c8ed87e1a023ece8d8090a364641/.regioninfo
> > > > > > >
> > > > > > > drwxr-xr-x - root root 3 2012-07-26 13:32
> > > > > > > /hbase/SESSION_TIMELINE1/79e03d78a784601e8daa88aa85c39854
> > > > > > >
> > > > > > > drwxr-xr-x - root root 1 2012-07-26 13:32
> > > > > > >
> > /hbase/SESSION_TIMELINE1/79e03d78a784601e8daa88aa85c39854/.oldlogs
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 124 2012-07-26 13:32
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/SESSION_TIMELINE1/79e03d78a784601e8daa88aa85c39854/.oldlogs/hlog.1343334723093
> > > > > > >
> > > > > > > drwxr-xr-x - root root 0 2012-07-26 13:32
> > > > > > >
> /hbase/SESSION_TIMELINE1/79e03d78a784601e8daa88aa85c39854/S_T_MTX
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 764 2012-07-26 13:32
> > > > > > >
> > > /hbase/SESSION_TIMELINE1/79e03d78a784601e8daa88aa85c39854/.regioninfo
> > > > > > >
> > > > > > > drwxr-xr-x - root root 3 2012-07-26 13:32
> > > > > > > /hbase/SESSION_TIMELINE1/1054fe861f199a23ebef8f49f99c4aba
> > > > > > >
> > > > > > > drwxr-xr-x - root root 1 2012-07-26 13:32
> > > > > > >
> > /hbase/SESSION_TIMELINE1/1054fe861f199a23ebef8f49f99c4aba/.oldlogs
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 124 2012-07-26 13:32
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/SESSION_TIMELINE1/1054fe861f199a23ebef8f49f99c4aba/.oldlogs/hlog.1343334723240
> > > > > > >
> > > > > > > drwxr-xr-x - root root 0 2012-07-26 13:32
> > > > > > >
> /hbase/SESSION_TIMELINE1/1054fe861f199a23ebef8f49f99c4aba/S_T_MTX
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 764 2012-07-26 13:32
> > > > > > >
> > > /hbase/SESSION_TIMELINE1/1054fe861f199a23ebef8f49f99c4aba/.regioninfo
> > > > > > >
> > > > > > > drwxr-xr-x - root root 3 2012-07-26 13:32
> > > > > > > /hbase/SESSION_TIMELINE1/04295ef1d74887a107a013810f5aa26a
> > > > > > >
> > > > > > > drwxr-xr-x - root root 1 2012-07-26 13:32
> > > > > > >
> > /hbase/SESSION_TIMELINE1/04295ef1d74887a107a013810f5aa26a/.oldlogs
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 124 2012-07-26 13:32
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/SESSION_TIMELINE1/04295ef1d74887a107a013810f5aa26a/.oldlogs/hlog.1343334723171
> > > > > > >
> > > > > > > drwxr-xr-x - root root 0 2012-07-26 13:32
> > > > > > >
> /hbase/SESSION_TIMELINE1/04295ef1d74887a107a013810f5aa26a/S_T_MTX
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 764 2012-07-26 13:32
> > > > > > >
> > > /hbase/SESSION_TIMELINE1/04295ef1d74887a107a013810f5aa26a/.regioninfo
> > > > > > >
> > > > > > > drwxr-xr-x - root root 3 2012-07-26 13:32
> > > > > > > /hbase/SESSION_TIMELINE1/486b20400be4a901d92ecded96d737cf
> > > > > > >
> > > > > > > drwxr-xr-x - root root 1 2012-07-26 13:32
> > > > > > >
> > /hbase/SESSION_TIMELINE1/486b20400be4a901d92ecded96d737cf/.oldlogs
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 124 2012-07-26 13:32
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/SESSION_TIMELINE1/486b20400be4a901d92ecded96d737cf/.oldlogs/hlog.1343334723397
> > > > > > >
> > > > > > > drwxr-xr-x - root root 0 2012-07-26 13:32
> > > > > > >
> /hbase/SESSION_TIMELINE1/486b20400be4a901d92ecded96d737cf/S_T_MTX
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 762 2012-07-26 13:32
> > > > > > >
> > > /hbase/SESSION_TIMELINE1/486b20400be4a901d92ecded96d737cf/.regioninfo
> > > > > > >
> > > > > > > drwxr-xr-x - root root 4 2012-07-26 13:57
> > > > > > > /hbase/SESSION_TIMELINE1/0a5f6fadd0435898c6f4cf11daa9895a
> > > > > > >
> > > > > > > drwxr-xr-x - root root 0 2012-07-26 13:59
> > > > > > > /hbase/SESSION_TIMELINE1/0a5f6fadd0435898c6f4cf11daa9895a/.tmp
> > > > > > >
> > > > > > > drwxr-xr-x - root root 1 2012-07-26 13:32
> > > > > > >
> > /hbase/SESSION_TIMELINE1/0a5f6fadd0435898c6f4cf11daa9895a/.oldlogs
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 124 2012-07-26 13:32
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/SESSION_TIMELINE1/0a5f6fadd0435898c6f4cf11daa9895a/.oldlogs/hlog.1343334723004
> > > > > > >
> > > > > > > drwxr-xr-x - root root 2 2012-07-26 13:59
> > > > > > >
> /hbase/SESSION_TIMELINE1/0a5f6fadd0435898c6f4cf11daa9895a/S_T_MTX
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 1993369 2012-07-26 13:59
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/SESSION_TIMELINE1/0a5f6fadd0435898c6f4cf11daa9895a/S_T_MTX/1566523617482885717
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 2003372 2012-07-26 13:57
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/SESSION_TIMELINE1/0a5f6fadd0435898c6f4cf11daa9895a/S_T_MTX/7665015246030620502
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 760 2012-07-26 13:32
> > > > > > >
> > > /hbase/SESSION_TIMELINE1/0a5f6fadd0435898c6f4cf11daa9895a/.regioninfo
> > > > > > >
> > > > > > > drwxr-xr-x - root root 3 2012-07-26 13:32
> > > > > > > /hbase/SESSION_TIMELINE1/5eb02340b918e80e5018c908845a8495
> > > > > > >
> > > > > > > drwxr-xr-x - root root 1 2012-07-26 13:32
> > > > > > >
> > /hbase/SESSION_TIMELINE1/5eb02340b918e80e5018c908845a8495/.oldlogs
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 124 2012-07-26 13:32
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/SESSION_TIMELINE1/5eb02340b918e80e5018c908845a8495/.oldlogs/hlog.1343334723286
> > > > > > >
> > > > > > > drwxr-xr-x - root root 0 2012-07-26 13:32
> > > > > > >
> /hbase/SESSION_TIMELINE1/5eb02340b918e80e5018c908845a8495/S_T_MTX
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 764 2012-07-26 13:32
> > > > > > >
> > > /hbase/SESSION_TIMELINE1/5eb02340b918e80e5018c908845a8495/.regioninfo
> > > > > > >
> > > > > > > drwxr-xr-x - root root 3 2012-07-26 13:32
> > > > > > > /hbase/SESSION_TIMELINE1/1f0735a7e08504357d0bca07e6772a75
> > > > > > >
> > > > > > > drwxr-xr-x - root root 1 2012-07-26 13:32
> > > > > > >
> > /hbase/SESSION_TIMELINE1/1f0735a7e08504357d0bca07e6772a75/.oldlogs
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 124 2012-07-26 13:32
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/SESSION_TIMELINE1/1f0735a7e08504357d0bca07e6772a75/.oldlogs/hlog.1343334723134
> > > > > > >
> > > > > > > drwxr-xr-x - root root 0 2012-07-26 13:32
> > > > > > >
> /hbase/SESSION_TIMELINE1/1f0735a7e08504357d0bca07e6772a75/S_T_MTX
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 764 2012-07-26 13:32
> > > > > > >
> > > /hbase/SESSION_TIMELINE1/1f0735a7e08504357d0bca07e6772a75/.regioninfo
> > > > > > >
> > > > > > > drwxr-xr-x - root root 3 2012-07-26 13:32
> > > > > > > /hbase/SESSION_TIMELINE1/7ac1c8704c7d2e676b1dddd75c9b94f2
> > > > > > >
> > > > > > > drwxr-xr-x - root root 1 2012-07-26 13:32
> > > > > > >
> > /hbase/SESSION_TIMELINE1/7ac1c8704c7d2e676b1dddd75c9b94f2/.oldlogs
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 124 2012-07-26 13:32
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/SESSION_TIMELINE1/7ac1c8704c7d2e676b1dddd75c9b94f2/.oldlogs/hlog.1343334723323
> > > > > > >
> > > > > > > drwxr-xr-x - root root 0 2012-07-26 13:32
> > > > > > >
> /hbase/SESSION_TIMELINE1/7ac1c8704c7d2e676b1dddd75c9b94f2/S_T_MTX
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 764 2012-07-26 13:32
> > > > > > >
> > > /hbase/SESSION_TIMELINE1/7ac1c8704c7d2e676b1dddd75c9b94f2/.regioninfo
> > > > > > >
> > > > > > > drwxr-xr-x - root root 3 2012-07-26 13:32
> > > > > > > /hbase/SESSION_TIMELINE1/14e92b7f58e521fd8c0b2d56cd01aaba
> > > > > > >
> > > > > > > drwxr-xr-x - root root 1 2012-07-26 13:32
> > > > > > >
> > /hbase/SESSION_TIMELINE1/14e92b7f58e521fd8c0b2d56cd01aaba/.oldlogs
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 124 2012-07-26 13:32
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /hbase/SESSION_TIMELINE1/14e92b7f58e521fd8c0b2d56cd01aaba/.oldlogs/hlog.1343334723204
> > > > > > >
> > > > > > > drwxr-xr-x - root root 0 2012-07-26 13:32
> > > > > > >
> /hbase/SESSION_TIMELINE1/14e92b7f58e521fd8c0b2d56cd01aaba/S_T_MTX
> > > > > > >
> > > > > > > -rwxr-xr-x 3 root root 764 2012-07-26 13:32
> > > > > > >
> > > /hbase/SESSION_TIMELINE1/14e92b7f58e521fd8c0b2d56cd01aaba/.regioninfo
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Alex Baranau
> > > ------
> > > Sematext :: http://blog.sematext.com/ :: Hadoop - HBase -
> ElasticSearch
> > -
> > > Solr
> > >
> >
>
>
>
> --
> Alex Baranau
> ------
> Sematext :: http://blog.sematext.com/ :: Hadoop - HBase - ElasticSearch -
> Solr
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message