Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com
 designates 209.85.217.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <21359304-1019-41B8-A136-8E7852D5AB9A@stumbleupon.com>
References: <731FDD98-8C3C-47B2-B5C5-0FA5B1261A42@stumbleupon.com>
	<CALte62xruYOJpB3VHkG0c=gXM7jUzQrisq4JHbQym0rRnP+p0A@mail.gmail.com>
	<8CB132C0-7594-44EB-AA8C-5882FEA9AE98@stumbleupon.com>
	<DC5EBE7F3610EB4CA5C7E92D76873E15180DD42ABA@exchange2007.carrieriq.com>
	<6E97D45C-E501-4F75-83D2-002C2E9B1395@stumbleupon.com>
	<1379725746.27943.YahooMailNeo@web140604.mail.bf1.yahoo.com>
	<CALte62yJ0XLqDHuNn5EU9Mu8u58UDEgo3k6FvO3s_mPmYVg=Pg@mail.gmail.com>
	<70FF8C9F-6450-47D9-A0F9-13FB7874E0C2@stumbleupon.com>
	<CALte62wPH8cgP1_TdtMa1tFxJb48-TqDrUEjUMSnygu3xrM5JA@mail.gmail.com>
	<F6A76A55-C456-46F6-BB63-34A11B5D1392@stumbleupon.com>
	<CALte62wtq3csVbP--EBAORFisWe23LSL=4eCovKeS-UsA=HHLg@mail.gmail.com>
	<21359304-1019-41B8-A136-8E7852D5AB9A@stumbleupon.com>
Date: Fri, 20 Sep 2013 21:11:31 -0700
Message-ID: 
 <CALte62xhT+-La7iYwvtL7SaFC4Y7wr03Dw+CHLwYY=JAkJr8Lg@mail.gmail.com>
Subject: Re: HBase Table Row Count Optimization - A Solicitation For Help
From: Ted Yu <yuzhihong@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a1133eaca5f3b7204e6dcfb77

--001a1133eaca5f3b7204e6dcfb77
Content-Type: text/plain; charset=ISO-8859-1

Thanks for the feedback.

I logged HBASE-9605 for relaxation of this requirement for row
count aggregate.


On Fri, Sep 20, 2013 at 8:46 PM, James Birchfield <
jbirchfield@stumbleupon.com> wrote:

> Thanks.  I have ben taking a look this evening.  We enabled the
> Aggregation coprocessor and the Aggregation client works great.  I still
> have to execute it with the 'hadoop jar' command though, but can live with
> that.  When I try to run it in process, it just hangs.  I am not going to
> fight i though.
>
> The only thing I dislike about the AggrgationClient is that it requires a
> column family.  I was hoping to do this in a completely generic way,
> without having any information about a tables column families to get a row
> count.  The provided implementation requires exactly one.  I was hoping
> maybe there was always some sort of default column family always print on a
> table but it does not appear so.  I will look at the provided coprocessor
> implementation and see why it is required and see if it can be optional,
> and if so, what the performance penalty would be.  In the mean time, I am
> just using the first column family returned from a query to the admin
> client for a table.  Seems to work fine.
>
> Thanks!
> Birch
> On Sep 20, 2013, at 8:41 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > HBase is open source. You can check out the source code and look at the
> > source code.
> >
> > $ svn info
> > Path: .
> > URL: http://svn.apache.org/repos/asf/hbase/branches/0.94
> > Repository Root: http://svn.apache.org/repos/asf
> > Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> > Revision: 1525061
> >
> >
> > On Fri, Sep 20, 2013 at 6:46 PM, James Birchfield <
> > jbirchfield@stumbleupon.com> wrote:
> >
> >> Ted,
> >>
> >>        My apologies if I am being thick, but I am looking at the API
> docs
> >> here: http://hbase.apache.org/apidocs/index.html and I do not see that
> >> package.  And the coprocessor package only contains an exception.
> >>
> >>        Ok, weird.  Those classes do not show up through normal
> navigation
> >> from that link, however, the documentation does exist if I google for it
> >> directly.  Maybe the javadocs need to be regenerated???  Dunno, but I
> will
> >> check it out.
> >>
> >> Birch
> >>
> >> On Sep 20, 2013, at 6:32 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>
> >>> Please take a look at the javadoc
> >>> for
> >>
> src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java
> >>>
> >>> As long as the machine can reach your HBase cluster, you should be able
> >> to
> >>> run AggregationClient and utilize the AggregateImplementation endpoint
> in
> >>> the region servers.
> >>>
> >>> Cheers
> >>>
> >>>
> >>> On Fri, Sep 20, 2013 at 6:26 PM, James Birchfield <
> >>> jbirchfield@stumbleupon.com> wrote:
> >>>
> >>>> Thanks Ted.
> >>>>
> >>>> That was the direction I have been working towards as I am learning
> >> today.
> >>>> Much appreciation to all the replies to this thread.
> >>>>
> >>>> Whether I keep the MapReduce job or utilize the Aggregation
> coprocessor
> >>>> (which is turning out that it should be possible for me here), I need
> to
> >>>> make sure I am running the client in an efficient manner.  Lars may
> have
> >>>> hit upon the core problem.  I am not running the map reduce job on the
> >>>> cluster, but rather from a stand alone remote java client executing
> the
> >> job
> >>>> in process.  This may very well turn out to be the number one issue.
>  I
> >>>> would love it if this turns out to be true.  Would make this a great
> >>>> learning lesson for me as a relative newcomer to working with HBase,
> and
> >>>> potentially allow me to finish this initial task much quicker than I
> was
> >>>> thinking.
> >>>>
> >>>> So assuming the MapReduce jobs need to be run on the cluster instead
> of
> >>>> locally, does a coprocessor endpoint client need to be run the same,
> or
> >> is
> >>>> it safe to run it on a remote machine since the work gets distributed
> >> out
> >>>> to the region servers?  Just wondering if I would run into the same
> >> issues
> >>>> if what I said above holds true.
> >>>>
> >>>> Thanks!
> >>>> Birch
> >>>> On Sep 20, 2013, at 6:17 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>>>
> >>>>> In 0.94, we have AggregateImplementation, an endpoint coprocessor,
> >> which
> >>>>> implements getRowNum().
> >>>>>
> >>>>> Example is in AggregationClient.java
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>>
> >>>>> On Fri, Sep 20, 2013 at 6:09 PM, lars hofhansl <larsh@apache.org>
> >> wrote:
> >>>>>
> >>>>>> From your numbers below you have about 26k regions, thus each region
> >> is
> >>>>>> about 545tb/26k = 20gb. Good.
> >>>>>>
> >>>>>> How many mappers are you running?
> >>>>>> And just to rule out the obvious, the M/R is running on the cluster
> >> and
> >>>>>> not locally, right? (it will default to a local runner when it
> cannot
> >>>> use
> >>>>>> the M/R cluster).
> >>>>>>
> >>>>>> Some back of the envelope calculations tell me that assuming 1ge
> >> network
> >>>>>> cards, the best you can expect for 110 machines to map through this
> >>>> data is
> >>>>>> about 10h. (so way faster than what you see).
> >>>>>> (545tb/(110*1/8gb/s) ~ 40ks ~11h)
> >>>>>>
> >>>>>>
> >>>>>> We should really add a rowcounting coprocessor to HBase and allow
> >> using
> >>>> it
> >>>>>> via M/R.
> >>>>>>
> >>>>>> -- Lars
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ________________________________
> >>>>>> From: James Birchfield <jbirchfield@stumbleupon.com>
> >>>>>> To: user@hbase.apache.org
> >>>>>> Sent: Friday, September 20, 2013 5:09 PM
> >>>>>> Subject: Re: HBase Table Row Count Optimization - A Solicitation For
> >>>> Help
> >>>>>>
> >>>>>>
> >>>>>> I did not implement accurate timing, but the current table being
> >> counted
> >>>>>> has been running for about 10 hours, and the log is estimating the
> map
> >>>>>> portion at 10%
> >>>>>>
> >>>>>> 2013-09-20 23:40:24,099 INFO  [main] Job
>  :
> >>>> map
> >>>>>> 10% reduce 0%
> >>>>>>
> >>>>>> So a loooong time.  Like I mentioned, we have billions, if not
> >> trillions
> >>>>>> of rows potentially.
> >>>>>>
> >>>>>> Thanks for the feedback on the approaches I mentioned.  I was not
> sure
> >>>> if
> >>>>>> they would have any effect overall.
> >>>>>>
> >>>>>> I will look further into coprocessors.
> >>>>>>
> >>>>>> Thanks!
> >>>>>> Birch
> >>>>>> On Sep 20, 2013, at 4:58 PM, Vladimir Rodionov <
> >> vrodionov@carrieriq.com
> >>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> How long does it take for RowCounter Job for largest table to
> finish
> >> on
> >>>>>> your cluster?
> >>>>>>>
> >>>>>>> Just curious.
> >>>>>>>
> >>>>>>> On your options:
> >>>>>>>
> >>>>>>> 1. Not worth it probably - you may overload your cluster
> >>>>>>> 2. Not sure this one differs from 1. Looks the same to me but more
> >>>>>> complex.
> >>>>>>> 3. The same as 1 and 2
> >>>>>>>
> >>>>>>> Counting rows in efficient way can be done if you sacrifice some
> >>>>>> accuracy :
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> http://highscalability.com/blog/2012/4/5/big-data-counting-how-to-count-a-billion-distinct-objects-us.html
> >>>>>>>
> >>>>>>> Yeah, you will need coprocessors for that.
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>> Vladimir Rodionov
> >>>>>>> Principal Platform Engineer
> >>>>>>> Carrier IQ, www.carrieriq.com
> >>>>>>> e-mail: vrodionov@carrieriq.com
> >>>>>>>
> >>>>>>> ________________________________________
> >>>>>>> From: James Birchfield [jbirchfield@stumbleupon.com]
> >>>>>>> Sent: Friday, September 20, 2013 3:50 PM
> >>>>>>> To: user@hbase.apache.org
> >>>>>>> Subject: Re: HBase Table Row Count Optimization - A Solicitation
> For
> >>>> Help
> >>>>>>>
> >>>>>>> Hadoop 2.0.0-cdh4.3.1
> >>>>>>>
> >>>>>>> HBase 0.94.6-cdh4.3.1
> >>>>>>>
> >>>>>>> 110 servers, 0 dead, 238.2364 average load
> >>>>>>>
> >>>>>>> Some other info, not sure if it helps or not.
> >>>>>>>
> >>>>>>> Configured Capacity: 1295277834158080 (1.15 PB)
> >>>>>>> Present Capacity: 1224692609430678 (1.09 PB)
> >>>>>>> DFS Remaining: 624376503857152 (567.87 TB)
> >>>>>>> DFS Used: 600316105573526 (545.98 TB)
> >>>>>>> DFS Used%: 49.02%
> >>>>>>> Under replicated blocks: 0
> >>>>>>> Blocks with corrupt replicas: 1
> >>>>>>> Missing blocks: 0
> >>>>>>>
> >>>>>>> It is hitting a production cluster, but I am not really sure how to
> >>>>>> calculate the load placed on the cluster.
> >>>>>>> On Sep 20, 2013, at 3:19 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> How many nodes do you have in your cluster ?
> >>>>>>>>
> >>>>>>>> When counting rows, what other load would be placed on the
> cluster ?
> >>>>>>>>
> >>>>>>>> What is the HBase version you're currently using / planning to
> use ?
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Sep 20, 2013 at 2:47 PM, James Birchfield <
> >>>>>>>> jbirchfield@stumbleupon.com> wrote:
> >>>>>>>>
> >>>>>>>>>    After reading the documentation and scouring the mailing list
> >>>>>>>>> archives, I understand there is no real support for fast row
> >> counting
> >>>>>> in
> >>>>>>>>> HBase unless you build some sort of tracking logic into your
> code.
> >>>> In
> >>>>>> our
> >>>>>>>>> case, we do not have such logic, and have massive amounts of data
> >>>>>> already
> >>>>>>>>> persisted.  I am running into the issue of very long execution of
> >> the
> >>>>>>>>> RowCounter MapReduce job against very large tables (multi-billion
> >> for
> >>>>>> many
> >>>>>>>>> is our estimate).  I understand why this issue exists and am
> slowly
> >>>>>>>>> accepting it, but I am hoping I can solicit some possible ideas
> to
> >>>> help
> >>>>>>>>> speed things up a little.
> >>>>>>>>>
> >>>>>>>>>    My current task is to provide total row counts on about 600
> >>>>>>>>> tables, some extremely large, some not so much.  Currently, I
> have
> >> a
> >>>>>>>>> process that executes the MapRduce job in process like so:
> >>>>>>>>>
> >>>>>>>>>                    Job job = RowCounter.createSubmittableJob(
> >>>>>>>>>
> >>>> ConfigManager.getConfiguration(),
> >>>>>>>>> new String[]{tableName});
> >>>>>>>>>                    boolean waitForCompletion =
> >>>>>>>>> job.waitForCompletion(true);
> >>>>>>>>>                    Counters counters = job.getCounters();
> >>>>>>>>>                    Counter rowCounter =
> >>>>>>>>> counters.findCounter(hbaseadminconnection.Counters.ROWS);
> >>>>>>>>>                    return rowCounter.getValue();
> >>>>>>>>>
> >>>>>>>>>    At the moment, each MapReduce job is executed in serial order,
> >>>> so
> >>>>>>>>> counting one table at a time.  For the current implementation of
> >> this
> >>>>>> whole
> >>>>>>>>> process, as it stands right now, my rough timing calculations
> >>>> indicate
> >>>>>> that
> >>>>>>>>> fully counting all the rows of these 600 tables will take
> anywhere
> >>>>>> between
> >>>>>>>>> 11 to 22 days.  This is not what I consider a desirable
> timeframe.
> >>>>>>>>>
> >>>>>>>>>    I have considered three alternative approaches to speed things
> >>>>>> up.
> >>>>>>>>>
> >>>>>>>>>    First, since the application is not heavily CPU bound, I could
> >>>>>> use
> >>>>>>>>> a ThreadPool and execute multiple MapReduce jobs at the same time
> >>>>>> looking
> >>>>>>>>> at different tables.  I have never done this, so I am unsure if
> >> this
> >>>>>> would
> >>>>>>>>> cause any unanticipated side effects.
> >>>>>>>>>
> >>>>>>>>>    Second, I could distribute the processes.  I could find as
> many
> >>>>>>>>> machines that can successfully talk to the desired cluster
> >> properly,
> >>>>>> give
> >>>>>>>>> them a subset of tables to work on, and then combine the results
> >> post
> >>>>>>>>> process.
> >>>>>>>>>
> >>>>>>>>>    Third, I could combine both the above approaches and run a
> >>>>>>>>> distributed set of multithreaded process to execute the MapReduce
> >>>> jobs
> >>>>>> in
> >>>>>>>>> parallel.
> >>>>>>>>>
> >>>>>>>>>    Although it seems to have been asked and answered many times,
> I
> >>>>>>>>> will ask once again.  Without the need to change our current
> >>>>>> configurations
> >>>>>>>>> or restart the clusters, is there a faster approach to obtain row
> >>>>>> counts?
> >>>>>>>>> FYI, my cache size for the Scan is set to 1000.  I have
> >> experimented
> >>>>>> with
> >>>>>>>>> different numbers, but nothing made a noticeable difference.  Any
> >>>>>> advice or
> >>>>>>>>> feedback would be greatly appreciated!
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Birch
> >>>>>>>
> >>>>>>>
> >>>>>>> Confidentiality Notice:  The information contained in this message,
> >>>>>> including any attachments hereto, may be confidential and is
> intended
> >>>> to be
> >>>>>> read only by the individual or entity to whom this message is
> >>>> addressed. If
> >>>>>> the reader of this message is not the intended recipient or an agent
> >> or
> >>>>>> designee of the intended recipient, please note that any review,
> use,
> >>>>>> disclosure or distribution of this message or its attachments, in
> any
> >>>> form,
> >>>>>> is strictly prohibited.  If you have received this message in error,
> >>>> please
> >>>>>> immediately notify the sender and/or Notifications@carrieriq.comand
> >>>>>> delete or destroy any copy of this message and its attachments.
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>

--001a1133eaca5f3b7204e6dcfb77--