Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: unknown (athena.apache.org: error in processing during lookup of
 dave@urbanairship.com)
MIME-Version: 1.0
In-Reply-To: <BLU0-SMTP243B71549A23D7A49CF08C08F180@phx.gbl>
References: 
 <CALiJDj2c-2ce6ukqpyxAxwP5hNdgu=sJr1N7fHCZg81uzEpNtg@mail.gmail.com>
	<BLU0-SMTP408369C83EE4C024BE09F978F1A0@phx.gbl>
	<CALiJDj1+g4OYE0FzOz=X=J2HKeYB=XTu6P6O5x-AB_Nq3Wr4Mg@mail.gmail.com>
	<BLU0-SMTP4142656D249755D138C50568F1B0@phx.gbl>
	<CALiJDj1UY0dZdRJWDoPM88OsyvztjcV+_mPaLeozEVBJPGdNjA@mail.gmail.com>
	<BLU0-SMTP153610E2A362ED0D357F7478F180@phx.gbl>
	<CA+RK=_Bp0+DKy24oTd+JEJ2v=v3sYw2e+bPc4YK62RYaUt9EFg@mail.gmail.com>
	<BLU0-SMTP243B71549A23D7A49CF08C08F180@phx.gbl>
Date: Wed, 16 May 2012 14:40:52 -0700
Message-ID: 
 <CAN_xCOcHkwvefiCYg2ODdMmowhtZGaedQjfxJxNmDkYztunE3A@mail.gmail.com>
Subject: Re: EndPoint Coprocessor could be dealocked?
From: Dave Revell <dave@urbanairship.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=e89a8f6428ec63897404c02e2cf1

--e89a8f6428ec63897404c02e2cf1
Content-Type: text/plain; charset=ISO-8859-1

Many people will probably try to use coprocessors as a way of implementing
app logic on top of HBase without the headaches of writing a daemon.
Sometimes client-side approaches are inadvisable; for example, there may be
several client languages/runtimes and the app logic should not be
reimplemented in each.

It's understandable that people wouldn't want to deal with setting up a
daemon and RPC mechanism if they can piggyback on the existing HBase
coprocessor mechanism.

Are HBase coprocessors explicitly wrong for this use case if the app logic
needs to access multiple regions in a single call?

Cheers,
Dave

On Wed, May 16, 2012 at 12:07 PM, Michael Segel
<michael_segel@hotmail.com>wrote:

>
>  I think we need to look at the base problem that is trying to be solved.
>
> I mean the discussion on the RPC mechanism. but the problem that the OP is
> trying to solve is how to use multiple indexes in a 'query'.
>
> Note: I put ' ' around query because its a m/r job or a single thread
> where the user is trying to get a result set which is a significantly
> smaller subset, using more than 1 index.
>
> So the idea is to do a quick get() against each index and the result would
> be a list of row keys. The next step is to get the intersection(s) quickly
> (which I proposed), and then you would just need to do a quick series of
> get()s  to pull back the list of rows.
>
> If I understand the OP's problem, its not a co-processor type of problem.
>
> Its one of where you submit a m/r job. Within your toolRunner, you would
> actually do the fetches against the indexes and then build the ultimate
> result set. then you just need a map job to take your result set as an
> input.
>
> Drawback... if the list of rows is very, very long, you may run out of
> memory. So you need to resolve that...
> (Which is why I was suggesting on using a temp table and then you can use
> the rows in the temp table as input in to your fetch...
>
> While not something I would use for 'real time' its something where I can
> really shrink the number of rows you have to fetch for further processing.
>  So if your full table scan takes an hour, but we can do N get()s to get
> the rows in the Index, find the intersection I and then do I.size() get()s
> to fetch the data.  This should take much less time.
>
>
> Again, I don't see this in a coprocessor based solution, however, the N
> get()s and intersection could be done at the start of the job, or could be
> part of a Map only job.
>
> Kind of an interesting problem...  but if anyone has a large set of data
> and some time to play, you will end up solving a problem that you can' do
> in an RDBMS easily.
>
> On May 16, 2012, at 1:17 PM, Andrew Purtell wrote:
>
> >> On May 16, 2012, at 1:12 AM, fding hbase wrote:
> >>> But sadly, HBase ipc doesn't allow coprocessor chaining mechanism...
> >>> Someone mentioned on
> >>>
> http://grokbase.com/t/hbase/user/116hrhhf8m/coprocessor-failure-question-and-examples
> >>> :
> >>>
> >>> If a RegionObserver issues RPC to another table from any of the hooks
> that
> >>> are called out of RPC handlers (for Gets, Puts, Deletes, etc.), you
> risk
> >>> deadlock. Whatever activity you want to check should be in the same
> >>> region as account data to avoid that.
> >>> (Or HBase RPC needs to change.)
> >>>
> >>> So, that means, the deadlock is inevitable under current circumstance.
> The
> >>> coprocessors are still limited.
> >>>
> >>> What I'm seeking is possible extensions of coprocessors or workaround
> for
> >>> such situations that extra RPC is needed in the RPC handlers.
> >
> > This isn't a limitation, this is a design choice. Such extensions of
> > coprocessors most likely won't happen. What a RegionObserver allows
> > you to do is exactly this: Intercept and potentially modify lifecycle
> > or user operations on that single region alone. If it helps, think of
> > each region as its own independent database.
> >
> > If you need to take cross-region actions according to some user
> > action, then you should be looking first at extending the client, not
> > the server.
> >
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein (via Tom White)
> >
>
>

--e89a8f6428ec63897404c02e2cf1--