Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of wtan@us.ibm.com designates
 32.97.182.139 as permitted sender)
In-Reply-To: 
 <CA+RK=_DWk5w0o+HoCFumFiieihnfSuq2GipsAuCoYyrhKj0BGQ@mail.gmail.com>
References: 
 <CADt=Q7=vRLuH5JOB+2ouz=GUBcEdEsL09XscYi4OXYC4cmq_Yw@mail.gmail.com>
	<CA+RK=_BBnnz2LOCwpvsHz=JCtg3v_8wOyjMAe29kjROyEpEqVQ@mail.gmail.com>
	<OF9042D789.69DFDFD6-ON85257AF4.0066DE91-85257AF4.0066F687@us.ibm.com>
 <CA+RK=_DWk5w0o+HoCFumFiieihnfSuq2GipsAuCoYyrhKj0BGQ@mail.gmail.com>
To: user@hbase.apache.org
MIME-Version: 1.0
Subject: Re: Coprocessor / threading model
Message-ID: 
 <OFA63E5E5A.F744F2B5-ON85257AF4.007C3478-85257AF4.007CA918@us.ibm.com>
From: Wei Tan <wtan@us.ibm.com>
Date: Tue, 15 Jan 2013 17:41:40 -0500
Content-Type: multipart/alternative;
 boundary="=_alternative 007CA91685257AF4_="

--=_alternative 007CA91685257AF4_=
Content-Type: text/plain; charset="US-ASCII"

Thanks Andrew for your detailed clarification.
Now I understand that in general, the system is subject to CAP theorem. 
You want good consistency AND latency, then partition tolerance needs to 
be sacrificed: this is the "local index" approach, i.e., colocate index 
and data and avoid RPC.

Otherwise, if you can tolerate consistency but not latency, you put RPCs 
in a queue and process them in the background. By this means you can have 
a "global" index with some lag. 


Best Regards,
Wei

Wei Tan 
Research Staff Member 
IBM T. J. Watson Research Center
Yorktown Heights, NY 10598
wtan@us.ibm.com; 914-945-4386


From:   Andrew Purtell <apurtell@apache.org>
To:     "user@hbase.apache.org" <user@hbase.apache.org>, 
Date:   01/15/2013 02:20 PM
Subject:        Re: Coprocessor / threading model


HTable is a blocking interface. When a client issues a put, for example, 
we
do not want to return until we can confirm the store has been durably
persisted. For client convenience many additional details of remote region
invocation are hidden, for example META table lookups for relocated
regions, reconnection, retries. Just about all coprocessor upcalls for the
Observer interface happen with the RPC handler context. RPC handlers are
drawn from a fixed pool of threads. Your CP code is tying up one of a 
fixed
resource for as long as it has control. And in here you are running the
complex HTable machinery. For many reasons your method call on HTable may
block (potentially for a long time) and therefore the RPC handler your
invocation is executing within will also block. An accidental cycle can
cause a deadlock once there are no free handlers somewhere, which will
happen as part of normal operation when the cluster is loaded, and the
higher the load the more likely.

Instead you can do what Anoop has described in this thread and install a 
CP
into the master that insures index regions are assigned to the same
regionserver as the primary table, and then call from a region of the
primary table into a colocated region of the index table, or vice versa,
bypassing HTable and the RPC stack. This is just making an in process
method call on one object from another.

Or, you could allocate a small executor pool for cross region RPC. When 
the
upcall into your CP happens, dispatch work to the executor and return
immediately to release the RPC worker thread back to the pool. This would
avoid the possibility of deadlock but this may not give you the semantics
you want because that background work could lag unpredictably.


On Tue, Jan 15, 2013 at 10:44 AM, Wei Tan <wtan@us.ibm.com> wrote:

> Andrew, could you explain more, why doing cross-table operation is an
> anti-pattern of using CP?
> Durability might be an issue, as far as I understand. Thanks,
>
>
> Best Regards,
> Wei
>

-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)


--=_alternative 007CA91685257AF4_=--