Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=w7Oe8MizSkl8y6LG+mfjTi8Ylyy+4gwcw8UeJ9fcqkP56ySxKoZJEAWU3h88vcTNGGFZGz9dWb73P6ED6Dj9wBEdKGYIhKRWc19QnN89AQMIhvVOj4jhgCM1vQKELNDwKuP09WsJVDuSgJS9s2zS20Ikk8kAaS3xNs+fSkbVizw=;
References: 
 <CACFR+W=EEYChdQKMhncxgH5pqYmNm=mL2ncexaK3p4T64vAFmw@mail.gmail.com>
 <CA+RK=_DBThTt=a4m4rxZh5MB5UZqwLparnhKbuM--SjaSHUh4Q@mail.gmail.com>
 <CA+RK=_Am4j4EOd+JS_VvhN=016EuqMj3y9LNqTjtx09r8_P5Kg@mail.gmail.com>
 <CA+RK=_Afi4dxV5xLBfOrtHqGeju=STfd1u4D8mvFciw_80OFwg@mail.gmail.com>
Message-ID: <1346177552.57605.YahooMailNeo@web121702.mail.ne1.yahoo.com>
Date: Tue, 28 Aug 2012 11:12:32 -0700 (PDT)
From: lars hofhansl <lhofhansl@yahoo.com>
Reply-To: lars hofhansl <lhofhansl@yahoo.com>
Subject: Re: Improving Coprocessor postSplit/postOpen synchronization
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
In-Reply-To: 
 <CA+RK=_Afi4dxV5xLBfOrtHqGeju=STfd1u4D8mvFciw_80OFwg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

That approach sounds good to me.=0A=0A=0A=0A----- Original Message -----=0A=
From: Andrew Purtell <apurtell@apache.org>=0ATo: dev@hbase.apache.org=0ACc:=
 =0ASent: Tuesday, August 28, 2012 3:05 AM=0ASubject: Re: Improving Coproce=
ssor postSplit/postOpen synchronization=0A=0ANever mind, I went to look at =
the code. Should have done that first.=0A=0ALooking at 0.94 sources, in Spl=
itTransaction, first we notify the master=0Athat the split has happened, an=
d wait for the master to process it (which=0Aopens daughters), and then cal=
l up to the CP with the daughter regions as=0Aarguments.=0A=0AI seem to rem=
ember that in my prototype patch for the CP framework,=0ApostSplit notifica=
tion let the CP know the split took place and allow it to=0Atake actions be=
fore the master opened the daughters. In any event that's=0Anot the code no=
w, so it seems what you need here is for us to move the=0ApostSplit upcall =
up prior to master notification or add another hook at=0Athat location.=0A=
=0AOn Tue, Aug 28, 2012 at 12:53 PM, Andrew Purtell <apurtell@apache.org>wr=
ote:=0A=0A> (from postSplit)=0A>=0A>=0A> On Tue, Aug 28, 2012 at 12:53 PM, =
Andrew Purtell <apurtell@apache.org>wrote:=0A>=0A>> What about writing a ma=
rker (a file) into the region at split (from=0A>> preSplit) which is then e=
xistence checked and read at open (postOpen)? This=0A>> file would contain =
whatever indexing metadata is required.=0A>>=0A>> Also, splits are nearly i=
nstant because the daughters are created with=0A>> reference files to the p=
arent, until a later compaction brings the data=0A>> from the parent over. =
Can you do the same with your indexes? Reason I ask=0A>> is this notion of =
"ignoring" new data until indexes are available seems=0A>> undesirable.=0A>=
>=0A>>=0A>> On Mon, Aug 27, 2012 at 11:29 PM, Kevin Shin <=0A>> kevin.shin@=
thinkbiganalytics.com> wrote:=0A>>=0A>>> Hi everyone,=0A>>>=0A>>> A colleag=
ue and I were working with HBase coprocessors for secondary=0A>>> indexes a=
nd ran into an interesting problem regarding splits=0A>>> and synchronizing=
 the corresponding parent/daughter regions.=0A>>>=0A>>> The goal with split=
s is to create two new daughter regions with the=0A>>> corresponding splits=
 of the secondary indexes and lock these regions such=0A>>> that Puts/Delet=
es that occur while postSplit is in progress will be=0A>>> queued=0A>>> up =
so we don't run into consistency issues. IE, if a delete gets called=0A>>> =
before a daughter region receives the split index, that delete would=0A>>> =
essentially have been ignored, so we would want to wait until postSplit=0A>=
>> is=0A>>> finished before running any new Puts/Deletes on the split regio=
ns.=0A>>>=0A>>> As of right now, the HBase coprocessors do not easily suppo=
rt a way to=0A>>> achieve this level of consistency in that there is no way=
 to distinguish=0A>>> a=0A>>> region being opened from a split or a regular=
 open. If we could=0A>>> distinguish, we could open up the correct index fr=
om the start and stall=0A>>> until postSplit is finished in the background =
in the event of a split. I=0A>>> would thus like to propose a way to "lock"=
 the daughter regions when=0A>>> postSplit is called. That is, when we open=
 a daughter region from a=0A>>> split,=0A>>> we can pass in the parent regi=
on name alongside it (or Null if there is=0A>>> no=0A>>> parent) to disting=
uish a region being opened from a split or open. I am=0A>>> thinking about =
submitting a patch into JIRA but would greatly appreciate=0A>>> any thought=
s or suggestions for another solution to the problem or=0A>>> perhaps=0A>>>=
 a better patch. I am using HBase 0.92 for development at this moment.=0A>>=
>=0A>>> Best,=0A>>> Kevin=0A>>>=0A>>=0A>>=0A>>=0A>> --=0A>> Best regards,=
=0A>>=0A>>=A0 =A0 - Andy=0A>>=0A>> Problems worthy of attack prove their wo=
rth by hitting back. - Piet Hein=0A>> (via Tom White)=0A>>=0A>>=0A>=0A>=0A>=
 --=0A> Best regards,=0A>=0A>=A0 =A0 - Andy=0A>=0A> Problems worthy of atta=
ck prove their worth by hitting back. - Piet Hein=0A> (via Tom White)=0A>=
=0A>=0A=0A=0A-- =0ABest regards,=0A=0A=A0  - Andy=0A=0AProblems worthy of a=
ttack prove their worth by hitting back. - Piet Hein=0A(via Tom White)=0A