Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CALte62w37QzbPP-rwjj5ZxpQ_Cdy0RzwzwNdBXGZfSbQLMEGaA@mail.gmail.com>
References: 
 <CAOVwZ_VzyK3-5xtFxAvJTosKajizw6uLLOYxkgDCyK7KDU+GVA@mail.gmail.com>
	<CALte62x=L7WgsXJ_TSy-yOnhxz=y79tKdVXAtpSak7eEFL_tyQ@mail.gmail.com>
	<CAOVwZ_Xi_uK2n55PrBoJALBDOP1b8bWzc-OezAWvVR5OA_Q=ag@mail.gmail.com>
	<CALte62xTctXjZgy992VLNVhi5F+hOorK9nvsWfoLAjPYsHohEw@mail.gmail.com>
	<CAOVwZ_Vq-U2i=YtgcBDHwM2wBwb01LNxEd33Vh7N26WR8Q0p=A@mail.gmail.com>
	<CALte62w37QzbPP-rwjj5ZxpQ_Cdy0RzwzwNdBXGZfSbQLMEGaA@mail.gmail.com>
Date: Sat, 26 Mar 2016 16:43:09 +0100
Message-ID: 
 <CAOVwZ_WEiRQk_9JpqRRw4VjGga2cFnHvpgZo2ffWJ4ytDZ1vbA@mail.gmail.com>
Subject: Re: processing in coprocessor and region splitting
From: =?UTF-8?B?RGFuaWVsIFBvxYJhY3phxYRza2k=?= <dpolaczanski@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a11c39e2652e038052ef589b1

--001a11c39e2652e038052ef589b1
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

unfortunately no

Regards

2016-03-25 21:13 GMT+01:00 Ted Yu <yuzhihong@gmail.com>:

> bq. calculating another new attributes of a trade
>
> Can you put the new attributes in separate columns ?
>
> Cheers
>
> On Fri, Mar 25, 2016 at 12:38 PM, Daniel Po=C5=82acza=C5=84ski <
> dpolaczanski@gmail.com
> > wrote:
>
> > The data is set of trades and the processing is some kind of enrichment
> > (calculating another new attributes of a trade). All attributes are
> needed
> > (the original and new)
> >
> > 2016-03-25 18:41 GMT+01:00 Ted Yu <yuzhihong@gmail.com>:
> >
> > > bq. During the processing the size of the data is doubled.
> > >
> > > This explains the frequent split :-)
> > >
> > > Is the original data needed after post-processing (maybe for auditing=
)
> ?
> > >
> > > Cheers
> > >
> > > On Fri, Mar 25, 2016 at 10:32 AM, Daniel Po=C5=82acza=C5=84ski <
> > > dpolaczanski@gmail.com
> > > > wrote:
> > >
> > > > I am testing different solutions (POC).
> > > > The region size currenlty is 32MB (I know it should be >=3D 1GB, bu=
t we
> > are
> > > > testing different solutions with smaller amount of the data ). So
> > > > increasing region size is not a solution. Our problems can happen
> even
> > > when
> > > > a region will be 1 GB. We want to proces the data with coprocessor
> and
> > > > hadoop map reduce. I can not have one big Region because I want
> > sensible
> > > > degree of paralerism (with Map Reduce and coprocessors).
> > > >
> > > > Increasing region size + pre-splitting  is not an option as well
> > because
> > > I
> > > > know nothing about keys(random long).
> > > >
> > > > During the processing the size of the data is doubled.
> > > >
> > > > And yes, coprocessor rewrites a lot of the data written into the
> table.
> > > The
> > > > whole record is serialized to avro and stored in one column (storin=
g
> > > single
> > > > attribute in single column we will try in the next POC)
> > > >
> > > > it is not a typical big data project where we can allow former
> analysis
> > > of
> > > > the data:)
> > > >
> > > > 2016-03-25 17:38 GMT+01:00 Ted Yu <yuzhihong@gmail.com>:
> > > >
> > > > > What's the current region size you use ?
> > > > >
> > > > > bq. During the processing size of the data gets increased
> > > > >
> > > > > Can you give us some quantitative measure as to how much increase
> you
> > > > > observed (w.r.t. region size) ?
> > > > >
> > > > > bq. I was looking for some "global lock" in source code
> > > > >
> > > > > Probably not a good idea using global lock.
> > > > >
> > > > > I am curious, looks like your coprocesser may rewrite a lot of da=
ta
> > > > written
> > > > > into the table.
> > > > > Can client side accommodate such logic so that the rewrite is
> > reduced ?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Fri, Mar 25, 2016 at 8:55 AM, Daniel Po=C5=82acza=C5=84ski <
> > > > > dpolaczanski@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I have some processing in my coprocesserService which modifies
> the
> > > > > existing
> > > > > > data in place. It iterates over every row, modifies and puts it
> > back
> > > to
> > > > > > region. The table can be modified by only one client.
> > > > > >
> > > > > > During the processing size of the data gets increased -> region=
's
> > > size
> > > > > get
> > > > > > increased -> region's split happens. It makes that the processi=
ng
> > is
> > > > > > stopped by exception NotServingRegionException (because region =
is
> > > > closed
> > > > > > and splited to two new regions so it is closed and doesn't exis=
t
> > > > > anymore).
> > > > > >
> > > > > > Is there any clean way to block Region's splitting?
> > > > > >
> > > > > > I was looking for some "global lock" in source code but I haven=
't
> > > found
> > > > > > anything helpfull.
> > > > > > Another idea is to create custom RegionSplitPolicy and explicil=
ty
> > set
> > > > > some
> > > > > > Flag which will return false in shouldSplit(), but I'm not sure
> yet
> > > if
> > > > it
> > > > > > is safe.
> > > > > > Could you advise?
> > > > > > Regards
> > > > > >
> > > > >
> > > >
> > >
> >
>

--001a11c39e2652e038052ef589b1--