Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <CAFAMeYJCSe9PpEv4PagAxzmNjsJzn+pcgP78RLgKPSojZ+MTBA@mail.gmail.com>
References: <CAN5cbe4QV3GmTHiKAFYoceCye4PmAWgW+jduzXZGS5tZFtXC0A@mail.gmail.com>
 <CADcMMgGPvL+o22vDOA+fV42xsM0dbTseM0Db2eSe82j+F0ttbg@mail.gmail.com>
 <CAN5cbe6E_yfXB2bZ77E+xDO1rnPHM-34sgEZ8F=1EEPgRiZT8g@mail.gmail.com>
 <CADcMMgG-9Dd=TUuyQ+b_UhGFsg0Gg3fW7umxhFTSoN1Kq4yJiA@mail.gmail.com>
 <CANczWyO11Oh3gTH5EwOqJ_H=F9cAvpnvhLY-d+EMxMNKMTJf0A@mail.gmail.com> <CAFAMeYJCSe9PpEv4PagAxzmNjsJzn+pcgP78RLgKPSojZ+MTBA@mail.gmail.com>
From: Weiqing Yang <yangweiqing001@gmail.com>
Date: Fri, 23 Jun 2017 13:47:13 -0700
Message-ID: <CANczWyNEuJWmaY+L4T5iS23C3X2LjdmJLSo4mWr6qcBFCxSAfw@mail.gmail.com>
Subject: Re: [DISCUSS] status of and plans for our hbase-spark integration
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary="94eb2c084e84c3ce410552a6b4d0"
archived-at: Fri, 23 Jun 2017 20:47:26 -0000

--94eb2c084e84c3ce410552a6b4d0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Not yet. I'll create one after we decide next move.


Thanks,
weiqing


On Fri, Jun 23, 2017 at 1:08 PM, Mike Drob <mdrob@apache.org> wrote:

> Weiqing, do you have JIRA issues filed for your progress that folks can
> follow along?
>
> Sean, another item to watch might be the state of HBase+PySpark. Our
> friends at Huawei have done this work [1][2] and maybe it is another
> candidate for inclusion and first class support? I'm not sure who the bes=
t
> contact for this would be, however.
>
> Mike
>
> [1]: http://huaweibigdata.github.io/astro/
> [2]: https://github.com/Huawei-Spark/Spark-SQL-on-HBase
>
> On Fri, Jun 23, 2017 at 2:42 PM, Weiqing Yang <yangweiqing001@gmail.com>
> wrote:
>
> > Thanks, Sean!
> >
> > You are right, SHC has its own pluggable system for data
> encoding/decoding.
> > Only phoenix encoding is not in Apache Hbase Spark.
> >
> > We have shepherded almost all of SHC changes from SHC Git repo to Apach=
e
> > Hbase except the features of supporting multiple secure Hbase clusters
> and
> > Phoenix data coder. Next week we=E2=80=99ll have a discussion to decide=
 whether
> > these latest code changes will be shepherded to Apache Hbase since SHC
> has
> > used Phoenix encoding/decoding to support Phoenix data. I'll update the
> > next steps here after the discussion next week.
> >
> > For Composite Key, the current patch is still under reviewing, but it
> > brings some concerns. That's also one of the reasons to bring the Phoen=
ix
> > encoding/decoding in SHC.
> >
> > Regards,
> >
> > Weiqing
> >
> > On Fri, Jun 23, 2017 at 12:20 PM, Stack <stack@duboce.net> wrote:
> >
> > > On Fri, Jun 23, 2017 at 10:30 AM, Sean Busbey <busbey@apache.org>
> wrote:
> > >
> > > > On Fri, Jun 23, 2017 at 12:06 PM, Stack <stack@duboce.net> wrote:
> > > > > On Wed, Jun 21, 2017 at 9:31 AM, Sean Busbey <busbey@apache.org>
> > > wrote:
> > > > >....
> > > > > I don't know enough about the integration but is the 'handling of
> > > Phoenix
> > > > > encoded data' about mapping spark types to a serialization in
> hbase?
> > If
> > > > > not, where is the need for seamless transforms between spark type=
s
> > and
> > > a
> > > > > natural hbase serialization listed. We need this IIRC.
> > > > >
> > > >
> > > > It's a subtask, really. We already have a pluggable system for
> mapping
> > > > between spark types and a couple of serialization options (the docs
> > > > need improvement?).
> > > >
> > >
> > >
> > > > SHC has its own pluggable system and has the addition of a phoenix
> > > > encoding. The set seems like the most likely out-of-the-box formats
> > > > folks might have something in. (I thinkMaybe Kite? I think it's
> > > > different than the rest.)
> > > >
> > > > Or are you saying we can just map all of it the the hbase-common
> > > > "types" and then do the pluggable part under it?
> > > >
> > >
> > >
> > > Not making any prescription. Was just worried about type marshalling =
in
> > and
> > > out of spark concerned that the serialization would be other than
> > something
> > > 'natural' for hbase, that it not performant, and that we might have a
> > > profusion of mechanisms.
> > >
> > > If a noted subtask, thats grand.
> > >
> > > Thanks,
> > > S
> > >
> >
>

--94eb2c084e84c3ce410552a6b4d0--