hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@apache.org>
Subject Re: Writing to HBase from Spark
Date Tue, 18 Apr 2017 19:22:34 GMT
PySpark just runs a normal Python interpreter, right? (It doesn't use e.g.
Jython?) If that's the case then the hbase-spark module as it currently
stands isn't going to buy folks anything because it is all JVM based.

AFAIK, your best bet is going to be to use one or more thrift proxy
services for HBase and then rely on the python binding for that thrift RPC
to talk to HBase.

-busbey

On Tue, Apr 18, 2017 at 2:10 PM Ted Yu <yuzhihong@gmail.com> wrote:

> Not that I know of.
>
> On Tue, Apr 18, 2017 at 12:02 PM, Eric Wilson <eric.wilson@aver.io> wrote:
>
> > So if I'm using PySpark 2.1 and HBase 1.2.5 there is no way for them to
> > communicate with each other?
> >
> > Eric
> >
> > On Tue, Apr 18, 2017 at 2:03 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > There is some document here (slightly outdated) :
> > > http://hbase.apache.org/book.html#spark
> > >
> > > Currently hbase-spark module in master branch only supports Spark 1.6.x
> > >
> > > See HBASE-16179 for on-going work for Spark 2.y
> > >
> > > FYI
> > >
> > > On Tue, Apr 18, 2017 at 10:58 AM, Eric Wilson <eric.wilson@aver.io>
> > wrote:
> > >
> > > > I've been spending a few days trying to figure out how to write to
> > HBase
> > > > from a Spark RDD -- using Python.
> > > >
> > > > I recently found that Spark has removed some of it's
> pythonconverters,
> > > > because (as this commit
> > > > <
> https://github.com/apache/spark/commit/a680562a6f87a03a00f71bad1c4242
> > > > 67ae75c641>
> > > >  shows):
> > > > *"HBase has had Spark bindings for **a while and is even including
> them
> > > in
> > > > the HBase distribution in the next **version, making the examples
> > > obsolete.
> > > > The same applies to Cassandra, which **seems to have a proper Spark
> > > binding
> > > > library already."*
> > > >
> > > > Can anyone point me to documentation of Spark/HBase/Python
> integration?
> > > >
> > > > Thanks.
> > > >
> > > >
> > > > --
> > > >
> > > > <http://aver.io> <http://aver.io>[image: aver] <http://aver.io>
> > > > Eric Wilson
> > > > Software Engineer
> > > >
> > > > --
> > > >
> > > > ------------------------------
> > > > This email, including attachments, may contain information that is
> > > > privileged, confidential or is exempt from disclosure under
> applicable
> > > law
> > > > (including, but not limited to, protected health information). It is
> > not
> > > > intended for transmission to, or receipt by, any unauthorized
> persons.
> > If
> > > > the reader of this message is not the intended recipient, or the
> > employee
> > > > or agent responsible for delivering the message to the intended
> > > recipient,
> > > > you are hereby notified that any dissemination, distribution or
> copying
> > > of
> > > > this communication is strictly prohibited. If you believe this email
> > was
> > > > sent to you in error, do not read it. Please notify the sender
> > > immediately
> > > > informing them of the error and delete all copies and attachments of
> > the
> > > > message from your system. Thank you.
> > > >
> > >
> >
> >
> >
> > --
> >
> >
> > --
> >
> > <http://aver.io>[image: aver] <http://aver.io> <http://aver.io>
> > Eric Wilson
> > Software Engineer
> >
> > --
> >
> > ------------------------------
> > This email, including attachments, may contain information that is
> > privileged, confidential or is exempt from disclosure under applicable
> law
> > (including, but not limited to, protected health information). It is not
> > intended for transmission to, or receipt by, any unauthorized persons. If
> > the reader of this message is not the intended recipient, or the employee
> > or agent responsible for delivering the message to the intended
> recipient,
> > you are hereby notified that any dissemination, distribution or copying
> of
> > this communication is strictly prohibited. If you believe this email was
> > sent to you in error, do not read it. Please notify the sender
> immediately
> > informing them of the error and delete all copies and attachments of the
> > message from your system. Thank you.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message