Return-Path: X-Original-To: apmail-kudu-user-archive@minotaur.apache.org Delivered-To: apmail-kudu-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D917618AAC for ; Sat, 9 Apr 2016 21:00:15 +0000 (UTC) Received: (qmail 70800 invoked by uid 500); 9 Apr 2016 21:00:15 -0000 Delivered-To: apmail-kudu-user-archive@kudu.apache.org Received: (qmail 70736 invoked by uid 500); 9 Apr 2016 21:00:15 -0000 Mailing-List: contact user-help@kudu.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kudu.incubator.apache.org Delivered-To: mailing list user@kudu.incubator.apache.org Received: (qmail 70728 invoked by uid 99); 9 Apr 2016 21:00:15 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Apr 2016 21:00:15 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 319531A039D for ; Sat, 9 Apr 2016 21:00:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -2.021 X-Spam-Level: X-Spam-Status: No, score=-2.021 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001] autolearn=disabled Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id jLa2TKccwsUz for ; Sat, 9 Apr 2016 21:00:13 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with SMTP id 566D25F2ED for ; Sat, 9 Apr 2016 21:00:12 +0000 (UTC) Received: (qmail 70706 invoked by uid 99); 9 Apr 2016 21:00:11 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Apr 2016 21:00:11 +0000 Received: from mail-qg0-f47.google.com (mail-qg0-f47.google.com [209.85.192.47]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 529A31A003F for ; Sat, 9 Apr 2016 21:00:11 +0000 (UTC) Received: by mail-qg0-f47.google.com with SMTP id j35so116827192qge.0 for ; Sat, 09 Apr 2016 14:00:11 -0700 (PDT) X-Gm-Message-State: AD7BkJJOq8QXUPONRhx+BIsi+unv8N68TGvhZ8sbqJ48+ZWThoTA04bxQc3lLx+un1g8L7a8saj71Rp0EtPOXQ== MIME-Version: 1.0 X-Received: by 10.140.16.165 with SMTP id 34mr19521686qgb.79.1460235610536; Sat, 09 Apr 2016 14:00:10 -0700 (PDT) Received: by 10.55.179.5 with HTTP; Sat, 9 Apr 2016 14:00:10 -0700 (PDT) In-Reply-To: <57D278EF-7D6D-4021-A84F-44EAB26F25DC@gmail.com> References: <99411687-F4C6-4DD5-896D-132543AAE005@amobee.com> <57D278EF-7D6D-4021-A84F-44EAB26F25DC@gmail.com> Date: Sat, 9 Apr 2016 14:00:10 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Spark on Kudu From: Jean-Daniel Cryans To: user@kudu.incubator.apache.org Content-Type: multipart/alternative; boundary=001a11c041ccdac8f805301398d8 --001a11c041ccdac8f805301398d8 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Ha first time I'm hearing about SMACK. Inside Cloudera we like to refer to "Impala + Kudu" as Kimpala, but yeah it's not as sexy. My colleagues who were also there did say that the hype around Spark isn't dying down. There's definitely an overlap in the use cases that Cassandra, HBase, and Kudu cater to. I wouldn't go as far as saying that C* is just an interim solution for the use case you describe. Nothing significant happened in Kudu over the past month, it's a storage engine so things move slowly *smile*. I'd love to see more contributions on the Spark front. I know there's code out there that could be integrated in kudu-spark, it just needs to land in gerrit. I'm sure folks will happily review it. Do you have relevant experiences you can share? I'd love to learn more about the use cases for which you envision using Kudu as a C* replacement. Thanks, J-D On Fri, Apr 8, 2016 at 12:45 PM, Benjamin Kim wrote: > Hi J-D, > > My colleagues recently came back from Strata in San Jose. They told me > that everything was about Spark and there is a big buzz about the SMACK > stack (Spark, Mesos, Akka, Cassandra, Kafka). I still think that Cassandr= a > is just an interim solution as a low-latency, easily queried data store. = I > was wondering if anything significant happened in regards to Kudu, > especially on the Spark front. Plus, can you come up with your own propos= ed > stack acronym to promote? > > Cheers, > Ben > > > On Mar 1, 2016, at 12:20 PM, Jean-Daniel Cryans > wrote: > > Hi Ben, > > AFAIK no one in the dev community committed to any timeline. I know of on= e > person on the Kudu Slack who's working on a better RDD, but that's about = it. > > Regards, > > J-D > > On Tue, Mar 1, 2016 at 11:00 AM, Benjamin Kim wrote: > >> Hi J-D, >> >> Quick question=E2=80=A6 Is there an ETA for KUDU-1214? I want to target = a version >> of Kudu to begin real testing of Spark against it for our devs. At least= , I >> can tell them what timeframe to anticipate. >> >> Just curious, >> *Benjamin Kim* >> *Data Solutions Architect* >> >> [a=E2=80=A2mo=E2=80=A2bee] *(n.)* the company defining digital marketing= . >> >> *Mobile: +1 818 635 2900 <%2B1%20818%20635%202900>* >> 3250 Ocean Park Blvd, Suite 200 | Santa Monica, CA 90405 | >> www.amobee.com >> >> On Feb 24, 2016, at 3:51 PM, Jean-Daniel Cryans >> wrote: >> >> The DStream stuff isn't there at all. I'm not sure if it's needed either= . >> >> The kuduRDD is just leveraging the MR input format, ideally we'd use >> scans directly. >> >> The SparkSQL stuff is there but it doesn't do any sort of pushdown. It's >> really basic. >> >> The goal was to provide something for others to contribute to. We have >> some basic unit tests that others can easily extend. None of us on the t= eam >> are Spark experts, but we'd be really happy to assist one improve the >> kudu-spark code. >> >> J-D >> >> On Wed, Feb 24, 2016 at 3:41 PM, Benjamin Kim wrote= : >> >>> J-D, >>> >>> It looks like it fulfills most of the basic requirements (kudu RDD, kud= u >>> DStream) in KUDU-1214. Am I right? Besides shoring up more Spark SQL >>> functionality (Dataframes) and doing the documentation, what more needs= to >>> be done? Optimizations? >>> >>> I believe that it=E2=80=99s a good place to start using Spark with Kudu= and >>> compare it to HBase with Spark (not clean). >>> >>> Thanks, >>> Ben >>> >>> >>> On Feb 24, 2016, at 3:10 PM, Jean-Daniel Cryans >>> wrote: >>> >>> AFAIK no one is working on it, but we did manage to get this in for >>> 0.7.0: https://issues.cloudera.org/browse/KUDU-1321 >>> >>> It's a really simple wrapper, and yes you can use SparkSQL on Kudu, but >>> it will require a lot more work to make it fast/useful. >>> >>> Hope this helps, >>> >>> J-D >>> >>> On Wed, Feb 24, 2016 at 3:08 PM, Benjamin Kim >>> wrote: >>> >>>> I see this KUDU-1214 ta= rgeted >>>> for 0.8.0, but I see no progress on it. When this is complete, will th= is >>>> mean that Spark will be able to work with Kudu both programmatically a= nd as >>>> a client via Spark SQL? Or is there more work that needs to be done on= the >>>> Spark side for it to work? >>>> >>>> Just curious. >>>> >>>> Cheers, >>>> Ben >>>> >>>> >>> >>> >> >> > > --001a11c041ccdac8f805301398d8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Ha first time I'm hearing about SMACK. Inside Cloudera= we like to refer to "Impala + Kudu" as Kimpala, but yeah it'= s not as sexy. My colleagues who were also there did say that the hype arou= nd Spark isn't dying down.

There's definitely an overl= ap in the use cases that Cassandra, HBase, and Kudu cater to. I wouldn'= t go as far as saying that C* is just an interim solution for the use case = you describe.

Nothing significant happened i= n Kudu over the past month, it's a storage engine so things move slowly= *smile*. I'd love to see more contributions on the Spark front. I know= there's code out there that could be integrated in kudu-spark, it just= needs to land in gerrit. I'm sure folks will happily review it.
<= div>
Do you have relevant experiences you can share? I'd = love to learn more about the use cases for which you envision using Kudu as= a C* replacement.

Thanks,

J-D

= On Fri, Apr 8, 2016 at 12:45 PM, Benjamin Kim <bbuild11@gmail.com>= wrote:
Hi J-D,

My colleagues recently came back from = Strata in San Jose. They told me that everything was about Spark and there = is a big buzz about the SMACK stack (Spark, Mesos, Akka, Cassandra, Kafka).= I still think that Cassandra is just an interim solution as a low-latency,= easily queried data store. I was wondering if anything significant happene= d in regards to Kudu, especially on the Spark front. Plus, can you come up = with your own proposed stack acronym to promote?

C= heers,
Ben


On Mar 1, 2016, at 12:20 PM, Jean-D= aniel Cryans <j= dcryans@apache.org> wrote:

Hi Ben,
AFAIK no one in the dev community committed to any timelin= e. I know of one person on the Kudu Slack who's working on a better RDD= , but that's about it.

Regards,

=
J-D

On Tue, Mar 1, 2016 at 11:00 AM, Benjamin Kim &l= t;bkim@amobee.com&= gt; wrote:
Hi J-D,

Quick question=E2=80=A6 Is there an ETA for KUDU-1214? I want to targe= t a version of Kudu to begin real testing of Spark against it for our devs.= At least, I can tell them what timeframe to anticipate.

Just curious,
Benjamin Kim
Data Solutions Architect

[a=E2=80=A2mo=E2=80= =A2bee]=C2=A0(n.)=C2=A0the company defining digital marketing.

M= obile: +1 818 635 2900
3250= Ocean Park Blvd, Suite 200 =C2=A0|=C2=A0 Santa Monica, CA 90405=C2=A0 |=C2= =A0=C2=A0www.amobee.com=

On Feb 24, 2016, at 3:51 PM, Jean-Daniel Cryans <jdcryans@apache.org> wrote:
The DStream stuff isn't there at all. I'm not sure= if it's needed either.

The kuduRDD is just leveraging the MR input format, ideally we'd u= se scans directly.

The SparkSQL stuff is there but it doesn't do any sort of pushdown= . It's really basic.

The goal was to provide something for others to contribute to. We have= some basic unit tests that others can easily extend. None of us on the tea= m are Spark experts, but we'd be really happy to assist one improve the= kudu-spark code.

J-D

On Wed, Feb 24, 2016 at 3:41 PM, Benjamin Kim <bbuild11@gmail.= com> wrote:
J-D,

It looks like it fulfills most of the basic requirements (kudu RDD, ku= du DStream) in KUDU-1214. Am I right? Besides shoring up more Spark SQL fun= ctionality (Dataframes) and doing the documentation, what more needs to be = done? Optimizations?

I believe that it=E2=80=99s a good place to start using Spark with Kud= u and compare it to HBase with Spark (not clean).

Thanks,
Ben


On Feb 24, 2016, at 3:10 PM, Jean-Daniel Cryans <jdcryans@apache.org> wrote:
AFAIK no one is working on it, but we did manage to get th= is in for 0.7.0:=C2=A0https://issues.cloudera.org/browse/KUDU-1321

It's a really simple wrapper, and yes you can use SparkSQL on Kudu= , but it will require a lot more work to make it fast/useful.

Hope this helps,

J-D

On Wed, Feb 24, 2016 at 3:08 PM, Benjamin Kim <bbuild11@gmail.= com> wrote:
I see this=C2=A0KUDU-1214=C2=A0targ= eted for 0.8.0, but I see no progress on it. When this is complete, will th= is mean that Spark will be able to work with Kudu both programmatically and as a client via Spark SQL? Or = is there more work that needs to be done on the Spark side for it to work?

Just curious.

Cheers,
Ben







--001a11c041ccdac8f805301398d8--