Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 894E4200B74 for ; Thu, 18 Aug 2016 08:17:26 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 87C7D160AB5; Thu, 18 Aug 2016 06:17:26 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 83CEC160A8C for ; Thu, 18 Aug 2016 08:17:25 +0200 (CEST) Received: (qmail 94683 invoked by uid 500); 18 Aug 2016 06:17:24 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 94672 invoked by uid 99); 18 Aug 2016 06:17:23 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Aug 2016 06:17:23 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 2E88D18064C for ; Thu, 18 Aug 2016 06:17:23 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.48 X-Spam-Level: * X-Spam-Status: No, score=1.48 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, KAM_TRACKIMAGE=0.2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=lateral-thoughts-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id MnxGV8FWK4st for ; Thu, 18 Aug 2016 06:17:18 +0000 (UTC) Received: from mail-qt0-f177.google.com (mail-qt0-f177.google.com [209.85.216.177]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id D095E5F642 for ; Thu, 18 Aug 2016 06:17:11 +0000 (UTC) Received: by mail-qt0-f177.google.com with SMTP id w38so3999206qtb.0 for ; Wed, 17 Aug 2016 23:17:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lateral-thoughts-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:message-id:date:in-reply-to:references :delivered-to:mime-version; bh=RE9p/F2poooLr+xXZ+GJ0qLAx59mjdPnWpA0FcR106w=; b=xpdkLkXC5xrHm2Q0f+cZx97cfRls0oMYVmAlawouVG1qbXAvLC6NTB8+OuKYvYchy9 vMQuhoTTvdPjSEZQZnty31mW5fmqJ3YWci80ew4F92b/cOkXYGvyaWiLUHe9xjNxpioK JJRtd3X5+TDXNEUuZKwQkHmIDu+vF1d5wzNS8qrRxYLVIPgw/AHbVdnRvDLYc7USasx5 GnR0r4FFL3ObUSYsnwnWFTQqq9hOHv2BxZgV23wSRNOrWLqdYogxUY/kQLVzLgpEEH8Q vePHhj1TrSRLOzzqKP7SgnphdmZ/bUsLVU7xga4blx33nu91l1EdRRg9xNusH5rHWkS9 D8yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:message-id:date:in-reply-to :references:delivered-to:mime-version; bh=RE9p/F2poooLr+xXZ+GJ0qLAx59mjdPnWpA0FcR106w=; b=mSst/9O2xRcy3b0Q2X/IS4Uc0a7/WtejcQTeB6Sok9F9XHoYVPXN/oEM9MvfGSvUFu +PiPFYuuYCHbcWTxZ6HvcpAF1Vk2tgwmYU+xf9U9HdCjktcka6oT4gXU9ym/sJaFh1k8 KB5N5Te2FbHBVIj0+BoHjJQnRLm3M56YPwv45vgIo1FyIRXL7r2wG5SRNTwoU0FBzJsk w3P8Ax0h0b671ungQUZPmMipY6naxcRlC8XwyEd4diw3g3tLSIsWRIkXf6luuAMqjLfk PzGyyScSSxZY1E1xjYGrgDdlYXXsSnTjb0MWaFsqqxnFs8ORgrlUhR+eXcBCgtCHaKff jY+A== X-Gm-Message-State: AEkoouuAeCzGQT8ez1v2TVtn2EPKufoa/qsL1RJ4V0XhyT9MTdbqlSKSL8NC2pufYMEX8ywL X-Received: by 10.237.43.71 with SMTP id p65mr562699qtd.116.1471501025142; Wed, 17 Aug 2016 23:17:05 -0700 (PDT) Received: from [127.0.0.1] (send-14.prod.mixmax-mailer.com. [52.200.43.104]) by smtp.gmail.com with ESMTPSA id l63sm293220qkf.16.2016.08.17.23.17.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Aug 2016 23:17:04 -0700 (PDT) Content-Type: multipart/alternative; boundary="----sinikael-?=_1-14715010233680.9582477940712124" From: Olivier Girardot To: "=?UTF-8?Q?Maciej_Bry=C5=84ski?=" Cc: Amit Sela , Jacek Laskowski , dev Subject: Re: Spark SQL and Kryo registration Message-Id: <1471501020206-d95f6700-8c5cb156-b3555660@mixmax.com> Date: Thu, 18 Aug 2016 06:16:59 +0000 In-Reply-To: References: <1470214190966-a2b29d1b-b51cdd19-a6ca17e0@mixmax.com> X-Mixmax-Message-Id: he47N9boUxTt9pRgG Delivered-To: rsm@mixmax.com X-Mailer: Mixmax (mixmax.com) MIME-Version: 1.0 archived-at: Thu, 18 Aug 2016 06:17:26 -0000 ------sinikael-?=_1-14715010233680.9582477940712124 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi everyone, it seems that it works now out of the box. So nevermind, = registration is compatible with spark 2.0 when using dataframes. Regards, Olivier. On Fri, Aug 5, 2016 10:07 AM, Maciej Bry=C5=84ski = maciek@brynski.pl wrote: Hi Olivier, Did you check performance of Kryo ? I = have observations that Kryo is slightly slower than Java Serializer. Regards, Maciek 2016-08-04 17:41 GMT+02:00 Amit Sela < amitsela33@gmail.com= > : It should. Codegen uses the SparkConf in SparkEnv when instantiating a= new Serializer. On Thu, Aug 4, 2016 at 6:14 PM Jacek Laskowski < = jacek@japila.pl > wrote: Hi Olivier, I don't know either, but am curious = what you've tried already. Jacek On 3 Aug 2016 10:50 a.m., "Olivier = Girardot" < o.girardot@lateral-thoughts. com > wrote: Hi everyone, I'm currently to use Spark 2.0.0 and making Dataframes work = with kryo. registrationRequired=3Dtrue Is it even possible at all = considering the codegen ? Regards, Olivier Girardot | Associ=C3=A9 o.girardot@lateral-thoughts. com +33 6 24 09 17 94 -- Maciek Bry=C5=84ski Olivier Girardot | Associ=C3=A9 o.girardot@lateral-thoughts.com +33 6 24 09 17 94 ------sinikael-?=_1-14715010233680.9582477940712124 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable =20 =20 =20 =20 =20 =20 =20 =20
Hi everyone,= =C2=A0
it seems that it works now out of the box.
So = nevermind, registration is compatible with spark 2.0 when using dataframes.=

Regards,=C2=A0

Olivier.=
3D""



On Fri, Aug 5, 2016 10:07 AM, Maciej Bry=C5=84ski = maciek@brynski.= pl wrote:
Hi Olivier,
Did you check = performance of Kryo ?
I have observations that Kryo is slightly = slower than Java Serializer.

Regards,=
Maciek

2016-08-04 17:41 GMT+02:00 Amit = Sela <amitsela33@gmail.com>:
It should. Codegen uses the SparkConf in SparkEnv when = instantiating a new Serializer.

On= Thu, Aug 4, 2016 at 6:14 PM Jacek Laskowski <jacek@japila.pl> = wrote:

Hi Olivier,

I don't know either, but am curious what you've tried = already.

Jacek


On 3 Aug 2016 10:50 a= .m., "Olivier Girardot" <o.girardot@lateral-thought= s.com> wrote:
=20 =20 =20 =20 =20 =20 =20 =20 =20 =20 =20
=20 =20 =20 =20
=20
Hi everyone,=C2=A0
I'm currently to= use Spark 2.0.0 and making Dataframes work with=C2=A0kryo.= registrationRequired=3Dtrue
Is it even possible at all = considering the codegen ?

Regards, = =C2=A0

Olivier Girardot=C2=A0| Associ=C3=A9
3D"" =20



-- =
Maciek Bry=C5=84ski


Olivier Girardot=C2=A0| = Associ=C3=A9
------sinikael-?=_1-14715010233680.9582477940712124--