Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6C47417666 for ; Wed, 7 Oct 2015 11:08:18 +0000 (UTC) Received: (qmail 23940 invoked by uid 500); 7 Oct 2015 11:08:15 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 23841 invoked by uid 500); 7 Oct 2015 11:08:15 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 23829 invoked by uid 99); 7 Oct 2015 11:08:15 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Oct 2015 11:08:15 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id BF286C4165 for ; Wed, 7 Oct 2015 11:08:14 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.88 X-Spam-Level: ** X-Spam-Status: No, score=2.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id DP4spZ0WkQjl for ; Wed, 7 Oct 2015 11:08:07 +0000 (UTC) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id C5AA5203BA for ; Wed, 7 Oct 2015 11:08:06 +0000 (UTC) Received: by wicfx3 with SMTP id fx3so206861358wic.1 for ; Wed, 07 Oct 2015 04:08:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=5J9X7mrDEd2v5rTt5saEqk/DfbBbd3Y5XxxFgMf5tYs=; b=wbu8kycABj6uc5IrnzAUP8YipxRp0DPmxWrzDevqkKH0ACL3icE4JTL0OyetLkK516 OfV5aUVQIrdkXrNT0Rg7a3yrLLQNB0SujXKjm4+UPWQ3lfYPGlWgE2fbkmpmMrOJrvQN HAMZGizipf5GL0lIgnjDUhja13uMFwvUA1Y+xjci/CY19L0IvaxUo3KFSkTmfoDumMns O+JTn3Vln49YEgTYeUjrZwpOhSYxg7ZwI7LOjjeefv0ywX4VVpnXpPQsC+0ckAGWrXjM 9z1ASfN0vSRyXMZ+/w2HpI9yftCvkWr0o/gUr3tE+8iydAVka4zEaZEbgcTD9Mdcbjz5 APqw== X-Received: by 10.180.37.199 with SMTP id a7mr23399585wik.2.1444216085328; Wed, 07 Oct 2015 04:08:05 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.22.136 with HTTP; Wed, 7 Oct 2015 04:07:45 -0700 (PDT) In-Reply-To: References: From: Ophir Cohen Date: Wed, 7 Oct 2015 14:07:45 +0300 Message-ID: Subject: Re: GenericMutableRow and Row mismatch on Spark 1.5? To: Hemant Bhanawat Cc: user Content-Type: multipart/alternative; boundary=e89a8f503482beb96d052181c22a --e89a8f503482beb96d052181c22a Content-Type: text/plain; charset=UTF-8 Thanks! Can you check if you can provide example of the conversion? On Wed, Oct 7, 2015 at 2:05 PM, Hemant Bhanawat wrote: > Oh, this is an internal class of our project and I had used it without > realizing the source. > > Anyway, the idea is to wrap the InternalRow in a class that derives from > Row. When you implement the functions of the trait 'Row ', the type > conversions from Row types to InternalRow types has to be done for each of > the types. But, as I can see, the primitive types (apart from String) don't > need conversions. Map and Array would need some handling. > > I will check with the author of this code, I think this code can be > contributed to Spark. > > Hemant > www.snappydata.io > linkedin.com/company/snappydata > > On Wed, Oct 7, 2015 at 3:30 PM, Ophir Cohen wrote: > >> From which jar WrappedInternalRow comes from? >> It seems that I can't find it. >> >> BTW >> What I'm trying to do now is to create scala array from the fields and >> than create Row out of that array. >> The problem is that I get types mismatches... >> >> On Wed, Oct 7, 2015 at 8:03 AM, Hemant Bhanawat >> wrote: >> >>> An approach can be to wrap your MutableRow in WrappedInternalRow which >>> is a child class of Row. >>> >>> Hemant >>> www.snappydata.io >>> linkedin.com/company/snappydata >>> >>> >>> On Tue, Oct 6, 2015 at 3:21 PM, Ophir Cohen wrote: >>> >>>> Hi Guys, >>>> I'm upgrading to Spark 1.5. >>>> >>>> In our previous version (Spark 1.3 but it was OK on 1.4 as well) we >>>> created GenericMutableRow >>>> (org.apache.spark.sql.catalyst.expressions.GenericMutableRow) and return it >>>> as org.apache.spark.sql.Row >>>> >>>> Starting from Spark 1.5 GenericMutableRow isn't extends Row. >>>> >>>> What do you suggest to do? >>>> How can I convert GenericMutableRow to Row? >>>> >>>> Prompt answer will be highly appreciated! >>>> Thanks, >>>> Ophir >>>> >>> >>> >> > --e89a8f503482beb96d052181c22a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks!
Can you check if you can provid= e example of the conversion?

=
On Wed, Oct 7, 2015 at 2:05 PM, Hemant Bhana= wat <hemant9379@gmail.com> wrote:
Oh, this is an internal class of our project an= d I had used it without realizing the source.=C2=A0

Anyw= ay, the idea is to =C2=A0wrap the InternalRow in a class that derives from = Row. When you implement the functions of the trait 'Row ', the type= conversions from Row types to InternalRow types has to be done for each of= the types. But, as I can see, the primitive types (apart from String) don&= #39;t need conversions. Map and Array would need some handling.=C2=A0
=

I will check with the author of this code, I think= this code can be contributed to Spark.=C2=A0

Hemant=C2=A0

On Wed, Oct 7, 2015 at 3:30 PM, Ophir Cohen <ophchu@gmail.com> wrote:
From which jar WrappedInternalRow comes from?
It seems tha= t I can't find it.

BTW
What I'm trying to do = now is to create scala array from the fields and than create Row out of tha= t array.
The problem is that I get types mismatches...

On Wed, O= ct 7, 2015 at 8:03 AM, Hemant Bhanawat <hemant9379@gmail.com> wrote:
An approach = can be to wrap your MutableRow in WrappedInternalRow which is a child class= of Row.=C2=A0


On Tu= e, Oct 6, 2015 at 3:21 PM, Ophir Cohen <ophchu@gmail.com> wro= te:
=
Hi Guys,
I'm upgrading to Spark 1.5.
In our previous version (Spark 1.3 but it was OK on 1.4 as well) we= created GenericMutableRow (org.apache.spark.sql.catalyst.expressions.Gener= icMutableRow) and return it as org.apache.spark.sql.Row

Starti= ng from Spark 1.5 GenericMutableRow isn't extends Row.

Wha= t do you suggest to do?
How can I convert GenericMutableRow to Row= ?

Prompt answer will be highly appreciated!
Thanks,
Ophir




--e89a8f503482beb96d052181c22a--