Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E3DEE200D1E for ; Wed, 4 Oct 2017 07:49:09 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id E2B971609DE; Wed, 4 Oct 2017 05:49:09 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 34D94160BD5 for ; Wed, 4 Oct 2017 07:49:09 +0200 (CEST) Received: (qmail 90395 invoked by uid 500); 4 Oct 2017 05:49:08 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 90386 invoked by uid 99); 4 Oct 2017 05:49:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Oct 2017 05:49:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 45D321A07B0 for ; Wed, 4 Oct 2017 05:49:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id uobWRqyLCwIm for ; Wed, 4 Oct 2017 05:49:06 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 6EF695FB57 for ; Wed, 4 Oct 2017 05:49:06 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id B18DCE0EEF for ; Wed, 4 Oct 2017 05:49:04 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id BB74F242EB for ; Wed, 4 Oct 2017 05:49:02 +0000 (UTC) Date: Wed, 4 Oct 2017 05:49:02 +0000 (UTC) From: "Asif Hussain Shahid (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (SPARK-22192) An RDD of nested POJO objects cannot be converted into a DataFrame using SQLContext.createDataFrame API MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 04 Oct 2017 05:49:10 -0000 [ https://issues.apache.org/jira/browse/SPARK-22192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190852#comment-16190852 ] Asif Hussain Shahid commented on SPARK-22192: --------------------------------------------- Not at all. I have added a bug test in org.apache.spark.sql.SQLContextSuite in the branch SPARK-22192 > An RDD of nested POJO objects cannot be converted into a DataFrame using SQLContext.createDataFrame API > ------------------------------------------------------------------------------------------------------- > > Key: SPARK-22192 > URL: https://issues.apache.org/jira/browse/SPARK-22192 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.0 > Environment: Independent of OS / platform > Reporter: Asif Hussain Shahid > Priority: Minor > > If an RDD contains nested POJO objects, then SQLContext.createDataFrame(RDD, Class) api only handles the top level POJO object. It throws ScalaMatchError exception when handling the nested POJO object as the code does not recursively handle the nested POJOs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org