From users-return-59-archive-asf-public=cust-asf.ponee.io@hudi.apache.org Wed Oct 14 23:03:06 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id D374D18063F for ; Thu, 15 Oct 2020 01:03:06 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id 5280A63F6D for ; Wed, 14 Oct 2020 23:03:06 +0000 (UTC) Received: (qmail 32128 invoked by uid 500); 14 Oct 2020 23:03:05 -0000 Mailing-List: contact users-help@hudi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@hudi.apache.org Delivered-To: mailing list users@hudi.apache.org Received: (qmail 32118 invoked by uid 99); 14 Oct 2020 23:03:05 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2020 23:03:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id B8922BFE11 for ; Wed, 14 Oct 2020 23:03:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id 8JmAOiEjuOe9 for ; Wed, 14 Oct 2020 23:03:04 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::731; helo=mail-qk1-x731.google.com; envelope-from=bdighe@gmail.com; receiver= Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 077AF7F9CB for ; Wed, 14 Oct 2020 23:03:03 +0000 (UTC) Received: by mail-qk1-x731.google.com with SMTP id z6so926944qkz.4 for ; Wed, 14 Oct 2020 16:03:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=OWiGvfZiiGmiSt9UoGL8WnQMVecOU5uLu87JMlD5zWg=; b=RFj8u+TSrS7rCq/jOA7KKF5gScJAZwSWetD/mY9RfpY2kDkb5/Oh/dAkBQaZxPGj8z iOchbR4lvrEmbjkIJaTzdJ1z9VmIMkD/nDZpPrqmsXSZUFA+4S6PHQadhtUceIJkS+DT hiovU/iO7PfSnpUGtZNOZgThTEcdi4Dnth0KkT1yC0YqwWJJEEYx6J+TWppvXYO8mdTd 3p77nroGaTNwVlJKmwlUvMn2pQU69nZb4N+h9GenG3w0e52JPZ/89WagNAxO/nJxASbZ W5nREA3ppUmvso5LYShVt9f3H5k3h4Q4zteJ0x6deqXYF25qbMHHlkq0ruxhXDLMPV60 eM0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=OWiGvfZiiGmiSt9UoGL8WnQMVecOU5uLu87JMlD5zWg=; b=AoaZJc7B9AkaLb5F1ZhrRk8ee9P0LTAV04XnSAcvuZIrj+bnnQD51VjDh7rbSf4Tu8 L5LBf/vzQ5iCjEP6TyDFZ3AXg3nTeOYQ2C362QigzpJKbLRLJEXvH8oX+BHRD+cSGle8 OWy01Dd4gl3Fb83vZGlkM/G1TVFTcaQt8hbFKVRnAEYoU1m2LGkM3FLztj+rF/ddycP2 S9M3G/LLY/c5Zsq2k4MWGvy/Hs+vu/617/ubAD5RrI4PAKwnULGZauIj0eAh96qKs9ZI ZNl4yd3kLXdqgx2gvCrpFKe/dQz9gQX9qQ0XDPpgCphNeb6TFb+Tr4zfmaM/yLZrqRZs dedg== X-Gm-Message-State: AOAM530F7bycnEtZ4DjLlFQfqVvaeupTqrvq9I7n/V2C73sSJgp+xHYv jFEmp73ePJJDVIprE4HCquOH1gzXKV5yxr+jiTF7aoW9pLx32A== X-Google-Smtp-Source: ABdhPJz2WZ8mNZnqvtFNgivd0oVB5z87I1K8XhA+BYp2NvUCgMpL8DUU8l11c8MXJI5K2bkgfKw90CGFg5p4mpH0+3k= X-Received: by 2002:a37:dcc3:: with SMTP id v186mr1399574qki.218.1602716582537; Wed, 14 Oct 2020 16:03:02 -0700 (PDT) MIME-Version: 1.0 From: Bharat Dighe Date: Wed, 14 Oct 2020 16:02:51 -0700 Message-ID: Subject: Not able to query real time table when rows contains nested elements To: users@hudi.apache.org Content-Type: multipart/alternative; boundary="00000000000093ece505b1a988dc" --00000000000093ece505b1a988dc Content-Type: text/plain; charset="UTF-8" Hi, I have a MOR hudi table created with records which has some nested elements. I am doing it in the docker demo environment. I get an exception when I run a select query with columns which are nested for real time view. For example: 1) spark.sql("select name, experience from users_mor_ro") //works fine for RO view 2) spark.sql("select name from users_mor_rt") //works fine for RT view 3) spark.sql("select name, experience from users_mor_rt") //fails RT view The 'experience' above is a nested field. I am seeing the following exception. 20/10/11 19:53:58 ERROR executor.Executor: Exception in task 0.0 in stage 147.0 (TID 153) java.lang.UnsupportedOperationException: Cannot inspect org.apache.hadoop.io.Text at org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector.getStructFieldData(ArrayWritableObjectInspector.java:152) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$4$$anonfun$apply$7.apply(HiveInspectors.scala:688) I have created https://issues.apache.org/jira/browse/HUDI-1340 I have added my code, avro files, and scala code to this JIRA. Queries work fine with Hive. Please share if there is a workaround. Thanks Bharat --00000000000093ece505b1a988dc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

I have a MOR hudi table created wit= h records which has some nested elements. I am doing it in the docker demo = environment.
I get an exception=C2=A0when I run a select query wi= th columns which are nested for real time view. For example:
1) s= park.sql("select name, experience from users_mor_ro") //works fin= e for RO view
2) spark.sql("select name from users_mor_= rt") //works fine for RT view
3) spark.sql("select name= , experience from users_mor_rt") //fails RT view

<= div>The 'experience' above is a nested field.

<= div>I am seeing the following exception.

20/10/11 19:53= :58 ERROR executor.Executor: Exception in task 0.0 in stage 147.0 (TID 153)= java.lang.UnsupportedOperationException: Cannot inspect org.apache.hadoop.= io.Text at org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectIn= spector.getStructFieldData(ArrayWritableObjectInspector.java:152) at org.ap= ache.spark.sql.hive.HiveInspectors$$anonfun$4$$anonfun$apply$7.apply(HiveIn= spectors.scala:688)

I have added my code, avr= o files, and scala code to this JIRA.

Queries work= fine with Hive.

Please share if there is a workar= ound.

Thanks
Bharat
--00000000000093ece505b1a988dc--