spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dian Fay (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-22220) Spark SQL: LATERAL VIEW OUTER null pointer exception with GROUP BY
Date Mon, 09 Oct 2017 23:56:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-22220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16197899#comment-16197899
] 

Dian Fay edited comment on SPARK-22220 at 10/9/17 11:55 PM:
------------------------------------------------------------

I did find a slightly more detailed mention in the Zeppelin logs, but it's no stack trace:

{code}
WARN [2017-10-09 23:28:39,445] ({pool-2-thread-32} NotebookServer.java[afterStatusChange]:2058)
- Job 20171006-205907_443943867 is finished, status: ERROR, exception: null, result: %text
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 549.0 failed
4 times, most recent failure: Lost task 0.3 in stage 549.0 (TID 28218, $hostname, executor
1): java.lang.NullPointerException
{code}


was (Author: dmfay):
I did find a slightly more detailed mention in the Zeppelin logs, but it's no stack trace:

WARN [2017-10-09 23:28:39,445] ({pool-2-thread-32} NotebookServer.java[afterStatusChange]:2058)
- Job 20171006-205907_443943867 is finished, status: ERROR, exception: null, result: %text
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 549.0 failed
4 times, most recent failure: Lost task 0.3 in stage 549.0 (TID 28218, wsdemo-dash03.dequecloud.com,
executor 1): java.lang.NullPointerException

> Spark SQL: LATERAL VIEW OUTER null pointer exception with GROUP BY
> ------------------------------------------------------------------
>
>                 Key: SPARK-22220
>                 URL: https://issues.apache.org/jira/browse/SPARK-22220
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.1.1
>         Environment: We have Zeppelin using Spark and Livy (error is reproducible without
Livy) on an Ambari cluster.
>            Reporter: Dian Fay
>
> Given a DataFrame having the fields name (a string) and tags (an array of strings), the
following Spark SQL query fails with a NullPointerException:
> {code}
> SELECT name, tag, COUNT(*)
> FROM records
> LATERAL VIEW OUTER explode(tags) AS tag
> GROUP BY name, tag
> {code}
> {code}
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 137.0
failed 4 times, most recent failure: Lost task 0.3 in stage 137.0 (TID 9109, $hostname, executor
1): java.lang.NullPointerException
> {code}
> The query is successful without the "outer", but obviously this excludes rows with empty
tags arrays. A version with outer but without aggregation also succeeds, making it possible
to work around this issue with a subquery:
> {code}
> SELECT name, tag
> FROM records
> LATERAL VIEW OUTER explode(tags) AS tag
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message