spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-14536) NPE in JDBCRDD when array column contains nulls (postgresql)
Date Thu, 22 Sep 2016 02:01:21 GMT

    [ https://issues.apache.org/jira/browse/SPARK-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511837#comment-15511837
] 

Hyukjin Kwon commented on SPARK-14536:
--------------------------------------

I see. I rushed to read this and didn't noticed that this is actually a PostgreSQL specific
issue (I thought this JIRA describes a general JDBC problem).
Yea, {{ArrayType}} seems only supported in {{PostgreSQL}} in Spark. Maybe we should make some
relations with those JIRAs with SPARK-8500 to prevent confusion.

> NPE in JDBCRDD when array column contains nulls (postgresql)
> ------------------------------------------------------------
>
>                 Key: SPARK-14536
>                 URL: https://issues.apache.org/jira/browse/SPARK-14536
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.1
>            Reporter: Jeremy Smith
>              Labels: NullPointerException
>
> At https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala#L453
it is assumed that the JDBC driver will definitely return a non-null `Array` object from the
call to `getArray`, and that in the event of a null array it will return an non-null `Array`
object with a null underlying array.  But as you can see here https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/jdbc/PgResultSet.java#L387
that isn't the case, at least for PostgreSQL.  This causes a `NullPointerException` whenever
an array column contains null values. It seems like the PostgreSQL JDBC driver is probably
doing the wrong thing, but even so there should be a null check in JDBCRDD.  I'm happy to
submit a PR if that would be helpful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message