spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-8868) SqlSerializer2 can go into infinite loop when row consists only of NullType columns
Date Wed, 08 Jul 2015 01:53:04 GMT

     [ https://issues.apache.org/jira/browse/SPARK-8868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Josh Rosen resolved SPARK-8868.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 1.5.0
                   1.4.2

Issue resolved by pull request 7262
[https://github.com/apache/spark/pull/7262]

> SqlSerializer2 can go into infinite loop when row consists only of NullType columns
> -----------------------------------------------------------------------------------
>
>                 Key: SPARK-8868
>                 URL: https://issues.apache.org/jira/browse/SPARK-8868
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.4.0, 1.5.0
>            Reporter: Josh Rosen
>            Assignee: Yin Huai
>            Priority: Minor
>             Fix For: 1.4.2, 1.5.0
>
>
> The following SQL query will cause an infinite loop in SqlSerializer2 code:
> {code}
> val df = sqlContext.sql("select null where 1 = 1")
> df.unionAll(df).sort("_c0").collect()
> {code}
> The same problem occurs if we add more null-literals, but does not occur as long as there
is a column of any other type (e.g. {{select 1, null where 1 == 1}}).
> I think that what's happening here is that if you have a row that consists only of columns
of NullType (not columns of other types which happen to only contain null values, but only
columns of null literals), SqlSerializer will end up writing / reading no data for rows of
this type.  Since the deserialization stream will never try to read any data but nevertheless
will be able to return an empty row, DeserializationStream.asIterator will go into an infinite
loop since there will never be a read to trigger an EOF exception.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message