spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saisai Shao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24008) SQL/Hive Context fails with NullPointerException
Date Thu, 19 Apr 2018 02:41:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443495#comment-16443495
] 

Saisai Shao commented on SPARK-24008:
-------------------------------------

The case you provided seems not so valid. You're trying to broadcast SQL entry point and RDDs,
which should not use broadcasting. I'm not sure if the issue here is due to invalid usage
pattern. If you met such issue, would you please provide a more meaningful case. 

> SQL/Hive Context fails with NullPointerException 
> -------------------------------------------------
>
>                 Key: SPARK-24008
>                 URL: https://issues.apache.org/jira/browse/SPARK-24008
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.6.3
>            Reporter: Prabhu Joseph
>            Priority: Major
>         Attachments: Repro
>
>
> SQL / Hive Context fails with NullPointerException while getting configuration from SQLConf.
This happens when the MemoryStore is filled with lot of broadcast and started dropping and
then SQL / Hive Context is created and broadcast. When using this Context to access a table
fails with below NullPointerException.
> Repro is attached - the Spark Example which fills the MemoryStore with broadcasts and
then creates and accesses a SQL Context.
> {code}
> java.lang.NullPointerException
>         at org.apache.spark.sql.SQLConf.getConf(SQLConf.scala:638)
>         at org.apache.spark.sql.SQLConf.defaultDataSourceName(SQLConf.scala:558)
>         at org.apache.spark.sql.DataFrameReader.<init>(DataFrameReader.scala:362)
>         at org.apache.spark.sql.SQLContext.read(SQLContext.scala:623)
>         at SparkHiveExample$.main(SparkHiveExample.scala:76)
>         at SparkHiveExample.main(SparkHiveExample.scala)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
> 18/04/06 14:17:42 ERROR ApplicationMaster: User class threw exception: java.lang.NullPointerException

> java.lang.NullPointerException 
> at org.apache.spark.sql.SQLConf.getConf(SQLConf.scala:638) 
> at org.apache.spark.sql.SQLContext.getConf(SQLContext.scala:153) 
> at org.apache.spark.sql.hive.HiveContext.hiveMetastoreVersion(HiveContext.scala:166)

> at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:258)

> at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:255) 
> at org.apache.spark.sql.hive.HiveContext$$anon$2.<init>(HiveContext.scala:475)

> at org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:475) 
> at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:474) 
> at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:90) 
> at org.apache.spark.sql.SQLContext.table(SQLContext.scala:831) 
> at org.apache.spark.sql.SQLContext.table(SQLContext.scala:827) 
> {code}
> MemoryStore got filled and started dropping the blocks.
> {code}
> 18/04/17 08:03:43 INFO MemoryStore: 2 blocks selected for dropping
> 18/04/17 08:03:43 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated
size 78.1 MB, free 64.4 MB)
> 18/04/17 08:03:43 INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory
(estimated size 1522.0 B, free 64.4 MB)
> 18/04/17 08:03:43 INFO MemoryStore: Block broadcast_15 stored as values in memory (estimated
size 350.9 KB, free 64.1 MB)
> 18/04/17 08:03:43 INFO MemoryStore: Block broadcast_15_piece0 stored as bytes in memory
(estimated size 29.9 KB, free 64.0 MB)
> 18/04/17 08:03:43 INFO MemoryStore: 10 blocks selected for dropping
> 18/04/17 08:03:43 INFO MemoryStore: Block broadcast_16 stored as values in memory (estimated
size 78.1 MB, free 64.7 MB)
> 18/04/17 08:03:43 INFO MemoryStore: Block broadcast_16_piece0 stored as bytes in memory
(estimated size 1522.0 B, free 64.7 MB)
> 18/04/17 08:03:43 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated
size 136.0 B, free 64.7 MB)
> 18/04/17 08:03:44 INFO MemoryStore: MemoryStore cleared
> 18/04/17 08:03:20 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
> 18/04/17 08:03:44 INFO MemoryStore: MemoryStore cleared
> 18/04/17 08:03:57 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
> 18/04/17 08:04:23 INFO MemoryStore: MemoryStore cleared
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message