spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sital Kedia (JIRA)" <>
Subject [jira] [Commented] (SPARK-23502) Support async init of spark context during spark-shell startup
Date Wed, 28 Feb 2018 19:49:00 GMT


Sital Kedia commented on SPARK-23502:

>> what happens when you operate on {{sc}} before it's initialized?

We will wait for the future to complete before triggering any action based on user input,
so that should be fine.

>> Is it surprising if Spark shell starts but errors out 20 seconds later? 

Yes, that might be one of the side effects.  Another major side effect as I mentioned is
not able to print the messages like UI link and app id when the spark-shell starts.


I just wanted to get some opinion to see if this is something useful for the community. If
we do not think so, we can close this.




> Support async init of spark context during spark-shell startup
> --------------------------------------------------------------
>                 Key: SPARK-23502
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Shell
>    Affects Versions: 2.0.0
>            Reporter: Sital Kedia
>            Priority: Minor
> Currently, whenever a user starts the spark shell, we initialize the spark context before
returning the prompt to the user. In environments, where spark context initialization takes
several seconds, it is not a very good user experience for the user to wait for the prompt.
Instead of waiting for the initialization of spark context, we can initialize it in the background
while we return the prompt to the user as soon as possible. Please note that even if we return
the prompt to the user soon, we still need to make sure to wait for the spark context initialization
to complete before any query is executed. 
> Please note that the scala interpreter already does very similar async initialization
in order to return the prompt to the user faster -
We will be emulating the behavior for Spark. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message