spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Resolved] (SPARK-8134) Lifecycle of RDDs in Cluster envs
Date Fri, 05 Jun 2015 22:47:00 GMT


Sean Owen resolved SPARK-8134.
    Resolution: Invalid

As I say, you should email user@. JIRA is not the place for questions.

> Lifecycle of RDDs in Cluster envs
> ---------------------------------
>                 Key: SPARK-8134
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 1.3.1
>            Reporter: sid
> Is there a way to implement simple driver lifecycle like this.
> init() - connect to JMS
> RDD - read a file and pump out messages
> close() - close JMS 
> Currently we are hacking RDDs to create singleton connections (worked node agnostic);
but then unable to close them.
> also when i call SparkContext.close(); it immediately kills jobs in worker nodes.
> There is NO WAY of knowing when to close the context in driver
> I looked into SparkListener and that doesnt help solve our problem.
> I assumed driver would stay alive until jobs are done and cleanup. Driver simply spawns
jobs and ends

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message