spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jose Torres (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-23558) clean up StreamWriter factory lifecycle
Date Thu, 01 Mar 2018 23:40:00 GMT
Jose Torres created SPARK-23558:
-----------------------------------

             Summary: clean up StreamWriter factory lifecycle
                 Key: SPARK-23558
                 URL: https://issues.apache.org/jira/browse/SPARK-23558
             Project: Spark
          Issue Type: Sub-task
          Components: Structured Streaming
    Affects Versions: 2.4.0
            Reporter: Jose Torres


Right now, StreamWriter and children have different lifecycles in continuous processing and
microbatch mode. Both execution modes impose significant constraints on what that lifecycle
must be, so the achievable consistent semantic is:
 * StreamWriter lasts for the duration of the query execution
 * DataWriterFactory lasts for the duration of the query execution
 * DataWriter (the task-level writer) has a lifecycle tied to each individual epoch

This also allows us to restore the implicit semantic that DataWriter.commit()/abort() terminates
the lifecycle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message