spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-3505) Augmenting SparkStreaming updateStateByKey API with timestamp
Date Mon, 15 Sep 2014 23:13:36 GMT

     [ https://issues.apache.org/jira/browse/SPARK-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Wendell updated SPARK-3505:
-----------------------------------
    Fix Version/s:     (was: 1.1.0)
                   1.2.0

> Augmenting SparkStreaming updateStateByKey API with timestamp
> -------------------------------------------------------------
>
>                 Key: SPARK-3505
>                 URL: https://issues.apache.org/jira/browse/SPARK-3505
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 1.0.0
>            Reporter: Xi Liu
>            Priority: Minor
>             Fix For: 1.2.0
>
>
> The current updateStateByKey API in Spark Streaming does not expose timestamp to the
application. 
> In our use case, the application need to know the batch timestamp to decide whether to
keep the state or not. And we do not want to use real system time because we want to decouple
the two (because the same code base is used for streaming and offline processing).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message