spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tathagata Das (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-4964) Exactly-once + WAL-free Kafka Support in Spark Streaming
Date Tue, 27 Jan 2015 07:53:34 GMT

    [ https://issues.apache.org/jira/browse/SPARK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293129#comment-14293129
] 

Tathagata Das edited comment on SPARK-4964 at 1/27/15 7:53 AM:
---------------------------------------------------------------

I am renaming this JIRA to "Exactly-once + WAL-free Kafka Support in Spark Streaming" because
there are two problems that we are trying to solve, which gets solved by the associated PR.
See the design doc for more details. 

Also, I updated the description to reflect the two issues, and added references to the design
doc.


was (Author: tdas):
I am renaming this JIRA to "Exactly-once + WAL-free Kafka Support in Spark Streaming
" because there are two problems that we are trying to solve, which gets solved by the associated
PR. See the design doc for more details. 

> Exactly-once + WAL-free Kafka Support in Spark Streaming
> --------------------------------------------------------
>
>                 Key: SPARK-4964
>                 URL: https://issues.apache.org/jira/browse/SPARK-4964
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>            Reporter: Cody Koeninger
>
> There are two issues with the current Kafka support 
>  - Use of Write Ahead Logs in Spark Streaming to ensure no data is lost - Causes data
replication in both Kafka AND Spark Streaming. 
>  - Lack of exactly-once semantics - For background, see http://apache-spark-developers-list.1001551.n3.nabble.com/Which-committers-care-about-Kafka-td9827.html
> We want to solve both these problem in JIRA. Please see the following design doc for
the solution. 
> https://docs.google.com/a/databricks.com/document/d/1IuvZhg9cOueTf1mq4qwc1fhPb5FVcaRLcyjrtG4XU1k/edit#heading=h.itproy77j3p



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message