spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "holdenk (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-16454) Consider adding a per-batch transform for structured streaming
Date Fri, 08 Jul 2016 23:25:11 GMT
holdenk created SPARK-16454:
-------------------------------

             Summary: Consider adding a per-batch transform for structured streaming
                 Key: SPARK-16454
                 URL: https://issues.apache.org/jira/browse/SPARK-16454
             Project: Spark
          Issue Type: Improvement
          Components: SQL, Streaming
            Reporter: holdenk


The new structured streaming API lacks the DStream functionality of transform (which allowed
one to mix in existing RDD transformation logic). It would be useful to be able to do per-batch
(even without any specific gaurantees about the batch being complete provided you eventually
get called with the "catch up" records) processing as was done in the DStream API.

This might be useful for implementing Streaming Machine Learning on Structured Streaming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message