beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-549) SparkRunner should support Beam's KafkaIO instead of providing it's own.
Date Thu, 22 Sep 2016 15:37:20 GMT

    [ https://issues.apache.org/jira/browse/BEAM-549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513629#comment-15513629
] 

ASF GitHub Bot commented on BEAM-549:
-------------------------------------

Github user amitsela closed the pull request at:

    https://github.com/apache/incubator-beam/pull/822


> SparkRunner should support Beam's KafkaIO instead of providing it's own.
> ------------------------------------------------------------------------
>
>                 Key: BEAM-549
>                 URL: https://issues.apache.org/jira/browse/BEAM-549
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Amit Sela
>
> For portability, and in the spirit of Apache Beam, the SparkRunner should use the Beam
implementation of KafkaIO instead of it's own.
> Having said that, the runner will translate the KafkaIO as defined in the pipeline into
it's own internal implementation, but should still map the properties the user defined in
the pipeline in a way that the IO behaves the same - i.e., brokers, topic, etc.
> Eventually, the SparkRunner will implement reading from Kafka using Spark's KafakUtils.createDirectStream()
as described here: http://spark.apache.org/docs/1.6.2/streaming-kafka-integration.html#approach-2-direct-approach-no-receivers




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message