ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Denis Magda (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-2016) Update KafkaStreamer to fit new features introduced in Kafka 0.9
Date Fri, 25 Dec 2015 10:46:49 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15071500#comment-15071500
] 

Denis Magda commented on IGNITE-2016:
-------------------------------------

Roman,

According to {{SinkTask.put(...)}} API
http://docs.confluent.io/2.0.0/connect/javadocs/org/apache/kafka/connect/sink/SinkTask.html#put(java.util.Collection)
There is no need to buffer the data waiting for {{SinkTask.flush()}} to be called. It's ok
and in my opinion preferable to start sending the data asynchronously.

I got your idea and concerns regarding the case when something going wrong during the streaming
and we want to let Kafka know to continue streaming from some offset. However to overcome
this issue we still can rely on {{IgniteDataStreamer}}. Just call {{IgniteDataStreamer.flush()}}
in the implementation of your sink's flush method. This will ensure that the streamer has
already flushed all the data properly at the time when implementation of your {{flush}} method
returns.

Makes sense?

> Update KafkaStreamer to fit new features introduced in Kafka 0.9
> ----------------------------------------------------------------
>
>                 Key: IGNITE-2016
>                 URL: https://issues.apache.org/jira/browse/IGNITE-2016
>             Project: Ignite
>          Issue Type: New Feature
>          Components: streaming
>            Reporter: Roman Shtykh
>            Assignee: Roman Shtykh
>
> Particularly,
> - new consumer
> - Kafka Connect (Copycat)
> http://www.confluent.io/blog/apache-kafka-0.9-is-released
> This can be a a different integration task or a complete re-write of the current implementation,
considering the fact that Kafka Connect is a new standard way for "large-scale, real-time
data import and export for Kafka."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message