ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anton Vinogradov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-529) Implement IgniteFlumeStreamer to stream data from Apache Flume
Date Thu, 12 Nov 2015 14:16:10 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15002144#comment-15002144

Anton Vinogradov commented on IGNITE-529:


In "Prepare than put" case DataStreamer should not give more speed.
But can give in "Put while preparing" can. 

My Idea was to do somethilg like this:

public boolean transform(Event event, IgniteDataStreamer<Long, String> dataStreamer){
   for (Something s: Event){
      List<Tuple> datas = s.extract();

      for (Tuple data: datas)
         dataStreamer.addData(data.get1(), data.get2());   

}catch(Exception e){
return false

return true;

This case can give more speed, but I think that usage of putAll is good enough for initial

> Implement IgniteFlumeStreamer to stream data from Apache Flume
> --------------------------------------------------------------
>                 Key: IGNITE-529
>                 URL: https://issues.apache.org/jira/browse/IGNITE-529
>             Project: Ignite
>          Issue Type: Sub-task
>          Components: streaming
>            Reporter: Dmitriy Setrakyan
>            Assignee: Roman Shtykh
> We have {{IgniteDataStreamer}} which is used to load data into Ignite under high load.
It was previously named {{IgniteDataLoader}}, see ticket IGNITE-394.
> See [Apache Flume|http://flume.apache.org/] for more information.
> We should create {{IgniteFlumeStreamer}} which will consume messages from Apache Flume
and stream them into Ignite caches. 
> More details to follow, but to the least we should be able to:
> * Convert Flume data to Ignite data using an optional pluggable converter.
> * Specify the cache name for the Ignite cache to load data into.
> * Specify other flags available on {{IgniteDataStreamer}} class.

This message was sent by Atlassian JIRA

View raw message