gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karol Brejna (JIRA)" <>
Subject [jira] [Commented] (GEARPUMP-167) Create Elasticsearch sink and example
Date Wed, 10 May 2017 10:04:04 GMT


Karol Brejna commented on GEARPUMP-167:

[~yanghua] I started developing PoC on the following branch:

My idea was to have a simple sink that would just accept a JSON document to be stored in elasticsearch
(not an object that would need to be serialized to JSON).

I anticipated two cases for the incoming message:
* Tuple2[String, String] - id, body - message gets stored under given id, so it's actually
an upsert 
* String - body - message is stored under new id
* A sequence of any of above

I even did some short "demo" that had:
* small app subscribing to Twitter and putting messages to Kafka
* a DAG that consumed messages from Kafka, did wordcount, stored data to elasticsearch
* Kibana visualization

The code was written in Java. I planned to convert it to Scala but got distracted by the last

[~yanghua] I'd be glad to hear your idea on this. Maybe we could cooperate on this feature.

> Create Elasticsearch sink and example
> -------------------------------------
>                 Key: GEARPUMP-167
>                 URL:
>             Project: Apache Gearpump
>          Issue Type: Task
>          Components: connectors
>            Reporter: Karol Brejna
>            Assignee: Karol Brejna
>              Labels: feature
>   Original Estimate: 18h
>  Remaining Estimate: 18h
> Create a simple Elasticsearch sink that enables indexing messages as Elasticsearch docs.
> Write an example demonstrating usage of the sink
> Write a doc on setting up local development env (installing and confifuring elastic,
etc.) for the users of the sink

This message was sent by Atlassian JIRA

View raw message