flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Multiple ElasticSearch sinks
Date Wed, 22 Jul 2015 09:07:06 GMT
I don't know anybody who has reported about something like this before on
our lists.
Since you don't know the types before, the mapPartition approach sounds

On Fri, Jul 10, 2015 at 5:02 PM, Flavio Pompermaier <pompermaier@okkam.it>

> Hi to all,
> I have a Flink job that produce json objects that I'd like to index in
> different Elasticsearch indices depending on the "type" attribute of my
> json object (e.g. "people", "places", etc..).
> Is there any previous attempt to do something like that in Flink?
> I was thinking to use the EsHadoopOutputFormat but it requires to specify
> the index name in the job conf..however, in my use case I'll know the
> target indices only once the computation finish so Flink can't know how
> many sinks there will be in the pre-flight phase..
> My solution at the moment was to implement my own mapPartition function
> that instantiate a client to ES and index the json documents in the right
> index at the end of the job pipeline..is there any better approach to it?
> Best,
> Flavio

View raw message