flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anissa moussaoui <anissa.moussa...@dcbrain.com>
Subject Re: state and Id of job in flatMap
Date Fri, 22 Mar 2019 16:09:44 GMT
Hello,
Thank you for your replay. I will use MapPartition for anomaly detection
for batch job. But i saw that flink has planned to unify stream and batch
according to the folowing link
https://flink.apache.org/news/2019/02/13/unified-batch-streaming-blink.html
, so in this case how can i use MapPartition with stream ?

Thank's in advance !

Anissa



Le mer. 20 mars 2019 à 18:53, Ken Krugler <kkrugler_lists@transpac.com> a
écrit :

> Hi Anissa,
>
> I assume you’re running a batch job, since there is no “end of job” with
> streaming.
>
> If so, then calling .mapPartition(new YourMapPartitionFunction) is one
> option that I’ve used for cases like this.
>
> See MapPartitionFunction
> <https://ci.apache.org/projects/flink/flink-docs-release-1.7/api/java/org/apache/flink/api/common/functions/MapPartitionFunction.html>
in
> the JavaDocs.
>
> — Ken
>
>
> On Mar 20, 2019, at 8:59 AM, anissa moussaoui <
> anissa.moussaoui@dcbrain.com> wrote:
>
> Hello,
>
> I created a process for an anomaly detection with a flatMap. I need to
> know the end of each job at the level of the flatMap to be able to flush a
> buffer in the output collector.
>
> I saw that it is possible to get status of job by using ExecutionEnvironment,
> but i don't know how i can implement it.
>
> Is it possible to recover the end of each job in the flatMap of any
> processing source with last iteration of job or to have in advance the size
> of the partition that each job must deal ?
>
> Thank's in advance !
>
> Anissa
>
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassandra
>
>

-- 

 <http://www.dcbrain.com/>   <https://twitter.com/dcbrain_feed?lang=fr>  

<https://www.linkedin.com/company/dcbrain>   
<https://www.youtube.com/channel/UCSJrWPBLQ58fHPN8lP_SEGw>


 Pensez à la 
planète, imprimer ce papier que si nécessaire 

Mime
View raw message