flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bhaskar.ebay77@gmail.com <bhaskar.eba...@gmail.com>
Subject Re: Question regarding Streaming Resources
Date Thu, 13 Sep 2018 00:41:46 GMT


On 2018/09/12 20:42:22, Ken Krugler <kkrugler_lists@transpac.com> wrote: 
> Hi Bhaskar,
> 
> I assume you don’t have 1000 streams, but rather one (keyed) stream with 1000 different
key values, yes?
> 
> If so, then this one stream is physically partitioned based on the parallelism of the
operator following the keyBy(), not per unique key.
> 
> The most common per-key “resource” is the memory required for each key's state, if
you’ve got any operations that need to maintain state (accumulators, windows, etc).
> 
> For 1000 unique keys, this should be negligible.
> 
> — Ken
> 
> 
> > On Sep 12, 2018, at 9:55 AM, bhaskar.ebay77@gmail.com <mailto:bhaskar.ebay77@gmail.com>
wrote:
> > 
> > Hi
> > 
> > I have created a KeyedStream with state as explained below
> > For example i have created 1000 streams,  out of which 50% of streams data is going
to come once in 8 hours. Will the resources of these under utilized streams are idle for that
duration? Or Flink internal task manager is having some strategy to utilize them for other
new streams that are coming?
> > Regards
> > Bhaskar
> 
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassan
> 
> 
Hi Ken
As per documentation it is showing: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/
On DataStream if we apply KeyBy  then output is KeyedStream. Once its stream means it should
execute in parallel right? There will be 1000 streams each is having Keyed State. What you
are saying is the main over head here is only memory. That means Does these 1000 streams are
going to be run across 1000 task slots in parallel?  These 1000 task slots is the main memory
over head? Even 50% of them idle its not harm?
Please clarify
Regards
Bhaskar
 

Mime
View raw message