flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: How can I prove ....
Date Fri, 23 Sep 2016 13:32:32 GMT
Are you sure you have the parallelism set to 448? You can see the
parallelism of operators in the web UI.


On Fri, Sep 23, 2016 at 12:15 AM, amir bahmanyari <amirtousa@yahoo.com>
wrote:

> Hi Again & sorry to take your time. But am puzzled by what I cannot
> explain why.
> The parallelism is set to 448. There are 112 tasks per TM.
> Why is Flink NOT allocating ALL 448 slots? It allocates only 1/2 of it.
> I also bumped up the # buffers to equate a 2GiB in each TM & see no
> difference :-(
> So I incremented my total-slots = 448. Kafka topic also has 448 partitions.
> Why am I having such a bad luck with this!!!!!!!??? LOL!!
> Thanks for your attention Aljoscha.
>
> [image: Inline image]
>
>
>
> ------------------------------
> *From:* amir bahmanyari <amirtousa@yahoo.com>
> *To:* Aljoscha Krettek <aljoscha@apache.org>; User <user@flink.apache.org>
>
> *Sent:* Thursday, September 22, 2016 10:10 AM
>
> *Subject:* Re: How can I prove ....
>
> Hi Again, following is from the dashboard while wverything is supposedlt
> running.
> No real-time change in send/received/#of records...but one node is
> definitely producing a *.out file...
> And all TMs are reporting in their *.log files. And the process will
> eventually end , but very slow.
> Thanks again Aljoscha.
> [image: Inline image]
>
>
>
> ------------------------------
> *From:* amir bahmanyari <amirtousa@yahoo.com>
> *To:* Aljoscha Krettek <aljoscha@apache.org>; User <user@flink.apache.org>
>
> *Sent:* Thursday, September 22, 2016 9:16 AM
> *Subject:* Re: How can I prove ....
>
> Thanks Aljoscha,
> Thats why I am wondering about this. I dont see send/receive columns
> change at all....just 0's all the time.
> The only thing that changes is time stamp.
> Is this an indication that the nodes in the cluster are not participating
> in execution of the data?
> Thanks again.
> Amir-
>
>
> ------------------------------
> *From:* Aljoscha Krettek <aljoscha@apache.org>
> *To:* amir bahmanyari <amirtousa@yahoo.com>; User <user@flink.apache.org>
> *Sent:* Thursday, September 22, 2016 5:01 AM
> *Subject:* Re: How can I prove ....
>
> Hi,
> depending on the data source you might not be able to stress CPU/MEM
> because the source might be to slow. As long as you see the numbers
> increasing in the Flink Dashboard for all operators you should be good.
>
> Cheers,
> Aljoscha
>
> On Thu, 22 Sep 2016 at 00:26 amir bahmanyari <amirtousa@yahoo.com> wrote:
>
> That all nodes in a Flink Cluster are involved simultaneously in
> processing the data?
> Programmatically, graphically...
> I need to stress CPU , MEM and all resources to their max.
> How can I guarantee this is happening in Flink Cluster?
> Out of 4 nodes, this is the highest resource usage I see from "top"...
> Everything else is not even close...
> t
> op - 22:22:45 up 41 days,  2:39,  1 user,  load average: 1.76, 1.55, 1.28
> Tasks: 344 total,   1 running, 343 sleeping,   0 stopped,   0 zombie
> %Cpu(s):  5.4 us,  1.0 sy,  0.0 ni, 93.5 id,  0.1 wa,  0.0 hi,  0.0 si,
>  0.0 st
> KiB Mem:  11551564+total, 65702020 used, 49813632 free,   115072 buffers
> KiB Swap:        0 total,        0 used,        0 free.  3148420 cached Mem
>
>
> I am pretty sure I can push FlinkRunner to way more extent than this....
> And thats where true realistic perf numbers start showing up.
> Thanks+regards,
> Amir-
>
>
>
>
>
>
>
>

Mime
View raw message