openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wangda Tan <wan...@apache.org>
Subject Re: PR to enable actions on YARN
Date Tue, 26 Feb 2019 19:40:48 GMT
We need to break down the time a bit to understand latencies. 

1) First YARN needs to launch application master for every application to request containers.


2) Then application master has its own internal logic to request resources from YARN scheduler.

3) YARN scheduler internally books resources for every container.

4) Once container resource booked by YARN scheduler, AM will get container tokens, and send
to NodeManager to launch container.

5) Node manager launch the container. 

For 1), it typically takes several seconds to launch an application master. (One for every
application). 

For 2), this depends on application's internal logic, mapreduce's AM is different from spark's
AM. Typically it takes several seconds.

For 3), YARN scheduler can do scheduling in batch, typically throughput limit of YARN scheduler
is thousands container allocations per second. 

For 4), AM gets allocated by regular heartbeat from heatbeat, in many cases AM talk to RM
from every 1 second to 10 seconds. 

For 5), NodeManager needs to download requested files, localize docker images, etc. for every
container need to be launched. Assume all requested resource downloaded already, it takes
a few seconds to launch container. 

So for end-to-end container request -> launch delay (excluded application master launch
time). It typically takes 1-10s. If the cluster is super busy, or requested application is
under a queue which doesn't have available resource, it takes longer since scheduler needs
to wait for others release resources.

Hope this helps, 
Wangda

On 2019/02/26 00:15:49, Dascalita Dragos <ddragosd@gmail.com> wrote: 
> Hi Samuel,
> This is an interesting contribution. Do you happen to have any performance
> numbers with YARN ? I'd be particularly interested in the cold start
> latencies.
> 
> Thanks,
> dragos
> 
> On Fri, Feb 22, 2019 at 5:21 PM Samuel Hjelmfelt
> <samhjelmfelt@yahoo.com.invalid> wrote:
> 
> >
> > Hi Rodric and Carlos,
> >
> >
> > ApacheHadoop has three major components: HDFS (distributed filesystem),
> > MapReduce(distributed batch processing engine), YARN (Yet Another Resource
> > Negotiator) (containerengine). While MapReduce has been largely replaced by
> > Apache Tez, Apache Spark,and Apache Flink, HDFS and YARN are still widely
> > used for data analytics use cases.
> >
> >
> >
> > YARN is unique as a container engine because, unlike Mesos and Kubernetes,
> > it was designed for ephemeral, short-livedcontainers rather than for long
> > running micro-services. The jobs and queries that run on YARN are split
> > intosmall tasks that run to completion and generally only last for seconds
> > or maybe minutes. Overthe last couple years, YARN has been expanding its
> > support for long running usecases, but is still focused on data-driven use
> > cases over more generic micro-serviceuse cases (like web apps). The primary
> > long running technologies on YARN are currently Spark Streamingand
> > TensorFlow. Here is an articlefrom LinkedIn about why they created a
> > project for TensorFlow on YARN. Asimilar case could be made for OpenWhisk:
> > https://engineering.linkedin.com/blog/2018/09/open-sourcing-tony--native-support-of-tensorflow-on-hadoop.
> >
> >
> >
> >
> > Bringing OpenWhisk onto YARN makes FaaS more accessible to thethousands of
> > organizations with existing Hadoop clusters. Between Cloudera’s 2,000+
> > customers; Azure, AWS,and GCP cloud customers; and the organizations
> > self-supporting like Netflix, theinstall base of YARN is very high and
> > still growing.
> >
> >
> >
> > ThisPR is a first level of integration, but YARN’s focus on ephemeral
> > containerscould be more fully leveraged by OpenWhisk to improve scalability
> > andperformance. Here is an interesting article on the scalability of YARN
> > fromMicrosoft:
> > https://azure.microsoft.com/en-us/blog/how-microsoft-drives-exabyte-analytics-on-the-world-s-largest-yarn-cluster/
> >
> > Thanks,
> > Sam Hjelmfelt
> >
> 

Mime
View raw message