mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philip Weaver <>
Subject High latency when scheduling and executing many tiny tasks.
Date Fri, 17 Jul 2015 19:26:50 GMT
I'm trying to understand the behavior of mesos, and if what I am observing
is typical or if I'm doing something wrong, and what options I have for
improving the performance of how offers are made and how tasks are executed
for my particular use case.

I have written a Scheduler that has a queue of very small tasks (for
testing, they are "echo hello world", but in production many of them won't
be much more expensive than that). Each task is configured to use 1 cpu
resource. When resourceOffers is called, I launch as many tasks as I can in
the given offers; that is, one call to driver.launchTasks for each offer,
with a list of tasks that has one task for each cpu in that offer.

On a cluster of 3 nodes and 4 cores each (12 total cores), it takes 120s to
execute 1000 tasks out of the queue. We are evaluting mesos because we want
to use it to replace our current homegrown cluster controller, which can
execute 1000 tasks in way less than 120s.

I am seeing two things that concern me:

   - The time between driver.launchTasks and receiving a callback to
   statusUpdate when the task completes is typically 200-500ms, and sometimes
   even as high as 1000-2000ms.
   - The time between when a task completes and when I get an offer for the
   newly freed resource is another 500ms or so.

These latencies explain why I can only execute tasks at a rate of about 8/s.

It looks like my offers always include all 4 cores on each machine, which
would indicate that mesos doesn't like to send an offer as soon as a single
resource is avaiable, and prefers to delay and send an offer with more
resources in it. Is this true?

Thanks in advance for any advice you can offer!

- Phllip

View raw message