mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eren G├╝ven <erenguv...@gmail.com>
Subject resource fragmentation/mesos-master offers too low
Date Tue, 29 Sep 2015 13:31:25 GMT
Hello,

I had an incident where mesos kept offering marathon insufficient resources
which resulted in lots of forever WAITING tasks in marathon. Below are some
example logs I saw. It kept repeating the same insufficient offers even
though the cluster had a lot of free resources like ~80cpus and 45G mem. A
master restart solved the issue but I was wondering what can result in this
situation or if anyone had any pointers on where to dig further.

Thanks!

cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (160.0 > 34.0)
cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.3500000000000014), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 62.0)
cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (160.0 > 62.0)
cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 62.0)
cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 62.0)
cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 62.0)
cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 62.0)
cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 62.0)
cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 62.0)
cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 62.0)
cpu SATISFIED (0.25 <= 1.1500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 62.0)
cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (160.0 > 6.0)
cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.7999999999999998), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (160.0 > 6.0)
cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 0.6500000000000004), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 6.0)
cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT
SATISFIED (128.0 > 42.0)
cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT
SATISFIED (160.0 > 42.0)
cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT
SATISFIED (128.0 > 42.0)
cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT
SATISFIED (128.0 > 42.0)
cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT
SATISFIED (128.0 > 42.0)
cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT
SATISFIED (128.0 > 42.0)
cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT
SATISFIED (128.0 > 42.0)
cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT
SATISFIED (128.0 > 42.0)
cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT
SATISFIED (128.0 > 42.0)
cpu SATISFIED (0.25 <= 1.25), disk SATISFIED (0.0 <= 0.0), mem NOT
SATISFIED (128.0 > 42.0)
cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), mem
NOT SATISFIED (128.0 > 120.0)
cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), mem
NOT SATISFIED (160.0 > 120.0)
cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), mem
NOT SATISFIED (128.0 > 120.0)
cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), mem
NOT SATISFIED (128.0 > 120.0)
cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), mem
NOT SATISFIED (128.0 > 120.0)
cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), mem
NOT SATISFIED (128.0 > 120.0)
cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), mem
NOT SATISFIED (128.0 > 120.0)
cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), mem
NOT SATISFIED (128.0 > 120.0)
cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), mem
NOT SATISFIED (128.0 > 120.0)
cpu SATISFIED (0.25 <= 1.549999999999999), disk SATISFIED (0.0 <= 0.0), mem
NOT SATISFIED (128.0 > 120.0)
cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (160.0 > 34.0)
cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)
cpu SATISFIED (0.25 <= 1.5999999999999996), disk SATISFIED (0.0 <= 0.0),
mem NOT SATISFIED (128.0 > 34.0)

Cheers
Eren

Mime
View raw message