mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Hindman (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MESOS-547) MasterTest.MasterLost is flaky on Jenkins
Date Thu, 05 Jun 2014 19:27:10 GMT

     [ https://issues.apache.org/jira/browse/MESOS-547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Benjamin Hindman updated MESOS-547:
-----------------------------------

    Sprint:   (was: Q2)

> MasterTest.MasterLost is flaky on Jenkins
> -----------------------------------------
>
>                 Key: MESOS-547
>                 URL: https://issues.apache.org/jira/browse/MESOS-547
>             Project: Mesos
>          Issue Type: Bug
>            Reporter: Benjamin Mahler
>              Labels: twitter
>
> This is because the MasterLost test expects the Scheduler.registered call to happen a
single time. However, on the Apache Jenkins machine, it runs slow enough for the reliable
registration in the scheduler driver to retry registration (1 second timeout).
> The fix here will affect a lot of tests, so I'd like to do the right thing, possible
fixes:
>   EXPECT_CALL(sched, registered(&driver, _, _))
>     .Times(1); // Change this to .WillRepeatedly(Return())
> Also, pausing the clock prior to starting the driver will work. But may have further
impact on the tests.
> [ RUN      ] MasterTest.MasterLost
> I0712 03:48:50.415976  4397 master.cpp:230] Master started on 67.195.138.9:47248
> I0712 03:48:50.416594  4399 slave.cpp:113] Slave started on 11)@67.195.138.9:47248
> I0712 03:48:50.417034  4395 sched.cpp:177] New master at master@67.195.138.9:47248
> I0712 03:48:50.423070  4397 master.cpp:245] Master ID: 201307120348-160088899-47248-4046
> I0712 03:48:50.431115  4399 slave.cpp:205] Slave resources: cpus=2; mem=1024; ports=[31000-32000];
disk=1024
> I0712 03:48:50.508512  4398 hierarchical_allocator_process.hpp:305] Initializing hierarchical
allocator process with master : master@67.195.138.9:47248
> W0712 03:48:50.508554  4394 master.cpp:83] No whitelist given. Advertising offers for
all slaves
> I0712 03:48:50.508767  4397 master.cpp:583] Elected as master!
> I0712 03:48:50.791844  4399 slave.cpp:529] New master detected at master@67.195.138.9:47248
> I0712 03:48:51.564163  4399 slave.cpp:544] Postponing registration until recovery is
complete
> I0712 03:48:51.564257  4396 status_update_manager.cpp:155] New master detected at master@67.195.138.9:47248
> I0712 03:48:51.564288  4397 master.cpp:633] Registering framework 201307120348-160088899-47248-4046-0000
at scheduler(11)@67.195.138.9:47248
> I0712 03:48:51.751411  4399 slave.cpp:390] Finished recovery
> I0712 03:48:51.823364  4392 sched.cpp:222] Framework registered with 201307120348-160088899-47248-4046-0000
> I0712 03:48:51.823395  4397 master.cpp:620] Framework 201307120348-160088899-47248-4046-0000
(scheduler(11)@67.195.138.9:47248) already registered, resending acknowledgement
> I0712 03:48:51.823452  4395 hierarchical_allocator_process.hpp:335] Added framework 201307120348-160088899-47248-4046-0000
> I0712 03:48:52.110093  4395 hierarchical_allocator_process.hpp:704] No resources available
to allocate!
> I0712 03:48:51.946835  4392 sched.cpp:194] No master detected, waiting for another master
> I0712 03:48:52.110155  4397 master.cpp:962] Attempting to register slave on vesta.apache.org
at slave(11)@67.195.138.9:47248
> I0712 03:48:52.440776  4397 master.cpp:1929] Adding slave 201307120348-160088899-47248-4046-0
at vesta.apache.org with cpus=2; mem=1024; ports=[31000-32000]; disk=1024
> I0712 03:48:52.249390  4392 sched.cpp:222] Framework registered with 201307120348-160088899-47248-4046-0000
> ../../src/tests/master_tests.cpp:906: Failure
> Mock function called more times than expected - returning directly.
>     Function call: registered(0x7fffaf90ca00, @0x2b7bfc004140 201307120348-160088899-47248-4046-0000,
@0x2b7bfc003170 40-byte object <70-8B 2C-F5 7B-2B 00-00 00-00 00-00 00-00 00-00 F0-54 00-FC
7B-2B 00-00 43-C3 8A-09 90-B8 00-00 00-00 00-00 07-00 00-00>)
>          Expected: to be called once
>            Actual: called twice - over-saturated and active
> I0712 03:48:52.241426  4395 hierarchical_allocator_process.hpp:666] Performed allocation
for 0 slaves in 131.314878ms
> I0712 03:48:52.448761  4399 slave.cpp:589] Registered with master master@67.195.138.9:47248;
given slave ID 201307120348-160088899-47248-4046-0
> I0712 03:48:52.448806  4397 master.cpp:436] Master terminating
> I0712 03:48:52.481109  4397 slave.cpp:485] Slave asked to shut down by master@67.195.138.9:47248
> I0712 03:48:52.489028  4397 slave.cpp:440] Slave terminating
> I0712 03:48:52.456776  4392 sched.cpp:427] Stopping framework '201307120348-160088899-47248-4046-0000'
> I0712 03:48:52.481195  4046 master.cpp:209] Shutting down master
> I0712 03:48:52.464653  4395 hierarchical_allocator_process.hpp:448] Added slave 201307120348-160088899-47248-4046-0
(vesta.apache.org) with cpus=2; mem=1024; ports=[31000-32000]; disk=1024 (and cpus=2; mem=1024;
ports=[31000-32000]; disk=1024 available)
> I0712 03:48:52.560148  4395 hierarchical_allocator_process.hpp:726] Offering cpus=2;
mem=1024; ports=[31000-32000]; disk=1024 on slave 201307120348-160088899-47248-4046-0 to framework
201307120348-160088899-47248-4046-0000
> I0712 03:48:52.568372  4395 hierarchical_allocator_process.hpp:686] Performed allocation
for slave 201307120348-160088899-47248-4046-0 in 8.255916ms
> [  FAILED  ] MasterTest.MasterLost (2161 ms)
> [----------] 11 tests from MasterTest (15502 ms total)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message