mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joris Van Remoortere <jo...@mesosphere.io>
Subject Re: Build failed in Jenkins: Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui #2581
Date Fri, 21 Nov 2014 19:06:40 GMT
Hi guys,

I've been able to repro this more easily with some the libevent based clock
I've been working on.
I have a fix for this deadlock, just need to sync up with BenH to walk
through it. We should have this fixed soon.

@Vinod thanks for the extra information, that's exactly where I was able to
repro :-D

Joris

On Thu, Nov 20, 2014 at 7:14 PM, Jie Yu <yujie.jay@gmail.com> wrote:

> I've seen a similar issue on my dev box.
>
>
> On Thursday, November 20, 2014, Vinod Kone <vinodkone@gmail.com> wrote:
>
>> This particular test got blocked at the very beginning of the test. All it
>> did was start master, start slave, pause clock and wait for http request
>> to
>> /metrics/snapshot to return.
>>
>> That last part does AWAIT_READY() with default duration of 10s. AFAICT,
>> that code (process::internal::await()) should return within 10s even if
>> the
>> clock is paused. So it is strange that it didn't.
>>
>> My suspicion with recent libprocess changes is because 1) i haven't seen
>> this issue before and 2) seems to be related to the guts of
>> libprocess/clock/timers that were recently touched. Of course, it could be
>> totally unrelated to libprocess changes too.
>>
>> On Thu, Nov 20, 2014 at 5:34 PM, Benjamin Hindman <benh@mesosphere.io>
>> wrote:
>>
>> > What led you to that?
>> >
>> > On Thu, Nov 20, 2014 at 4:19 PM, Vinod Kone <vinodkone@gmail.com>
>> wrote:
>> >
>> >> +benh, joris
>> >>
>> >> from the looks of it, i suspect the test is doing a busy loop in
>> >>
>> https://github.com/apache/mesos/blob/master/3rdparty/libprocess/include/process/gtest.hpp#L48
>> >>
>> >> could this be related to recent libprocess changes?
>> >>
>> >> On Thu, Nov 20, 2014 at 4:07 PM, Vinod Kone <vinodkone@gmail.com>
>> wrote:
>> >>
>> >>> too bad there is no gdb trace on this box to understand where this
>> test
>> >>> is stuck :(
>> >>>
>> >>> On Thu, Nov 20, 2014 at 3:55 PM, Apache Jenkins Server <
>> >>> jenkins@builds.apache.org> wrote:
>> >>>
>> >>>> See <
>> >>>>
>> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/2581/changes
>> >>>> >
>> >>>>
>> >>>> Changes:
>> >>>>
>> >>>> [niklas] Changed master label test to strictly enforce format.
>> >>>>
>> >>>> [niklas] Changed slave label test to strictly enforce format.
>> >>>>
>> >>>> [tnachen] Match future dispatch messages with type info.
>> >>>>
>> >>>> [tnachen] Fixed future dispatch in tests to match function type.
>> >>>>
>> >>>> ------------------------------------------
>> >>>> [...truncated 3727 lines...]
>> >>>> make[3]: `no-executor-framework' is up to date.
>> >>>> make[3]: `docker-no-executor-framework' is up to date.
>> >>>> make[3]: `balloon-framework' is up to date.
>> >>>> make[3]: `balloon-executor' is up to date.
>> >>>> make[3]: `load-generator-framework' is up to date.
>> >>>> make[3]: `setns-test-helper' is up to date.
>> >>>> make[3]: `mesos-tests' is up to date.
>> >>>> make[3]: Nothing to be done for
>> >>>> `../../src/tests/balloon_framework_test.sh'.
>> >>>> make[3]: Nothing to be done for
>> >>>> `../../src/tests/low_level_scheduler_libprocess_test.sh'.
>> >>>> make[3]: Nothing to be done for
>> >>>> `../../src/tests/low_level_scheduler_pthread_test.sh'.
>> >>>> make[3]: Nothing to be done for
>> >>>> `../../src/tests/java_exception_test.sh'.
>> >>>> make[3]: Nothing to be done for
>> >>>> `../../src/tests/java_framework_test.sh'.
>> >>>> make[3]: Nothing to be done for `../../src/tests/java_log_test.sh'.
>> >>>> make[3]: Nothing to be done for
>> >>>> `../../src/tests/no_executor_framework_test.sh'.
>> >>>> make[3]: Nothing to be done for
>> >>>> `../../src/tests/python_framework_test.sh'.
>> >>>> make[3]: Nothing to be done for
>> >>>> `../../src/tests/test_framework_test.sh'.
>> >>>> make[3]: Leaving directory `<
>> >>>>
>> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/build/src
>> >>>> '>
>> >>>> make  check-local
>> >>>> make[3]: Entering directory `<
>> >>>>
>> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/build/src
>> >>>> '>
>> >>>> ./mesos-tests
>> >>>> WARNING: Logging before InitGoogleLogging() is written to STDERR
>> >>>> I1120 23:24:12.553694 30655 process.cpp:983] libprocess is
>> initialized
>> >>>> on 67.195.81.186:33912 for 16 cpus
>> >>>> I1120 23:24:12.553962 30655 logging.cpp:177] Logging to STDERR
>> >>>> Source directory: <
>> >>>>
>> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/
>> >>>> >
>> >>>> Build directory: <
>> >>>>
>> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/build
>> >>>> >
>> >>>> -------------------------------------------------------------
>> >>>> We cannot run any cgroups tests that require mounting
>> >>>> hierarchies because you have the following hierarchies mounted:
>> >>>> /sys/fs/cgroup/blkio, /sys/fs/cgroup/cpu, /sys/fs/cgroup/cpuacct,
>> >>>> /sys/fs/cgroup/cpuset, /sys/fs/cgroup/devices,
>> /sys/fs/cgroup/freezer,
>> >>>> /sys/fs/cgroup/hugetlb, /sys/fs/cgroup/memory,
>> /sys/fs/cgroup/perf_event,
>> >>>> /sys/fs/cgroup/systemd
>> >>>> We'll disable the CgroupsNoHierarchyTest test fixture for now.
>> >>>> -------------------------------------------------------------
>> >>>> Note: Google Test filter =
>> >>>>
>> *-DockerContainerizerTest.ROOT_DOCKER_Launch_Executor:DockerContainerizerTest.ROOT_DOCKER_Launch_Executor_Bridged:DockerContainerizerTest.ROOT_DOCKER_Launch:DockerContainerizerTest.ROOT_DOCKER_Kill:DockerContainerizerTest.ROOT_DOCKER_Usage:DockerContainerizerTest.ROOT_DOCKER_Update:DockerContainerizerTest.DISABLED_ROOT_DOCKER_Recover:DockerContainerizerTest.ROOT_DOCKER_Logs:DockerContainerizerTest.ROOT_DOCKER_Default_CMD:DockerContainerizerTest.ROOT_DOCKER_Default_CMD_Override:DockerContainerizerTest.ROOT_DOCKER_Default_CMD_Args:DockerContainerizerTest.ROOT_DOCKER_SlaveRecoveryTaskContainer:DockerContainerizerTest.DISABLED_ROOT_DOCKER_SlaveRecoveryExecutorContainer:DockerContainerizerTest.ROOT_DOCKER_PortMapping:DockerContainerizerTest.ROOT_DOCKER_LaunchSandboxWithColon:DockerContainerizerTest.ROOT_DOCKER_DestroyWhileFetching:DockerContainerizerTest.ROOT_DOCKER_DestroyWhilePulling:DockerTest.ROOT_DOCKER_interface:DockerTest.ROOT_DOCKER_CheckCommandWithShell:DockerTest.ROOT_DOCKER_CheckPortResource:DockerTest.ROOT_DOCKER_CancelPull:CpuIsolatorTest/1.UserCpuUsage:CpuIsolatorTest/1.SystemCpuUsage:LimitedCpuIsolatorTest.ROOT_CGROUPS_Cfs:LimitedCpuIsolatorTest.ROOT_CGROUPS_Cfs_Big_Quota:MemIsolatorTest/0.MemUsage:MemIsolatorTest/1.MemUsage:MemIsolatorTest/2.MemUsage:PerfEventIsolatorTest.ROOT_CGROUPS_Sample:SharedFilesystemIsolatorTest.ROOT_RelativeVolume:SharedFilesystemIsolatorTest.ROOT_AbsoluteVolume:NamespacesPidIsolatorTest.ROOT_PidNamespace:UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup:UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup:UserCgroupIsolatorTest/2.ROOT_CGROUPS_UserCgroup:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PerfRollForward:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PidNamespaceForward:MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PidNamespaceBackward:SlaveTest.ROOT_RunTaskWithCommandInfoWithoutUser:SlaveTest.DISABLED_ROOT_RunTaskWithCommandInfoWithUser:ContainerizerTest.ROOT_CGROUPS_BalloonFramework:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Enabled:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Subsystems:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Mounted:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Get:CgroupsAnyHierarchyTest.ROOT_CGROUPS_NestedCgroups:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Tasks:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Read:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Write:CgroupsAnyHierarchyTest.ROOT_CGROUPS_Cfs_Big_Quota:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Busy:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_SubsystemsHierarchy:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_FindCgroupSubsystems:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_MountedSubsystems:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_CreateRemove:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_Listen:CgroupsAnyHierarchyWithCpuMemoryTest.ROOT_CGROUPS_FreezeNonFreezer:CgroupsNoHierarchyTest.ROOT_CGROUPS_NOHIERARCHY_MountUnmountHierarchy:CgroupsAnyHierarchyWithCpuAcctMemoryTest.ROOT_CGROUPS_Stat:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Freeze:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Kill:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_Destroy:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_AssignThreads:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_DestroyStoppedProcess:CgroupsAnyHierarchyWithFreezerTest.ROOT_CGROUPS_DestroyTracedProcess:CgroupsAnyHierarchyWithPerfEventTest.ROOT_CGROUPS_Perf:NsTest.ROOT_setns:NsTest.ROOT_setnsMultipleThreads:NsTest.ROOT_getns:NsTest.ROOT_destroy:PerfTest.ROOT_Events:PerfTest.ROOT_SampleInit:SlaveCount/Registrar_BENCHMARK_Test.performance/0:SlaveCount/Registrar_BENCHMARK_Test.performance/1:SlaveCount/Registrar_BENCHMARK_Test.performance/2:SlaveCount/Registrar_BENCHMARK_Test.performance/3
>> >>>> Note: Randomizing tests' orders with a seed of 47110 .
>> >>>> [==========] Running 437 tests from 68 test cases.
>> >>>> [----------] Global test environment set-up.
>> >>>> [----------] 4 tests from CRAMMD5Authentication/2, where TypeParam
=
>> >>>>
>> mesos::internal::cram_md5::Authentication<mesos::internal::cram_md5::CRAMMD5Authenticatee,
>> >>>> mesos::internal::tests::Module<mesos::internal::Authenticator,
>> >>>> (mesos::internal::tests::ModuleID)3> >
>> >>>> [ RUN      ] CRAMMD5Authentication/2.failed1
>> >>>> Using temporary directory
>> '/tmp/CRAMMD5Authentication_2_failed1_V0pTyx'
>> >>>> I1120 23:24:12.685442 30676 authenticatee.hpp:114] Initializing
>> client
>> >>>> SASL
>> >>>> I1120 23:24:12.686884 30676 authenticatee.hpp:138] Creating new
>> client
>> >>>> SASL connection
>> >>>> I1120 23:24:12.687741 30678 authenticator.hpp:108] Initializing
>> server
>> >>>> SASL
>> >>>> I1120 23:24:12.687944 30678 auxprop.cpp:63] Initialized in-memory
>> >>>> auxiliary property plugin
>> >>>> I1120 23:24:12.687966 30678 authenticator.hpp:170] Creating new
>> server
>> >>>> SASL connection
>> >>>> I1120 23:24:12.688140 30678 authenticatee.hpp:229] Received SASL
>> >>>> authentication mechanisms: CRAM-MD5
>> >>>> I1120 23:24:12.688180 30678 authenticatee.hpp:255] Attempting to
>> >>>> authenticate with mechanism 'CRAM-MD5'
>> >>>> I1120 23:24:12.688294 30678 authenticator.hpp:276] Received SASL
>> >>>> authentication start
>> >>>> I1120 23:24:12.688361 30678 authenticator.hpp:398] Authentication
>> >>>> requires more steps
>> >>>> I1120 23:24:12.688472 30673 authenticatee.hpp:275] Received SASL
>> >>>> authentication step
>> >>>> I1120 23:24:12.688617 30673 authenticator.hpp:304] Received SASL
>> >>>> authentication step
>> >>>> I1120 23:24:12.688649 30673 auxprop.cpp:99] Request to lookup
>> >>>> properties for user: 'benh' realm: 'penates.apache.org' server
>> FQDN: '
>> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false
>> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
>> >>>> I1120 23:24:12.688663 30673 auxprop.cpp:171] Looking up auxiliary
>> >>>> property '*userPassword'
>> >>>> I1120 23:24:12.688814 30673 auxprop.cpp:171] Looking up auxiliary
>> >>>> property '*cmusaslsecretCRAM-MD5'
>> >>>> I1120 23:24:12.688846 30673 auxprop.cpp:99] Request to lookup
>> >>>> properties for user: 'benh' realm: 'penates.apache.org' server
>> FQDN: '
>> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false
>> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
>> >>>> I1120 23:24:12.688858 30673 auxprop.cpp:121] Skipping auxiliary
>> >>>> property '*userPassword' since SASL_AUXPROP_AUTHZID == true
>> >>>> I1120 23:24:12.688863 30673 auxprop.cpp:121] Skipping auxiliary
>> >>>> property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID ==
true
>> >>>> W1120 23:24:12.688887 30673 authenticator.hpp:404] Authentication
>> >>>> failure: authentication failure
>> >>>> [       OK ] CRAMMD5Authentication/2.failed1 (11 ms)
>> >>>> [ RUN      ] CRAMMD5Authentication/2.AuthenticatorDestructionRace
>> >>>> Using temporary directory
>> >>>> '/tmp/CRAMMD5Authentication_2_AuthenticatorDestructionRace_CRclP9'
>> >>>> I1120 23:24:12.690254 30684 authenticatee.hpp:138] Creating new
>> client
>> >>>> SASL connection
>> >>>> I1120 23:24:12.690830 30676 authenticator.hpp:170] Creating new
>> server
>> >>>> SASL connection
>> >>>> I1120 23:24:12.690989 30682 authenticatee.hpp:229] Received SASL
>> >>>> authentication mechanisms: CRAM-MD5
>> >>>> I1120 23:24:12.691015 30682 authenticatee.hpp:255] Attempting to
>> >>>> authenticate with mechanism 'CRAM-MD5'
>> >>>> I1120 23:24:12.691097 30671 authenticator.hpp:276] Received SASL
>> >>>> authentication start
>> >>>> I1120 23:24:12.691135 30671 authenticator.hpp:398] Authentication
>> >>>> requires more steps
>> >>>> [       OK ] CRAMMD5Authentication/2.AuthenticatorDestructionRace
(2
>> ms)
>> >>>> [ RUN      ] CRAMMD5Authentication/2.success
>> >>>> Using temporary directory
>> '/tmp/CRAMMD5Authentication_2_success_JZF85L'
>> >>>> I1120 23:24:12.692466 30682 authenticatee.hpp:138] Creating new
>> client
>> >>>> SASL connection
>> >>>> I1120 23:24:12.692939 30676 authenticator.hpp:170] Creating new
>> server
>> >>>> SASL connection
>> >>>> I1120 23:24:12.693069 30684 authenticatee.hpp:229] Received SASL
>> >>>> authentication mechanisms: CRAM-MD5
>> >>>> I1120 23:24:12.693094 30684 authenticatee.hpp:255] Attempting to
>> >>>> authenticate with mechanism 'CRAM-MD5'
>> >>>> I1120 23:24:12.693171 30682 authenticator.hpp:276] Received SASL
>> >>>> authentication start
>> >>>> I1120 23:24:12.693207 30682 authenticator.hpp:398] Authentication
>> >>>> requires more steps
>> >>>> I1120 23:24:12.693274 30684 authenticatee.hpp:275] Received SASL
>> >>>> authentication step
>> >>>> I1120 23:24:12.693366 30682 authenticator.hpp:304] Received SASL
>> >>>> authentication step
>> >>>> I1120 23:24:12.693394 30682 auxprop.cpp:99] Request to lookup
>> >>>> properties for user: 'benh' realm: 'penates.apache.org' server
>> FQDN: '
>> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false
>> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
>> >>>> I1120 23:24:12.693408 30682 auxprop.cpp:171] Looking up auxiliary
>> >>>> property '*userPassword'
>> >>>> I1120 23:24:12.693431 30682 auxprop.cpp:171] Looking up auxiliary
>> >>>> property '*cmusaslsecretCRAM-MD5'
>> >>>> I1120 23:24:12.693449 30682 auxprop.cpp:99] Request to lookup
>> >>>> properties for user: 'benh' realm: 'penates.apache.org' server
>> FQDN: '
>> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false
>> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
>> >>>> I1120 23:24:12.693456 30682 auxprop.cpp:121] Skipping auxiliary
>> >>>> property '*userPassword' since SASL_AUXPROP_AUTHZID == true
>> >>>> I1120 23:24:12.693461 30682 auxprop.cpp:121] Skipping auxiliary
>> >>>> property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID ==
true
>> >>>> I1120 23:24:12.693476 30682 authenticator.hpp:390] Authentication
>> >>>> success
>> >>>> I1120 23:24:12.693544 30684 authenticatee.hpp:315] Authentication
>> >>>> success
>> >>>> [       OK ] CRAMMD5Authentication/2.success (3 ms)
>> >>>> [ RUN      ] CRAMMD5Authentication/2.failed2
>> >>>> Using temporary directory
>> '/tmp/CRAMMD5Authentication_2_failed2_yugjno'
>> >>>> I1120 23:24:12.694799 30677 authenticatee.hpp:138] Creating new
>> client
>> >>>> SASL connection
>> >>>> I1120 23:24:12.695314 30676 authenticator.hpp:170] Creating new
>> server
>> >>>> SASL connection
>> >>>> I1120 23:24:12.695435 30674 authenticatee.hpp:229] Received SASL
>> >>>> authentication mechanisms: CRAM-MD5
>> >>>> I1120 23:24:12.695457 30674 authenticatee.hpp:255] Attempting to
>> >>>> authenticate with mechanism 'CRAM-MD5'
>> >>>> I1120 23:24:12.695544 30670 authenticator.hpp:276] Received SASL
>> >>>> authentication start
>> >>>> I1120 23:24:12.695582 30670 authenticator.hpp:398] Authentication
>> >>>> requires more steps
>> >>>> I1120 23:24:12.695657 30674 authenticatee.hpp:275] Received SASL
>> >>>> authentication step
>> >>>> I1120 23:24:12.695745 30674 authenticator.hpp:304] Received SASL
>> >>>> authentication step
>> >>>> I1120 23:24:12.695770 30674 auxprop.cpp:99] Request to lookup
>> >>>> properties for user: 'benh' realm: 'penates.apache.org' server
>> FQDN: '
>> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false
>> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
>> >>>> I1120 23:24:12.695778 30674 auxprop.cpp:171] Looking up auxiliary
>> >>>> property '*userPassword'
>> >>>> I1120 23:24:12.695788 30674 auxprop.cpp:171] Looking up auxiliary
>> >>>> property '*cmusaslsecretCRAM-MD5'
>> >>>> I1120 23:24:12.695797 30674 auxprop.cpp:99] Request to lookup
>> >>>> properties for user: 'benh' realm: 'penates.apache.org' server
>> FQDN: '
>> >>>> penates.apache.org' SASL_AUXPROP_VERIFY_AGAINST_HASH: false
>> >>>> SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
>> >>>> I1120 23:24:12.695803 30674 auxprop.cpp:121] Skipping auxiliary
>> >>>> property '*userPassword' since SASL_AUXPROP_AUTHZID == true
>> >>>> I1120 23:24:12.695809 30674 auxprop.cpp:121] Skipping auxiliary
>> >>>> property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID ==
true
>> >>>> W1120 23:24:12.695818 30674 authenticator.hpp:404] Authentication
>> >>>> failure: user not found
>> >>>> [       OK ] CRAMMD5Authentication/2.failed2 (2 ms)
>> >>>> [----------] 4 tests from CRAMMD5Authentication/2 (18 ms total)
>> >>>>
>> >>>> [----------] 6 tests from RateLimitingTest
>> >>>> [ RUN      ] RateLimitingTest.NoRateLimiting
>> >>>> Using temporary directory
>> '/tmp/RateLimitingTest_NoRateLimiting_XteRE0'
>> >>>> I1120 23:24:12.701375 30655 leveldb.cpp:176] Opened db in 3.4075ms
>> >>>> I1120 23:24:12.703060 30655 leveldb.cpp:183] Compacted db in
>> 1.64536ms
>> >>>> I1120 23:24:12.703178 30655 leveldb.cpp:198] Created db iterator
in
>> >>>> 78140ns
>> >>>> I1120 23:24:12.703205 30655 leveldb.cpp:204] Seeked to beginning
of
>> db
>> >>>> in 3649ns
>> >>>> I1120 23:24:12.703217 30655 leveldb.cpp:273] Iterated through 0
keys
>> in
>> >>>> the db in 400ns
>> >>>> I1120 23:24:12.703368 30655 replica.cpp:741] Replica recovered with
>> log
>> >>>> positions 0 -> 0 with 1 holes and 0 unlearned
>> >>>> I1120 23:24:12.705715 30678 recover.cpp:437] Starting replica
>> recovery
>> >>>> I1120 23:24:12.706840 30678 recover.cpp:463] Replica is in EMPTY
>> status
>> >>>> I1120 23:24:12.711232 30681 replica.cpp:638] Replica in EMPTY status
>> >>>> received a broadcasted recover request
>> >>>> I1120 23:24:12.712901 30683 recover.cpp:188] Received a recover
>> >>>> response from a replica in EMPTY status
>> >>>> I1120 23:24:12.713747 30673 recover.cpp:554] Updating replica status
>> to
>> >>>> STARTING
>> >>>> I1120 23:24:12.714715 30683 leveldb.cpp:306] Persisting metadata
(8
>> >>>> bytes) to leveldb took 708621ns
>> >>>> I1120 23:24:12.714754 30683 replica.cpp:320] Persisted replica status
>> >>>> to STARTING
>> >>>> I1120 23:24:12.715100 30684 recover.cpp:463] Replica is in STARTING
>> >>>> status
>> >>>> I1120 23:24:12.716419 30679 replica.cpp:638] Replica in STARTING
>> status
>> >>>> received a broadcasted recover request
>> >>>> I1120 23:24:12.716961 30676 recover.cpp:188] Received a recover
>> >>>> response from a replica in STARTING status
>> >>>> I1120 23:24:12.717490 30677 recover.cpp:554] Updating replica status
>> to
>> >>>> VOTING
>> >>>> I1120 23:24:12.718242 30680 leveldb.cpp:306] Persisting metadata
(8
>> >>>> bytes) to leveldb took 484920ns
>> >>>> I1120 23:24:12.718276 30680 replica.cpp:320] Persisted replica status
>> >>>> to VOTING
>> >>>> I1120 23:24:12.718385 30672 recover.cpp:568] Successfully joined
the
>> >>>> Paxos group
>> >>>> I1120 23:24:12.718698 30672 recover.cpp:452] Recover process
>> terminated
>> >>>> I1120 23:24:12.723183 30670 master.cpp:318] Master
>> >>>> 20141120-232412-3125920579-33912-30655 (penates.apache.org) started
>> on
>> >>>> 67.195.81.186:33912
>> >>>> I1120 23:24:12.723278 30670 master.cpp:364] Master only allowing
>> >>>> authenticated frameworks to register
>> >>>> I1120 23:24:12.723353 30670 master.cpp:369] Master only allowing
>> >>>> authenticated slaves to register
>> >>>> I1120 23:24:12.723420 30670 credentials.hpp:36] Loading credentials
>> for
>> >>>> authentication from
>> >>>> '/tmp/RateLimitingTest_NoRateLimiting_XteRE0/credentials'
>> >>>> I1120 23:24:12.723983 30670 master.cpp:413] Authorization enabled
>> >>>> I1120 23:24:12.724187 30670 master.cpp:460] Framework rate limiting
>> >>>> enabled
>> >>>> I1120 23:24:12.724884 30678 master.cpp:126] No whitelist given.
>> >>>> Advertising offers for all slaves
>> >>>> I1120 23:24:12.725131 30684 hierarchical_allocator_process.hpp:299]
>> >>>> Initializing hierarchical allocator process with master :
>> >>>> master@67.195.81.186:33912
>> >>>> I1120 23:24:12.728044 30676 master.cpp:1263] The newly elected leader
>> >>>> is master@67.195.81.186:33912 with id
>> >>>> 20141120-232412-3125920579-33912-30655
>> >>>> I1120 23:24:12.728078 30676 master.cpp:1276] Elected as the leading
>> >>>> master!
>> >>>> I1120 23:24:12.728116 30676 master.cpp:1094] Recovering from
>> registrar
>> >>>> I1120 23:24:12.728454 30679 registrar.cpp:313] Recovering registrar
>> >>>> I1120 23:24:12.730170 30677 log.cpp:656] Attempting to start the
>> writer
>> >>>> I1120 23:24:12.733129 30676 replica.cpp:474] Replica received
>> implicit
>> >>>> promise request with proposal 1
>> >>>> I1120 23:24:12.733763 30676 leveldb.cpp:306] Persisting metadata
(8
>> >>>> bytes) to leveldb took 598467ns
>> >>>> I1120 23:24:12.733795 30676 replica.cpp:342] Persisted promised
to 1
>> >>>> I1120 23:24:12.735090 30680 coordinator.cpp:230] Coordinator
>> attemping
>> >>>> to fill missing position
>> >>>> I1120 23:24:12.737241 30677 replica.cpp:375] Replica received
>> explicit
>> >>>> promise request for position 0 with proposal 2
>> >>>> I1120 23:24:12.737690 30677 leveldb.cpp:343] Persisting action (8
>> >>>> bytes) to leveldb took 402621ns
>> >>>> I1120 23:24:12.737721 30677 replica.cpp:676] Persisted action at
0
>> >>>> I1120 23:24:12.739233 30681 replica.cpp:508] Replica received write
>> >>>> request for position 0
>> >>>> I1120 23:24:12.739351 30681 leveldb.cpp:438] Reading position from
>> >>>> leveldb took 79235ns
>> >>>> I1120 23:24:12.739878 30681 leveldb.cpp:343] Persisting action (14
>> >>>> bytes) to leveldb took 448572ns
>> >>>> I1120 23:24:12.739907 30681 replica.cpp:676] Persisted action at
0
>> >>>> I1120 23:24:12.740696 30679 replica.cpp:655] Replica received learned
>> >>>> notice for position 0
>> >>>> I1120 23:24:12.741052 30679 leveldb.cpp:343] Persisting action (16
>> >>>> bytes) to leveldb took 322694ns
>> >>>> I1120 23:24:12.741083 30679 replica.cpp:676] Persisted action at
0
>> >>>> I1120 23:24:12.741117 30679 replica.cpp:661] Replica learned NOP
>> action
>> >>>> at position 0
>> >>>> I1120 23:24:12.741811 30672 log.cpp:672] Writer started with ending
>> >>>> position 0
>> >>>> I1120 23:24:12.744951 30670 leveldb.cpp:438] Reading position from
>> >>>> leveldb took 30629ns
>> >>>> I1120 23:24:12.748874 30678 registrar.cpp:346] Successfully fetched
>> the
>> >>>> registry (0B) in 20.35584ms
>> >>>> I1120 23:24:12.749171 30678 registrar.cpp:445] Applied 1 operations
>> in
>> >>>> 87917ns; attempting to update the 'registry'
>> >>>> I1120 23:24:12.752719 30672 log.cpp:680] Attempting to append 140
>> bytes
>> >>>> to the log
>> >>>> I1120 23:24:12.752936 30671 coordinator.cpp:340] Coordinator
>> attempting
>> >>>> to write APPEND action at position 1
>> >>>> I1120 23:24:12.753829 30671 replica.cpp:508] Replica received write
>> >>>> request for position 1
>> >>>> I1120 23:24:12.754359 30671 leveldb.cpp:343] Persisting action (159
>> >>>> bytes) to leveldb took 494376ns
>> >>>> I1120 23:24:12.754387 30671 replica.cpp:676] Persisted action at
1
>> >>>> I1120 23:24:12.755213 30677 replica.cpp:655] Replica received learned
>> >>>> notice for position 1
>> >>>> I1120 23:24:12.755621 30677 leveldb.cpp:343] Persisting action (161
>> >>>> bytes) to leveldb took 375152ns
>> >>>> I1120 23:24:12.755648 30677 replica.cpp:676] Persisted action at
1
>> >>>> I1120 23:24:12.755669 30677 replica.cpp:661] Replica learned APPEND
>> >>>> action at position 1
>> >>>> I1120 23:24:12.757231 30673 registrar.cpp:490] Successfully updated
>> the
>> >>>> 'registry' in 7.922944ms
>> >>>> I1120 23:24:12.757463 30673 registrar.cpp:376] Successfully recovered
>> >>>> registrar
>> >>>> I1120 23:24:12.757665 30682 log.cpp:699] Attempting to truncate
the
>> log
>> >>>> to 1
>> >>>> I1120 23:24:12.757859 30679 coordinator.cpp:340] Coordinator
>> attempting
>> >>>> to write TRUNCATE action at position 2
>> >>>> I1120 23:24:12.757899 30681 master.cpp:1121] Recovered 0 slaves
from
>> >>>> the Registry (102B) ; allowing 10mins for slaves to re-register
>> >>>> I1120 23:24:12.758951 30680 replica.cpp:508] Replica received write
>> >>>> request for position 2
>> >>>> I1120 23:24:12.759395 30680 leveldb.cpp:343] Persisting action (16
>> >>>> bytes) to leveldb took 408946ns
>> >>>> I1120 23:24:12.759429 30680 replica.cpp:676] Persisted action at
2
>> >>>> I1120 23:24:12.760109 30675 replica.cpp:655] Replica received learned
>> >>>> notice for position 2
>> >>>> I1120 23:24:12.760496 30675 leveldb.cpp:343] Persisting action (18
>> >>>> bytes) to leveldb took 356913ns
>> >>>> I1120 23:24:12.760584 30675 leveldb.cpp:401] Deleting ~1 keys from
>> >>>> leveldb took 45969ns
>> >>>> I1120 23:24:12.760608 30675 replica.cpp:676] Persisted action at
2
>> >>>> I1120 23:24:12.760633 30675 replica.cpp:661] Replica learned TRUNCATE
>> >>>> action at position 2
>> >>>> I1120 23:24:12.779798 30680 process.cpp:2881] Handling HTTP event
for
>> >>>> process 'metrics' with path: '/metrics/snapshot'
>> >>>> Thu Nov 20 23:54:06 UTC 2014: process still running after 1800
>> seconds
>> >>>> ==========
>> >>>>
>> >>>> Attaching gdb to 30437 bash ../support/timed_tests.sh GLOG_v=1
>> >>>> MESOS_VERBOSE=1 make check GTEST_SHUFFLE=1 GTEST_OUTPUT=xml: 1800
>> >>>>
>> >>>> ==========
>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found
>> >>>> ==========
>> >>>>
>> >>>> Attaching gdb to 30440 bash ../support/timed_tests.sh GLOG_v=1
>> >>>> MESOS_VERBOSE=1 make check GTEST_SHUFFLE=1 GTEST_OUTPUT=xml: 1800
>> >>>>
>> >>>> ==========
>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found
>> >>>> ==========
>> >>>>
>> >>>> Attaching gdb to 30442 make check GTEST_SHUFFLE=1 GTEST_OUTPUT=xml:
>> >>>>
>> >>>> ==========
>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found
>> >>>> ==========
>> >>>>
>> >>>> Attaching gdb to 30446 /bin/bash -c fail=; \ if (target_option=k;
>> case
>> >>>> ${target_option-} in ?) ;; *) echo "am__make_running_with_option:
>> internal
>> >>>> error: invalid" "target option '${target_option-}' specified" >&2;
>> exit 1;;
>> >>>> esac; has_opt=no; sane_makeflags=$MAKEFLAGS; if test -n ' Makefile'
>> && test
>> >>>> -n '0'; then sane_makeflags=$MFLAGS; else case $MAKEFLAGS in *\\[\
>> \.]*)
>> >>>> bs=\\; sane_makeflags=`printf '%s\n' "$MAKEFLAGS" | sed "s/$bs$bs[$bs
>> >>>> $bs.]*//g"`;; esac; fi; skip_next=no; strip_trailopt () { flg=`printf
>> >>>> '%s\n' "$flg" | sed "s/$1.*$//"`; }; for flg in $sane_makeflags;
do
>> test
>> >>>> $skip_next = yes && { skip_next=no; continue; }; case $flg
in
>> *=*|--*)
>> >>>> continue;; -*I) strip_trailopt 'I'; skip_next=yes;; -*I?*)
>> strip_trailopt
>> >>>> 'I';; -*O) strip_trailopt 'O'; skip_next=yes;; -*O?*) strip_trailopt
>> 'O';;
>> >>>> -*l) strip_trailopt 'l'; skip_next=yes;; -*l?*) strip_trailopt 'l';;
>> >>>> -[dEDm]) skip_next=yes;; -[JT]) skip_next=yes;; esac; case $flg
in
>> >>>> *$target_option*) has_opt=yes; break;; esac; done; test $has_opt
=
>> yes);
>> >>>> then \   failcom='fail=yes'; \ else \   failcom='exit 1'; \ fi;
\
>> >>>> dot_seen=no; \ target=`echo check-recursive | sed s/-recursive//`;
\
>> case
>> >>>> "check-recursive" in \   distclean-* | maintainer-clean-*) list='.
>> 3rdparty
>> >>>> src ec2' ;; \   *) list='. 3rdparty src ec2' ;; \ esac; \ for subdir
>> in
>> >>>> $list; do \   echo "Making $target in $subdir"; \   if test
>> "$subdir" =
>> >>>> "."; then \     dot_seen=yes; \     local_target="$target-am"; \
>>  else \
>> >>>>    local_target="$target"; \   fi; \   (CDPATH="${ZSH_VERSION+.}:"
>> && cd
>> >>>> $subdir && make  $local_target) \   || eval $failcom; \
done; \ if
>> test
>> >>>> "$dot_seen" = "no"; then \   make  "$target-am" || exit 1; \ fi;
>> test -z
>> >>>> "$fail"
>> >>>>
>> >>>> ==========
>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found
>> >>>> ==========
>> >>>>
>> >>>> Attaching gdb to 30642 /bin/bash -c fail=; \ if (target_option=k;
>> case
>> >>>> ${target_option-} in ?) ;; *) echo "am__make_running_with_option:
>> internal
>> >>>> error: invalid" "target option '${target_option-}' specified" >&2;
>> exit 1;;
>> >>>> esac; has_opt=no; sane_makeflags=$MAKEFLAGS; if test -n ' Makefile'
>> && test
>> >>>> -n '0'; then sane_makeflags=$MFLAGS; else case $MAKEFLAGS in *\\[\
>> \.]*)
>> >>>> bs=\\; sane_makeflags=`printf '%s\n' "$MAKEFLAGS" | sed "s/$bs$bs[$bs
>> >>>> $bs.]*//g"`;; esac; fi; skip_next=no; strip_trailopt () { flg=`printf
>> >>>> '%s\n' "$flg" | sed "s/$1.*$//"`; }; for flg in $sane_makeflags;
do
>> test
>> >>>> $skip_next = yes && { skip_next=no; continue; }; case $flg
in
>> *=*|--*)
>> >>>> continue;; -*I) strip_trailopt 'I'; skip_next=yes;; -*I?*)
>> strip_trailopt
>> >>>> 'I';; -*O) strip_trailopt 'O'; skip_next=yes;; -*O?*) strip_trailopt
>> 'O';;
>> >>>> -*l) strip_trailopt 'l'; skip_next=yes;; -*l?*) strip_trailopt 'l';;
>> >>>> -[dEDm]) skip_next=yes;; -[JT]) skip_next=yes;; esac; case $flg
in
>> >>>> *$target_option*) has_opt=yes; break;; esac; done; test $has_opt
=
>> yes);
>> >>>> then \   failcom='fail=yes'; \ else \   failcom='exit 1'; \ fi;
\
>> >>>> dot_seen=no; \ target=`echo check-recursive | sed s/-recursive//`;
\
>> case
>> >>>> "check-recursive" in \   distclean-* | maintainer-clean-*) list='.
>> 3rdparty
>> >>>> src ec2' ;; \   *) list='. 3rdparty src ec2' ;; \ esac; \ for subdir
>> in
>> >>>> $list; do \   echo "Making $target in $subdir"; \   if test
>> "$subdir" =
>> >>>> "."; then \     dot_seen=yes; \     local_target="$target-am"; \
>>  else \
>> >>>>    local_target="$target"; \   fi; \   (CDPATH="${ZSH_VERSION+.}:"
>> && cd
>> >>>> $subdir && make  $local_target) \   || eval $failcom; \
done; \ if
>> test
>> >>>> "$dot_seen" = "no"; then \   make  "$target-am" || exit 1; \ fi;
>> test -z
>> >>>> "$fail"
>> >>>>
>> >>>> ==========
>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found
>> >>>> ==========
>> >>>>
>> >>>> Attaching gdb to 30643 make check
>> >>>>
>> >>>> ==========
>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found
>> >>>> ==========
>> >>>>
>> >>>> Attaching gdb to 30645 make check-am
>> >>>>
>> >>>> ==========
>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found
>> >>>> ==========
>> >>>>
>> >>>> Attaching gdb to 30652 make check-local
>> >>>>
>> >>>> ==========
>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found
>> >>>> ==========
>> >>>>
>> >>>> Attaching gdb to 30655 <
>> >>>>
>> https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-Out-Of-Src-Disable-Java-Disable-Python-Disable-Webui/ws/build/src/.libs/lt-mesos-tests
>> >>>> >
>> >>>>
>> >>>> ==========
>> >>>> ../support/timed_tests.sh: line 72: gdb: command not found
>> >>>> Test failed and killing the stuck test process
>> >>>> W1120 23:24:12.779798 30669 logging.cpp:81] RAW: Received signal
>> >>>> SIGTERM from process 30652 of user 2396; exiting
>> >>>> W1120 23:24:12.779798 30655 logging.cpp:81] RAW: Received signal
>> >>>> SIGTERM from process 20380 of user 2396; exiting
>> >>>> make: *** [check-recursive] Terminated
>> >>>> /tmp/hudson9138584248396286316.sh: line 16: 30437 Terminated
>> >>>>   ../support/timed_tests.sh "GLOG_v=1 MESOS_VERBOSE=1 make check
>> >>>> GTEST_SHUFFLE=1 GTEST_OUTPUT=xml:" 1800
>> >>>> make[3]: *** [check-local] Terminated
>> >>>> make[2]: *** [check-am] Terminated
>> >>>> make[1]: *** [check] Terminated
>> >>>> Build step 'Execute shell' marked build as failure
>> >>>> Recording test results
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message