mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kone (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MESOS-7601) Some container launch failures are mistakenly treated as errors.
Date Fri, 21 Jul 2017 22:52:01 GMT

     [ https://issues.apache.org/jira/browse/MESOS-7601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vinod Kone updated MESOS-7601:
------------------------------
    Sprint: Mesosphere Sprint 59, Mesosphere Sprint 60  (was: Mesosphere Sprint 59)

> Some container launch failures are mistakenly treated as errors.
> ----------------------------------------------------------------
>
>                 Key: MESOS-7601
>                 URL: https://issues.apache.org/jira/browse/MESOS-7601
>             Project: Mesos
>          Issue Type: Bug
>          Components: containerization
>    Affects Versions: 1.3.0
>            Reporter: Alexander Rukletsov
>            Assignee: Alexander Rukletsov
>              Labels: containerizer, mesosphere, tech-debt
>
> I've observed a case when a scheduler stops (i.e. calls TEARDOWN) while some of its tasks
are being launched. While this is a valid behaviour, the agent prints an error and increased
container launch errors metrics.
> Below are log excerpts for such framework, {{6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092}}.
> *Master log*
> {noformat}
> [centos@ip-172-31-6-200 ~]$ journalctl _PID=29716 --since "2 hours ago" --no-pager |
grep "6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092"
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:32:58.226218 29724 master.cpp:6072] Updating info for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:32:58.226356 29728 hierarchical.cpp:274] Added framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:32:58.226405 29728 hierarchical.cpp:379] Deactivated framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:32:58.228570 29728 hierarchical.cpp:343] Activated framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:32:58.246068 29721 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:32:58.247851 29721 master.cpp:7194] Sending 1 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:32:58 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:32:58.912937 29728 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509464
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:32:59 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:32:59.804184 29727 master.cpp:7105] Sending 2 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:32:59 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:32:59.804411 29727 master.cpp:7194] Sending 2 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:01.248924 29721 master.cpp:7105] Sending 2 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:01.249289 29721 master.cpp:7194] Sending 2 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:01.249724 29721 master.cpp:3851] Processing ACCEPT call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509469
] on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122)
for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:01.250141 29721 master.cpp:3851] Processing ACCEPT call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509470
] on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 at slave(1)@172.31.7.202:5051 (172.31.7.202)
for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:01.252516 29721 master.cpp:4501] Launching task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 with resources
cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35
at slave(1)@172.31.13.122:5051 (172.31.13.122)
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:01.254794 29721 master.cpp:4501] Launching task 1 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 with resources
cpus(*)(allocated: *):1; mem(*)(allocated: *):1408 on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2
at slave(1)@172.31.7.202:5051 (172.31.7.202)
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:01.255506 29725 hierarchical.cpp:855] Updated allocation of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 from ports(*)(allocated: *):[1025-2180,
2182-3887, 3889-5049, 5052-6999, 7002-7076, 7078-7079, 7082-7198, 7200-8079, 8082-8180, 8183-8608,
8610-8982, 8984-9002, 9006-9041, 9043-9159, 9161-10012, 10016-10666, 10669-11682, 11685-11872,
11874-21620, 21622-30643, 30646-32000]; cpus(*)(allocated: *):1.1; mem(*)(allocated: *):6456;
disk(*)(allocated: *):46032; disk(*)(allocated: *)[MOUNT:/dcos/volume3]:47540; disk(*)(allocated:
*)[MOUNT:/dcos/volume4]:47540 to ports(*)(allocated: *):[1025-2180, 2182-3887, 3889-5049,
5052-6999, 7002-7076, 7078-7079, 7082-7198, 7200-8079, 8082-8180, 8183-8608, 8610-8982, 8984-9002,
9006-9041, 9043-9159, 9161-10012, 10016-10666, 10669-11682, 11685-11872, 11874-21620, 21622-30643,
30646-32000]; cpus(*)(allocated: *):1.1; mem(*)(allocated: *):6456; disk(*)(allocated: *):46032;
disk(*)(allocated: *)[MOUNT:/dcos/volume3]:47540; disk(*)(allocated: *)[MOUNT:/dcos/volume4]:47540
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:01.258015 29725 hierarchical.cpp:855] Updated allocation of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 from ports(*)(allocated: *):[1027-2180, 2182-3887,
3889-5049, 5052-6875, 6877-8079, 8082-8180, 8182-9299, 9301-9543, 9545-14041, 14043-15028,
15030-24844, 24846-32000]; disk(*)(allocated: *)[MOUNT:/dcos/volume0]:51042; disk(*)(allocated:
*):43530; cpus(*)(allocated: *):1.08; mem(*)(allocated: *):9357 to ports(*)(allocated: *):[1027-2180,
2182-3887, 3889-5049, 5052-6875, 6877-8079, 8082-8180, 8182-9299, 9301-9543, 9545-14041, 14043-15028,
15030-24844, 24846-32000]; disk(*)(allocated: *)[MOUNT:/dcos/volume0]:51042; disk(*)(allocated:
*):43530; cpus(*)(allocated: *):1.08; mem(*)(allocated: *):9357
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:01.322147 29724 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509473
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:01 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:01.322619 29724 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509474
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:03.113775 29722 master.cpp:6269] Status update TASK_RUNNING (UUID: 646de179-526f-48e4-8fe9-4deda3a09179)
for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
from agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122)
> Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:03.113813 29722 master.cpp:6337] Forwarding status update TASK_RUNNING (UUID: 646de179-526f-48e4-8fe9-4deda3a09179)
for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:03.117269 29722 master.cpp:8368] Updating the state of task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(latest state: TASK_RUNNING, status update state: TASK_RUNNING)
> Jun 01 11:33:03 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:03.216639 29723 master.cpp:5163] Processing ACKNOWLEDGE call 646de179-526f-48e4-8fe9-4deda3a09179
for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 on agent
36a25adb-4ea2-49d3-a195-448cff1dc146-S35
> Jun 01 11:33:05 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:05.410168 29722 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:05 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:05.410367 29722 master.cpp:7194] Sending 1 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:05 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:05.413863 29722 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509489
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:06 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:06.643015 29723 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:06 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:06.645283 29722 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509492
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:08 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:08.385871 29728 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:08 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:08.388234 29726 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509495
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:11 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:11.465273 29723 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:11 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:11.467978 29725 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509499
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:12 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:12.481941 29726 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:12 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:12.484498 29721 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509500
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:16.552039 29724 master.cpp:6269] Status update TASK_FINISHED (UUID: f49ba849-90cc-4110-b897-0d5d16a17588)
for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
from agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122)
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:16.552119 29724 master.cpp:6337] Forwarding status update TASK_FINISHED (UUID: f49ba849-90cc-4110-b897-0d5d16a17588)
for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:16.553474 29724 master.cpp:8368] Updating the state of task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(latest state: TASK_FINISHED, status update state: TASK_FINISHED)
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:16.556002 29724 master.cpp:5163] Processing ACKNOWLEDGE call f49ba849-90cc-4110-b897-0d5d16a17588
for task 0 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531 on agent
36a25adb-4ea2-49d3-a195-448cff1dc146-S35
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:16.556046 29724 master.cpp:8462] Removing task 0 with resources cpus(*)(allocated: *):1;
mem(*)(allocated: *):1408 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S35 at slave(1)@172.31.13.122:5051 (172.31.13.122)
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:16.556318 29727 master.cpp:4911] Processing REVIVE call for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:16.556380 29727 hierarchical.cpp:1260] Revived offers for roles { * } of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:16.607833 29724 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:16 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:16.611508 29728 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509503
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:17 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:17.590775 29725 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:17 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:17.592618 29723 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509504
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:18 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:18.153723 29725 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:18 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:18.155370 29723 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509505
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:22 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:22.695742 29722 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:22 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:22.697412 29724 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509512
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:23 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:23.714365 29722 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:23 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:23.716039 29726 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509514
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:24 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:24.728322 29727 master.cpp:7105] Sending 1 offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:24 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:24.728845 29727 master.cpp:7194] Sending 1 inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:24 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:24.729948 29728 master.cpp:4806] Processing DECLINE call for offers: [ 92434aef-27da-4fd1-a5c4-b286d640d5b3-O509515
] for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092 (TeraValidate)
at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:25.295658 29723 master.cpp:7788] Processing TEARDOWN call for framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:25.295702 29723 master.cpp:7800] Removing framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:25.295724 29723 master.cpp:3160] Deactivating framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(TeraValidate) at scheduler-3b84262b-e1a6-47a8-ac0f-00af50b24f5c@172.31.7.83:45531
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:25.296236 29724 hierarchical.cpp:379] Deactivated framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:25.298550 29723 master.cpp:8368] Updating the state of task 1 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
(latest state: TASK_KILLED, status update state: TASK_KILLED)
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:25.298677 29723 master.cpp:8462] Removing task 1 with resources cpus(*)(allocated: *):1;
mem(*)(allocated: *):1408 of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
on agent 36a25adb-4ea2-49d3-a195-448cff1dc146-S2 at slave(1)@172.31.7.202:5051 (172.31.7.202)
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:25.298863 29726 hierarchical.cpp:326] Removed framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:25 ip-172-31-6-200.us-west-2.compute.internal mesos-master[29716]: I0601
11:33:25.299028 29723 master.cpp:7118] Master ignoring inverse offers to framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
because the framework has terminated or is inactive
> {noformat}
> *Agent log*
> {noformat}
> [centos@ip-172-31-7-202 ~]$ journalctl _PID=12073 --since "1 hour ago" --no-pager | grep
-C 10 "failed to start:"
> Jun 01 11:33:28 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:28.785028 12081 http.cpp:1115] HTTP GET for /slave(1)/state from 172.31.7.202:52855
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:29.109624 12080 logfmt.cpp:177] dstip=172.31.7.202 type=audit timestamp=2017-06-01 11:33:29.109526016+00:00
reason="Valid authorization token" uid="dcos_metrics_agent" object="/slave(1)/state" agent="dcos-metrics/1.1.0-64-g62702c3"
authorizer="mesos-agent" action="GET" result=allow srcip=172.31.7.202 dstport=5051 srcport=49102
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:29.109864 12081 http.cpp:1115] HTTP GET for /slave(1)/state from 172.31.7.202:49102
with User-Agent='dcos-metrics/1.1.0-64-g62702c3'
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:29.347921 12084 logfmt.cpp:177] dstip=172.31.7.202 type=audit timestamp=2017-06-01 11:33:29.347860992+00:00
reason="Valid authorization token" uid="dcos_metrics_agent" object="/slave(1)/containers"
agent="dcos-metrics/1.1.0-64-g62702c3" authorizer="mesos-agent" action="GET" result=allow
srcip=172.31.7.202 dstport=5051 srcport=49102
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:29.348116 12077 http.cpp:1115] HTTP GET for /slave(1)/containers from 172.31.7.202:49102
with User-Agent='dcos-metrics/1.1.0-64-g62702c3'
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: W0601
11:33:29.712091 12079 http.cpp:2160] Failed to get resource statistics for executor '"1"'
of framework "6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092": Failed
to run 'docker -H unix:///var/run/docker.sock inspect mesos-36a25adb-4ea2-49d3-a195-448cff1dc146-S2.5c1f53e4-fffe-4f3a-8847-c5a252a25d3b':
exited with status 1; stderr='Error: No such object: mesos-36a25adb-4ea2-49d3-a195-448cff1dc146-S2.5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
> Jun 01 11:33:29 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: '
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:30.298966 12083 slave.cpp:5548] Killing executor '1' of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:30.299078 12083 docker.cpp:2123] Destroying container 5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:30.299108 12083 docker.cpp:2165] Destroying container 5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
in PULLING state
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: E0601
11:33:30.415652 12082 slave.cpp:5041] Container '5c1f53e4-fffe-4f3a-8847-c5a252a25d3b' for
executor '1' of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
failed to start:  future discarded
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: E0601
11:33:30.415765 12082 slave.cpp:5148] Termination of executor '1' of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
failed: unknown container
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:30.415794 12082 slave.cpp:5261] Cleaning up executor '1' of framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: W0601
11:33:30.415937 12077 composing.cpp:638] Attempted to destroy unknown container 5c1f53e4-fffe-4f3a-8847-c5a252a25d3b
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:30.415966 12082 slave.cpp:5349] Cleaning up framework 6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:30.415992 12077 gc.cpp:55] Scheduling '/var/lib/mesos/slave/slaves/36a25adb-4ea2-49d3-a195-448cff1dc146-S2/frameworks/6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092/executors/1/runs/5c1f53e4-fffe-4f3a-8847-c5a252a25d3b'
for gc 1.99999518647111days in the future
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:30.416050 12082 status_update_manager.cpp:285] Closing status update streams for framework
6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:30.416061 12077 gc.cpp:55] Scheduling '/var/lib/mesos/slave/slaves/36a25adb-4ea2-49d3-a195-448cff1dc146-S2/frameworks/6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092/executors/1'
for gc 1.99999518583407days in the future
> Jun 01 11:33:30 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:30.416138 12077 gc.cpp:55] Scheduling '/var/lib/mesos/slave/slaves/36a25adb-4ea2-49d3-a195-448cff1dc146-S2/frameworks/6dd898d6-7f3a-406c-8ead-24b4d55ed262-0018-driver-20170601113252-0092'
for gc 1.99999518486222days in the future
> Jun 01 11:33:31 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:31.574026 12079 logfmt.cpp:177] dstip=172.31.7.202 type=audit timestamp=2017-06-01 11:33:31.573729024+00:00
reason="Valid authorization token" uid="dcos_navstar_agent" object="/slave(1)/state" authorizer="mesos-agent"
action="GET" result=allow srcip=172.31.7.202 dstport=5051 srcport=52855
> Jun 01 11:33:31 ip-172-31-7-202.us-west-2.compute.internal mesos-agent[12073]: I0601
11:33:31.574285 12079 http.cpp:1115] HTTP GET for /slave(1)/state from 172.31.7.202:52855
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message