Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 5EFBE200B61 for ; Tue, 9 Aug 2016 12:25:24 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 5DC43160AA4; Tue, 9 Aug 2016 10:25:24 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A98CF160AA5 for ; Tue, 9 Aug 2016 12:25:21 +0200 (CEST) Received: (qmail 8328 invoked by uid 500); 9 Aug 2016 10:25:20 -0000 Mailing-List: contact issues-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@mesos.apache.org Delivered-To: mailing list issues@mesos.apache.org Received: (qmail 8280 invoked by uid 99); 9 Aug 2016 10:25:20 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Aug 2016 10:25:20 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 7B9E12C02A4 for ; Tue, 9 Aug 2016 10:25:20 +0000 (UTC) Date: Tue, 9 Aug 2016 10:25:20 +0000 (UTC) From: "Michael Thomas (JIRA)" To: issues@mesos.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (MESOS-6004) Tasks fail when provisioning multiple containers with large docker images using copy backend MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 09 Aug 2016 10:25:24 -0000 [ https://issues.apache.org/jira/browse/MESOS-6004?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Thomas updated MESOS-6004: ---------------------------------- Description:=20 When scheduling more than one task on the same agent, all tasks fail a as c= ontainers seem to be destroyed during provisioning. Specifically, the errors on the agent logs are: {noformat} E0808 15:53:09.691315 30996 slave.cpp:3976] Container 'eb20f642-bb90-4293-= 8eec-6f1576ccaeb1' for executor '3' of framework c9852a23-bc07-422d-8d69-23= c167a1924d-0001 failed to start: Container is being destroyed during provis= ioning {noformat} and=20 {noformat} I0808 15:52:32.510210 30999 slave.cpp:4539] Terminating executor ''2' of fr= amework c9852a23-bc07-422d-8d69-23c167a1924d-0001' because it did not regis= ter within 3mins {noformat} As the default provisioning method {{copy}} is being used, I assume this is= due to the provisioning of multiple containers taking too long and the age= nt will not wait. For large images, this method is simply not performant. The issue did not occur, when only one tasks was scheduled. Increasing the {{executor_registration_timeout}} parameter, seemed to help = a bit as it allowed to schedule at least 2 tasks at the same time. But stil= l fails with more (5 in this case) h4. Complete logs (with GLOG_v=3D1) {noformat} Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.800375 3= 738 slave.cpp:198] Agent started on 1)@172.31.23.17:5051 Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.800403 3= 738 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix=3D"= http://" --appc_store_dir=3D"/tmp/mesos/store/appc" --authenticate_http_rea= donly=3D"false" --authenticate_http_readwrite=3D"false" --authenticatee=3D"= crammd5" --authentication_backoff_factor=3D"1secs" --authorizer=3D"local" -= -cgroups_cpu_enable_pids_and_tids_count=3D"false" --cgroups_enable_cfs=3D"f= alse" --cgroups_hierarchy=3D"/sys/fs/cgroup" --cgroups_limit_swap=3D"false"= --cgroups_root=3D"mesos" --container_disk_watch_interval=3D"15secs" --cont= ainerizers=3D"mesos,docker" --default_role=3D"*" --disk_watch_interval=3D"1= mins" --docker=3D"docker" --docker_config=3D"{"auths":{"https:\/\/index.doc= ker.io\/v1\/":{"auth":"dGVycmFsb3VwZTpUYWxFWUFOSXR5","email":"sebastian.ger= ke@terraloupe.com"}}}" --docker_kill_orphans=3D"true" --docker_registry=3D"= https://registry-1.docker.io" --docker_remove_delay=3D"6hrs" --docker_socke= t=3D"/var/run/docker.sock" --docker_stop_timeout=3D"0ns" --docker_store_dir= =3D"/mnt/t" --docker_volume_checkp Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: oint_dir=3D"/var/run/mes= os/isolators/docker/volume" --enforce_container_disk_quota=3D"false" --exec= utor_registration_timeout=3D"1mins" --executor_shutdown_grace_period=3D"5se= cs" --fetcher_cache_dir=3D"/tmp/mesos/fetch" --fetcher_cache_size=3D"2GB" -= -frameworks_home=3D"" --gc_delay=3D"1weeks" --gc_disk_headroom=3D"0.1" --ha= doop_home=3D"" --help=3D"false" --hostname=3D"ec2-52-59-113-0.eu-central-1.= compute.amazonaws.com" --hostname_lookup=3D"true" --http_authenticators=3D"= basic" --http_command_executor=3D"false" --image_providers=3D"docker" --ima= ge_provisioner_backend=3D"copy" --initialize_driver_logging=3D"true" --isol= ation=3D"filesystem/linux,docker/runtime,cgroups/devices,gpu/nvidia" --laun= cher_dir=3D"/usr/libexec/mesos" --log_dir=3D"/var/log/mesos" --logbufsecs= =3D"0" --logging_level=3D"INFO" --master=3D"zk://172.31.19.240:2181/mesos" = --oversubscribed_resources_interval=3D"15secs" --perf_duration=3D"10secs" -= -perf_interval=3D"1mins" --port=3D"5051" --qos_correction_interval_min=3D"0= ns" --quiet=3D"false" --recover=3D"reconnect" --re Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: covery_timeout=3D"15mins= " --registration_backoff_factor=3D"1secs" --revocable_cpu_low_priority=3D"t= rue" --sandbox_directory=3D"/mnt/mesos/sandbox" --strict=3D"true" --switch_= user=3D"false" --systemd_enable_support=3D"true" --systemd_runtime_director= y=3D"/run/systemd/system" --version=3D"false" --work_dir=3D"/mnt/mesos" Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: 2016-08-09 10:11:41,800:= 3738(0x7f71172a1700):ZOO_INFO@check_events@1728: initiated connection to se= rver [172.31.19.240:2181] Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801007 3= 738 slave.cpp:519] Agent resources: gpus(*):1; cpus(*):8; mem(*):14014; dis= k(*):60257; ports(*):[31000-32000] Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801059 3= 738 slave.cpp:527] Agent attributes: [ ] Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801077 3= 738 slave.cpp:532] Agent hostname: ec2-52-59-113-0.eu-central-1.compute.ama= zonaws.com Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.802232 3= 772 state.cpp:57] Recovering state from '/mnt/mesos/meta' Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803539 3= 774 status_update_manager.cpp:200] Recovering status update manager Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803745 3= 772 containerizer.cpp:522] Recovering containerizer Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803788 3= 770 docker.cpp:775] Recovering Docker containers Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.808550 3= 777 metadata_manager.cpp:251] Successfully loaded 2 Docker images Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.808688 3= 771 provisioner.cpp:253] Provisioner recovery complete Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: 2016-08-09 10:11:41,816:= 3738(0x7f71172a1700):ZOO_INFO@check_events@1775: session establishment comp= lete on server [172.31.19.240:2181], sessionId=3D0x1566a66ab9b0010, negotia= ted timeout=3D10000 Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817003 3= 774 group.cpp:349] Group process (group(1)@172.31.23.17:5051) connected to = ZooKeeper Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817116 3= 774 group.cpp:837] Syncing group operations: queue size (joins, cancels, da= tas) =3D (0, 0, 0) Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817147 3= 774 group.cpp:427] Trying to create path '/mesos' in ZooKeeper Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.818857 3= 774 detector.cpp:152] Detected a new leader: (id=3D'64') Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.819005 3= 776 group.cpp:706] Trying to get '/mesos/json.info_0000000064' in ZooKeeper Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.819855 3= 775 zookeeper.cpp:259] A new leading master (UPID=3Dmaster@172.31.19.240:50= 50) is detected Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.899876 3= 777 slave.cpp:4782] Finished recovery Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900573 3= 777 slave.cpp:4815] Garbage collecting old agent 524105e7-de7a-43f0-8b28-d3= ff3e0c4a44-S2 Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900687 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e= 0c4a44-S2' for gc 6.99998957623704days in the future Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900729 3= 777 slave.cpp:4815] Garbage collecting old agent c9852a23-bc07-422d-8d69-23= c167a1924d-S0 Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900779 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d= 3ff3e0c4a44-S2' for gc 6.99998957520889days in the future Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900817 3= 777 slave.cpp:4815] Garbage collecting old agent 524105e7-de7a-43f0-8b28-d3= ff3e0c4a44-S1 Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900825 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167= a1924d-S0' for gc 6.99998957451259days in the future Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900899 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/c9852a23-bc07-422d-8d69-2= 3c167a1924d-S0' for gc 6.99998957412148days in the future Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900933 3= 777 slave.cpp:4815] Garbage collecting old agent 524105e7-de7a-43f0-8b28-d3= ff3e0c4a44-S3 Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900944 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e= 0c4a44-S1' for gc 6.99998957332148days in the future Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901010 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d= 3ff3e0c4a44-S1' for gc 6.99998957291556days in the future Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901051 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e= 0c4a44-S3' for gc 6.99998957196741days in the future Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901101 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d= 3ff3e0c4a44-S3' for gc 6.99998957156148days in the future Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901201 3= 775 status_update_manager.cpp:174] Pausing sending status updates Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901203 3= 777 slave.cpp:895] New master detected at master@172.31.19.240:5050 Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901278 3= 777 slave.cpp:916] No credentials provided. Attempting to register without = authentication Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901317 3= 777 slave.cpp:927] Detecting new master Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267201 3= 774 slave.cpp:1197] Re-registered with master master@172.31.19.240:5050 Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267302 3= 771 status_update_manager.cpp:181] Resuming sending status updates Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267302 3= 774 slave.cpp:1233] Forwarding total oversubscribed resources Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267577 3= 774 slave.cpp:2526] Updated checkpointed resources from to Aug 9 10:11:48 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:48.269104 3= 777 slave.cpp:2218] Asked to shut down framework c9852a23-bc07-422d-8d69-23= c167a1924d-0002 by master@172.31.19.240:5050 Aug 9 10:11:48 ip-172-31-23-17 mesos-slave[3738]: W0809 10:11:48.269186 3= 777 slave.cpp:2233] Cannot shut down unknown framework c9852a23-bc07-422d-8= d69-23c167a1924d-0002 Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.969521 3= 772 slave.cpp:1495] Got assigned task 0 for framework c9852a23-bc07-422d-8d= 69-23c167a1924d-0003 Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.970026 3= 772 slave.cpp:1614] Launching task 0 for framework c9852a23-bc07-422d-8d69-= 23c167a1924d-0003 Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.970826 3= 772 slave.cpp:5674] Launching executor 0 of framework c9852a23-bc07-422d-8d= 69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work director= y '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c98= 52a23-bc07-422d-8d69-23c167a1924d-0003/executors/0/runs/09a77a80-f904-4581-= 966a-734d24ae96f3' Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.971051 3= 772 slave.cpp:1840] Queuing task '0' for executor '0' of framework c9852a23= -bc07-422d-8d69-23c167a1924d-0003 Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.971212 3= 773 containerizer.cpp:781] Starting container '09a77a80-f904-4581-966a-734d= 24ae96f3' for executor '0' of framework 'c9852a23-bc07-422d-8d69-23c167a192= 4d-0003' Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.973953 3= 774 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner/= containers/09a77a80-f904-4581-966a-734d24ae96f3/backends/copy/rootfses/ec9b= b10e-6726-4872-be2b-4a72dfc0fafc' for container 09a77a80-f904-4581-966a-734= d24ae96f3 Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978344 3= 774 slave.cpp:1495] Got assigned task 1 for framework c9852a23-bc07-422d-8d= 69-23c167a1924d-0003 Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978687 3= 774 slave.cpp:1614] Launching task 1 for framework c9852a23-bc07-422d-8d69-= 23c167a1924d-0003 Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978976 3= 774 slave.cpp:5674] Launching executor 1 of framework c9852a23-bc07-422d-8d= 69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work director= y '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c98= 52a23-bc07-422d-8d69-23c167a1924d-0003/executors/1/runs/0cea1dc7-b523-4b68-= 851a-531ed101de5a' Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.979173 3= 774 slave.cpp:1840] Queuing task '1' for executor '1' of framework c9852a23= -bc07-422d-8d69-23c167a1924d-0003 Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.979387 3= 777 containerizer.cpp:781] Starting container '0cea1dc7-b523-4b68-851a-531e= d101de5a' for executor '1' of framework 'c9852a23-bc07-422d-8d69-23c167a192= 4d-0003' Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.981374 3= 776 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner/= containers/0cea1dc7-b523-4b68-851a-531ed101de5a/backends/copy/rootfses/e3af= 89fe-c889-45fc-929b-f008b18e5d4e' for container 0cea1dc7-b523-4b68-851a-531= ed101de5a Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.983610 3= 776 slave.cpp:1495] Got assigned task 2 for framework c9852a23-bc07-422d-8d= 69-23c167a1924d-0003 Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.983947 3= 776 slave.cpp:1614] Launching task 2 for framework c9852a23-bc07-422d-8d69-= 23c167a1924d-0003 Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984236 3= 776 slave.cpp:5674] Launching executor 2 of framework c9852a23-bc07-422d-8d= 69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work director= y '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c98= 52a23-bc07-422d-8d69-23c167a1924d-0003/executors/2/runs/a4c972da-b12d-4b4b-= 8481-79ca320ddab7' Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984450 3= 776 slave.cpp:1840] Queuing task '2' for executor '2' of framework c9852a23= -bc07-422d-8d69-23c167a1924d-0003 Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984778 3= 777 containerizer.cpp:781] Starting container 'a4c972da-b12d-4b4b-8481-79ca= 320ddab7' for executor '2' of framework 'c9852a23-bc07-422d-8d69-23c167a192= 4d-0003' Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.986330 3= 776 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner/= containers/a4c972da-b12d-4b4b-8481-79ca320ddab7/backends/copy/rootfses/feaf= 70a4-df43-45c9-b6be-a654c9210a54' for container a4c972da-b12d-4b4b-8481-79c= a320ddab7 Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.996104 3= 771 slave.cpp:1495] Got assigned task 3 for framework c9852a23-bc07-422d-8d= 69-23c167a1924d-0003 Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.996462 3= 771 slave.cpp:1614] Launching task 3 for framework c9852a23-bc07-422d-8d69-= 23c167a1924d-0003 Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997241 3= 771 slave.cpp:5674] Launching executor 3 of framework c9852a23-bc07-422d-8d= 69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work director= y '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c98= 52a23-bc07-422d-8d69-23c167a1924d-0003/executors/3/runs/d9363f0d-645b-468f-= 947a-8bf97d15a630' Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997406 3= 771 slave.cpp:1840] Queuing task '3' for executor '3' of framework c9852a23= -bc07-422d-8d69-23c167a1924d-0003 Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997640 3= 777 containerizer.cpp:781] Starting container 'd9363f0d-645b-468f-947a-8bf9= 7d15a630' for executor '3' of framework 'c9852a23-bc07-422d-8d69-23c167a192= 4d-0003' Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.999593 3= 777 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner/= containers/d9363f0d-645b-468f-947a-8bf97d15a630/backends/copy/rootfses/eeb1= b066-94b8-4880-a696-090f830cf2d6' for container d9363f0d-645b-468f-947a-8bf= 97d15a630 Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.003924 3= 775 slave.cpp:1495] Got assigned task 4 for framework c9852a23-bc07-422d-8d= 69-23c167a1924d-0003 Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.004278 3= 775 slave.cpp:1614] Launching task 4 for framework c9852a23-bc07-422d-8d69-= 23c167a1924d-0003 Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005383 3= 775 slave.cpp:5674] Launching executor 4 of framework c9852a23-bc07-422d-8d= 69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work director= y '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c98= 52a23-bc07-422d-8d69-23c167a1924d-0003/executors/4/runs/58c398e2-38f9-4389-= 87ba-8696c9a523eb' Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005681 3= 775 slave.cpp:1840] Queuing task '4' for executor '4' of framework c9852a23= -bc07-422d-8d69-23c167a1924d-0003 Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005971 3= 772 containerizer.cpp:781] Starting container '58c398e2-38f9-4389-87ba-8696= c9a523eb' for executor '4' of framework 'c9852a23-bc07-422d-8d69-23c167a192= 4d-0003' Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.007786 3= 777 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner/= containers/58c398e2-38f9-4389-87ba-8696c9a523eb/backends/copy/rootfses/78b7= bf3a-6f3b-4f8a-b4fa-98a6ea4bf8f5' for container 58c398e2-38f9-4389-87ba-869= 6c9a523eb Aug 9 10:12:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:41.801560 3= 777 slave.cpp:4591] Current disk usage 43.28%. Max allowed age: 3.270091966= 898553days Aug 9 10:12:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:54.972057 3= 772 slave.cpp:4539] Terminating executor ''0' of framework c9852a23-bc07-42= 2d-8d69-23c167a1924d-0003' because it did not register within 1mins Aug 9 10:12:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:54.972322 3= 775 containerizer.cpp:1622] Destroying container '09a77a80-f904-4581-966a-7= 34d24ae96f3' Aug 9 10:13:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:00.979907 3= 777 slave.cpp:4539] Terminating executor ''1' of framework c9852a23-bc07-42= 2d-8d69-23c167a1924d-0003' because it did not register within 1mins Aug 9 10:13:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:00.980058 3= 775 containerizer.cpp:1622] Destroying container '0cea1dc7-b523-4b68-851a-5= 31ed101de5a' Aug 9 10:13:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:06.985003 3= 777 slave.cpp:4539] Terminating executor ''2' of framework c9852a23-bc07-42= 2d-8d69-23c167a1924d-0003' because it did not register within 1mins Aug 9 10:13:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:06.985255 3= 774 containerizer.cpp:1622] Destroying container 'a4c972da-b12d-4b4b-8481-7= 9ca320ddab7' Aug 9 10:13:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:12.998438 3= 777 slave.cpp:4539] Terminating executor ''3' of framework c9852a23-bc07-42= 2d-8d69-23c167a1924d-0003' because it did not register within 1mins Aug 9 10:13:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:12.998641 3= 776 containerizer.cpp:1622] Destroying container 'd9363f0d-645b-468f-947a-8= bf97d15a630' Aug 9 10:13:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:19.006058 3= 773 slave.cpp:4539] Terminating executor ''4' of framework c9852a23-bc07-42= 2d-8d69-23c167a1924d-0003' because it did not register within 1mins Aug 9 10:13:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:19.006228 3= 773 containerizer.cpp:1622] Destroying container '58c398e2-38f9-4389-87ba-8= 696c9a523eb' Aug 9 10:13:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:41.802209 3= 775 slave.cpp:4591] Current disk usage 57.59%. Max allowed age: 2.269006805= 055764days Aug 9 10:14:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:14:41.803022 3= 776 slave.cpp:4591] Current disk usage 68.70%. Max allowed age: 1.491204118= 572176days Aug 9 10:15:01 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:01.516207 3= 775 slave.cpp:3976] Container '58c398e2-38f9-4389-87ba-8696c9a523eb' for ex= ecutor '4' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to= start: Container is being destroyed during provisioning Aug 9 10:15:01 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:01.518277 3= 772 composing.cpp:541] Container '58c398e2-38f9-4389-87ba-8696c9a523eb' is = already destroyed Aug 9 10:15:02 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:02.291764 3= 775 slave.cpp:3976] Container '09a77a80-f904-4581-966a-734d24ae96f3' for ex= ecutor '0' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to= start: Container is being destroyed during provisioning Aug 9 10:15:02 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:02.291877 3= 775 composing.cpp:541] Container '09a77a80-f904-4581-966a-734d24ae96f3' is = already destroyed Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:03.022406 3= 774 slave.cpp:3976] Container 'd9363f0d-645b-468f-947a-8bf97d15a630' for ex= ecutor '3' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to= start: Container is being destroyed during provisioning Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:03.022595 3= 771 composing.cpp:541] Container 'd9363f0d-645b-468f-947a-8bf97d15a630' is = already destroyed Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:03.461519 3= 775 slave.cpp:3976] Container '0cea1dc7-b523-4b68-851a-531ed101de5a' for ex= ecutor '1' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to= start: Container is being destroyed during provisioning Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:03.461643 3= 774 composing.cpp:541] Container '0cea1dc7-b523-4b68-851a-531ed101de5a' is = already destroyed Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.190901 3= 776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provisi= oner/containers/58c398e2-38f9-4389-87ba-8696c9a523eb/backends/copy/rootfses= /78b7bf3a-6f3b-4f8a-b4fa-98a6ea4bf8f5' for container 58c398e2-38f9-4389-87b= a-8696c9a523eb Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:04.190904 3= 777 slave.cpp:3976] Container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' for ex= ecutor '2' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to= start: Container is being destroyed during provisioning Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:04.191128 3= 770 composing.cpp:541] Container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' is = already destroyed Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197221 3= 776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provisi= oner/containers/09a77a80-f904-4581-966a-734d24ae96f3/backends/copy/rootfses= /ec9bb10e-6726-4872-be2b-4a72dfc0fafc' for container 09a77a80-f904-4581-966= a-734d24ae96f3 Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197410 3= 776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provisi= oner/containers/d9363f0d-645b-468f-947a-8bf97d15a630/backends/copy/rootfses= /eeb1b066-94b8-4880-a696-090f830cf2d6' for container d9363f0d-645b-468f-947= a-8bf97d15a630 Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197510 3= 776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provisi= oner/containers/0cea1dc7-b523-4b68-851a-531ed101de5a/backends/copy/rootfses= /e3af89fe-c889-45fc-929b-f008b18e5d4e' for container 0cea1dc7-b523-4b68-851= a-531ed101de5a Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197619 3= 776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provisi= oner/containers/a4c972da-b12d-4b4b-8481-79ca320ddab7/backends/copy/rootfses= /feaf70a4-df43-45c9-b6be-a654c9210a54' for container a4c972da-b12d-4b4b-848= 1-79ca320ddab7 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.390928 3= 776 slave.cpp:4082] Executor '3' of framework c9852a23-bc07-422d-8d69-23c16= 7a1924d-0003 has terminated with unknown status Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.395370 3= 776 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 543e1e1e-ae47= -48ab-ad64-ee82fdd27418) for task 3 of framework c9852a23-bc07-422d-8d69-23= c167a1924d-0003 from @0.0.0.0:0 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.402472 3= 776 slave.cpp:4082] Executor '2' of framework c9852a23-bc07-422d-8d69-23c16= 7a1924d-0003 has terminated with unknown status Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.402556 3= 776 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 1a54664d-4157= -42a3-a97f-7189c137a769) for task 2 of framework c9852a23-bc07-422d-8d69-23= c167a1924d-0003 from @0.0.0.0:0 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.406177 3= 772 containerizer.cpp:1451] Ignoring update for unknown container: d9363f0d= -645b-468f-947a-8bf97d15a630 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.407325 3= 777 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID= : 543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework c9852a23-bc= 07-422d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.407598 3= 773 containerizer.cpp:1451] Ignoring update for unknown container: a4c972da= -b12d-4b4b-8481-79ca320ddab7 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.409890 3= 774 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 543e1e1e-ae47-= 48ab-ad64-ee82fdd27418) for task 3 of framework c9852a23-bc07-422d-8d69-23c= 167a1924d-0003 to master@172.31.19.240:5050 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.409893 3= 777 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID= : 1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework c9852a23-bc= 07-422d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.414727 3= 774 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 1a54664d-4157-= 42a3-a97f-7189c137a769) for task 2 of framework c9852a23-bc07-422d-8d69-23c= 167a1924d-0003 to master@172.31.19.240:5050 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.418180 3= 777 status_update_manager.cpp:392] Received status update acknowledgement (= UUID: 543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework c9852a2= 3-bc07-422d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419615 3= 777 status_update_manager.cpp:392] Received status update acknowledgement (= UUID: 1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework c9852a2= 3-bc07-422d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419636 3= 774 slave.cpp:4193] Cleaning up executor '3' of framework c9852a23-bc07-422= d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419924 3= 772 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167= a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3/= runs/d9363f0d-645b-468f-947a-8bf97d15a630' for gc 6.9999951410637days in th= e future Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420703 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167= a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3'= for gc 6.99999513154074days in the future Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420722 3= 774 slave.cpp:4193] Cleaning up executor '2' of framework c9852a23-bc07-422= d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420897 3= 774 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167= a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2/= runs/a4c972da-b12d-4b4b-8481-79ca320ddab7' for gc 6.99999512931852days in t= he future Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420940 3= 774 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167= a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2'= for gc 6.99999512869333days in the future Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490155 3= 772 slave.cpp:4082] Executor '4' of framework c9852a23-bc07-422d-8d69-23c16= 7a1924d-0003 has terminated with unknown status Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490289 3= 772 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 9c5528ec-0725= -42f2-a2e2-465d3c73b8d5) for task 4 of framework c9852a23-bc07-422d-8d69-23= c167a1924d-0003 from @0.0.0.0:0 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490414 3= 772 slave.cpp:4082] Executor '0' of framework c9852a23-bc07-422d-8d69-23c16= 7a1924d-0003 has terminated with unknown status Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490489 3= 772 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 3f801210-b029= -427f-af3f-e51b1dfa5cc8) for task 0 of framework c9852a23-bc07-422d-8d69-23= c167a1924d-0003 from @0.0.0.0:0 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.490687 3= 773 containerizer.cpp:1451] Ignoring update for unknown container: 58c398e2= -38f9-4389-87ba-8696c9a523eb Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.490774 3= 773 containerizer.cpp:1451] Ignoring update for unknown container: 09a77a80= -f904-4581-966a-734d24ae96f3 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490867 3= 773 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID= : 9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework c9852a23-bc= 07-422d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491082 3= 773 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID= : 3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework c9852a23-bc= 07-422d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491082 3= 772 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 9c5528ec-0725-= 42f2-a2e2-465d3c73b8d5) for task 4 of framework c9852a23-bc07-422d-8d69-23c= 167a1924d-0003 to master@172.31.19.240:5050 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491303 3= 774 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 3f801210-b029-= 427f-af3f-e51b1dfa5cc8) for task 0 of framework c9852a23-bc07-422d-8d69-23c= 167a1924d-0003 to master@172.31.19.240:5050 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494438 3= 775 status_update_manager.cpp:392] Received status update acknowledgement (= UUID: 9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework c9852a2= 3-bc07-422d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494572 3= 775 slave.cpp:4193] Cleaning up executor '4' of framework c9852a23-bc07-422= d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494757 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167= a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4/= runs/58c398e2-38f9-4389-87ba-8696c9a523eb' for gc 6.99999427436444days in t= he future Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494817 3= 775 status_update_manager.cpp:392] Received status update acknowledgement (= UUID: 3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework c9852a2= 3-bc07-422d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494937 3= 775 slave.cpp:4193] Cleaning up executor '0' of framework c9852a23-bc07-422= d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494823 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167= a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4'= for gc 6.99999427361185days in the future Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.495107 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167= a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0/= runs/09a77a80-f904-4581-966a-734d24ae96f3' for gc 6.99999427055407days in t= he future Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.495149 3= 773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167= a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0'= for gc 6.99999427024296days in the future Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591338 3= 775 slave.cpp:4082] Executor '1' of framework c9852a23-bc07-422d-8d69-23c16= 7a1924d-0003 has terminated with unknown status Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591469 3= 775 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 85b7c18a-c59c= -4449-827c-a3bfaa2a7de8) for task 1 of framework c9852a23-bc07-422d-8d69-23= c167a1924d-0003 from @0.0.0.0:0 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.591841 3= 776 containerizer.cpp:1451] Ignoring update for unknown container: 0cea1dc7= -b523-4b68-851a-531ed101de5a Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591974 3= 770 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID= : 85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework c9852a23-bc= 07-422d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.592157 3= 772 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 85b7c18a-c59c-= 4449-827c-a3bfaa2a7de8) for task 1 of framework c9852a23-bc07-422d-8d69-23c= 167a1924d-0003 to master@172.31.19.240:5050 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594518 3= 771 status_update_manager.cpp:392] Received status update acknowledgement (= UUID: 85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework c9852a2= 3-bc07-422d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594636 3= 771 slave.cpp:4193] Cleaning up executor '1' of framework c9852a23-bc07-422= d-8d69-23c167a1924d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594792 3= 770 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167= a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1/= runs/0cea1dc7-b523-4b68-851a-531ed101de5a' for gc 6.99999311635556days in t= he future Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594820 3= 771 slave.cpp:4281] Cleaning up framework c9852a23-bc07-422d-8d69-23c167a19= 24d-0003 Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594861 3= 770 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167= a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1'= for gc 6.99999311568296days in the future Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594897 3= 770 status_update_manager.cpp:282] Closing status update streams for framew= ork c9852a23-bc07-422d-8d69-23c167a1924d-0003 {noformat} was: When scheduling more than one task on the same agent, all tasks fail a as c= ontainers seem to be destroyed during provisioning. Specifically, the errors on the agent logs are: {noformat} E0808 15:53:09.691315 30996 slave.cpp:3976] Container 'eb20f642-bb90-4293-= 8eec-6f1576ccaeb1' for executor '3' of framework c9852a23-bc07-422d-8d69-23= c167a1924d-0001 failed to start: Container is being destroyed during provis= ioning {noformat} and=20 {noformat} I0808 15:52:32.510210 30999 slave.cpp:4539] Terminating executor ''2' of fr= amework c9852a23-bc07-422d-8d69-23c167a1924d-0001' because it did not regis= ter within 3mins {noformat} As the default provisioning method {{copy}} is being used, I assume this is= due to the provisioning of multiple containers taking too long and the age= nt will not wait. For large images, this method is simply not performant. The issue did not occur, when only one tasks was scheduled. Increasing the {{executor_registration_timeout}} parameter, seemed to help = a bit as it allowed to schedule at least 2 tasks at the same time. But stil= l fails with more (5 in this case) h4. Complete logs (with GLOG_v=3D0, as with 1 it was too long) {noformat} Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.661067 3= 0961 main.cpp:434] Starting Mesos agent Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.661551 3= 0961 slave.cpp:198] Agent started on 1)@172.31.23.17:5051 Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.661578 3= 0961 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix=3D= "http://" --appc_store_dir=3D"/tmp/mesos/store/appc" --authenticate_http_re= adonly=3D"false" --authenticate_http_readwrite=3D"false" --authenticatee=3D= "crammd5" --authentication_backoff_factor=3D"1secs" --authorizer=3D"local" = --cgroups_cpu_enable_pids_and_tids_count=3D"false" --cgroups_enable_cfs=3D"= false" --cgroups_hierarchy=3D"/sys/fs/cgroup" --cgroups_limit_swap=3D"false= " --cgroups_root=3D"mesos" --container_disk_watch_interval=3D"15secs" --con= tainerizers=3D"mesos,docker" --default_role=3D"*" --disk_watch_interval=3D"= 1mins" --docker=3D"docker" --docker_config=3D"XXX" --docker_kill_orphans=3D= "true" --docker_registry=3D"https://registry-1.docker.io" --docker_remove_d= elay=3D"6hrs" --docker_socket=3D"/var/run/docker.sock" --docker_stop_timeou= t=3D"0ns" --docker_store_dir=3D"/mnt/mesos/store/docker" --do Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: cker_volume_checkpoint_= dir=3D"/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quo= ta=3D"false" --executor_registration_timeout=3D"3mins" --executor_shutdown_= grace_period=3D"5secs" --fetcher_cache_dir=3D"/tmp/mesos/fetch" --fetcher_c= ache_size=3D"2GB" --frameworks_home=3D"" --gc_delay=3D"1weeks" --gc_disk_he= adroom=3D"0.1" --hadoop_home=3D"" --help=3D"false" --hostname=3D"ec2-52-59-= 113-0.eu-central-1.compute.amazonaws.com" --hostname_lookup=3D"true" --http= _authenticators=3D"basic" --http_command_executor=3D"false" --image_provide= rs=3D"docker" --image_provisioner_backend=3D"copy" --initialize_driver_logg= ing=3D"true" --isolation=3D"filesystem/linux,docker/runtime,cgroups/devices= ,gpu/nvidia" --launcher_dir=3D"/usr/libexec/mesos" --log_dir=3D"/var/log/me= sos" --logbufsecs=3D"0" --logging_level=3D"INFO" --master=3D"zk://172.31.19= .240:2181/mesos" --oversubscribed_resources_interval=3D"15secs" --perf_dura= tion=3D"10secs" --perf_interval=3D"1mins" --port=3D"5051" --qos_correction_= interval_min=3D"0ns" --quiet=3D"false" --recov Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: er=3D"reconnect" --reco= very_timeout=3D"15mins" --registration_backoff_factor=3D"1secs" --revocable= _cpu_low_priority=3D"true" --sandbox_directory=3D"/mnt/mesos/sandbox" --str= ict=3D"true" --switch_user=3D"false" --systemd_enable_support=3D"true" --sy= stemd_runtime_directory=3D"/run/systemd/system" --version=3D"false" --work_= dir=3D"/mnt/mesos" Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.662147 3= 0961 slave.cpp:519] Agent resources: gpus(*):1; cpus(*):8; mem(*):14014; di= sk(*):60257; ports(*):[31000-32000] Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.662211 3= 0961 slave.cpp:527] Agent attributes: [ ] Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.662230 3= 0961 slave.cpp:532] Agent hostname: ec2-52-59-113-0.eu-central-1.compute.am= azonaws.com Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.663354 3= 1000 state.cpp:57] Recovering state from '/mnt/mesos/meta' Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.663918 3= 0995 status_update_manager.cpp:200] Recovering status update manager Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.664131 3= 0996 containerizer.cpp:522] Recovering containerizer Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.664136 3= 1000 docker.cpp:775] Recovering Docker containers Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: 2016-08-08 15:48:32,665= :30961(0x7fce36077700):ZOO_INFO@check_events@1728: initiated connection to = server [172.31.19.240:2181] Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: 2016-08-08 15:48:32,667= :30961(0x7fce36077700):ZOO_INFO@check_events@1775: session establishment co= mplete on server [172.31.19.240:2181], sessionId=3D0x1566a66ab9b000a, negot= iated timeout=3D10000 Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.667783 3= 1000 group.cpp:349] Group process (group(1)@172.31.23.17:5051) connected to= ZooKeeper Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.667835 3= 1000 group.cpp:837] Syncing group operations: queue size (joins, cancels, d= atas) =3D (0, 0, 0) Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.667857 3= 1000 group.cpp:427] Trying to create path '/mesos' in ZooKeeper Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.668104 3= 0994 metadata_manager.cpp:251] Successfully loaded 1 Docker images Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.668200 3= 0993 provisioner.cpp:253] Provisioner recovery complete Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.671417 3= 1000 detector.cpp:152] Detected a new leader: (id=3D'64') Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.671563 3= 0998 group.cpp:706] Trying to get '/mesos/json.info_0000000064' in ZooKeepe= r Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.672611 3= 0996 zookeeper.cpp:259] A new leading master (UPID=3Dmaster@172.31.19.240:5= 050) is detected Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.761471 3= 0998 slave.cpp:4782] Finished recovery Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.761726 3= 0998 slave.cpp:4815] Garbage collecting old agent 524105e7-de7a-43f0-8b28-d= 3ff3e0c4a44-S2 Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.761812 3= 0998 slave.cpp:4815] Garbage collecting old agent c9852a23-bc07-422d-8d69-2= 3c167a1924d-S0 Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.761808 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3= e0c4a44-S2' for gc 6.99999118329778days in the future Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.761893 3= 0998 slave.cpp:4815] Garbage collecting old agent 524105e7-de7a-43f0-8b28-d= 3ff3e0c4a44-S1 Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.761904 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-= d3ff3e0c4a44-S2' for gc 6.99999118289481days in the future Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.761978 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c16= 7a1924d-S0' for gc 6.99999118225778days in the future Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.761988 3= 0998 slave.cpp:4815] Garbage collecting old agent 524105e7-de7a-43f0-8b28-d= 3ff3e0c4a44-S3 Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.762029 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/c9852a23-bc07-422d-8d69-= 23c167a1924d-S0' for gc 6.9999911819437days in the future Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.762064 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3= e0c4a44-S1' for gc 6.99999118122667days in the future Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.762105 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-= d3ff3e0c4a44-S1' for gc 6.9999911809037days in the future Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.762168 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3= e0c4a44-S3' for gc 6.9999911798963days in the future Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.762207 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-= d3ff3e0c4a44-S3' for gc 6.99999117958222days in the future Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.762230 3= 0998 slave.cpp:895] New master detected at master@172.31.19.240:5050 Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.762212 3= 0993 status_update_manager.cpp:174] Pausing sending status updates Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.762277 3= 0998 slave.cpp:916] No credentials provided. Attempting to register without= authentication Aug 8 15:48:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:32.762344 3= 0998 slave.cpp:927] Detecting new master Aug 8 15:48:33 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:33.067215 3= 0999 slave.cpp:1197] Re-registered with master master@172.31.19.240:5050 Aug 8 15:48:33 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:33.067312 3= 0999 slave.cpp:1233] Forwarding total oversubscribed resources Aug 8 15:48:33 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:33.067313 3= 0993 status_update_manager.cpp:181] Resuming sending status updates Aug 8 15:48:33 ip-172-31-23-17 mesos-slave[30961]: I0808 15:48:33.067610 3= 1000 slave.cpp:2526] Updated checkpointed resources from to Aug 8 15:49:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:32.507480 3= 1000 slave.cpp:1495] Got assigned task 2 for framework c9852a23-bc07-422d-8= d69-23c167a1924d-0001 Aug 8 15:49:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:32.508069 3= 1000 gc.cpp:83] Unscheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c= 167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0001' from gc Aug 8 15:49:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:32.508167 3= 0998 slave.cpp:1614] Launching task 2 for framework c9852a23-bc07-422d-8d69= -23c167a1924d-0001 Aug 8 15:49:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:32.509050 3= 0998 slave.cpp:5674] Launching executor 2 of framework c9852a23-bc07-422d-8= d69-23c167a1924d-0001 with resources cpus(*):0.1; mem(*):32 in work directo= ry '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9= 852a23-bc07-422d-8d69-23c167a1924d-0001/executors/2/runs/a58a1b2c-14a4-4ada= -a4b3-666d8d077597' Aug 8 15:49:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:32.509269 3= 0998 slave.cpp:1840] Queuing task '2' for executor '2' of framework c9852a2= 3-bc07-422d-8d69-23c167a1924d-0001 Aug 8 15:49:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:32.509430 3= 0993 containerizer.cpp:781] Starting container 'a58a1b2c-14a4-4ada-a4b3-666= d8d077597' for executor '2' of framework 'c9852a23-bc07-422d-8d69-23c167a19= 24d-0001' Aug 8 15:49:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:32.511643 3= 0995 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner= /containers/a58a1b2c-14a4-4ada-a4b3-666d8d077597/backends/copy/rootfses/a68= 8d80b-3eb9-4301-abfc-3e6a742cc8be' for container a58a1b2c-14a4-4ada-a4b3-66= 6d8d077597 Aug 8 15:49:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:32.663204 3= 0996 slave.cpp:4591] Current disk usage 17.44%. Max allowed age: 5.07895572= 3221944days Aug 8 15:49:38 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:38.511801 3= 0995 slave.cpp:1495] Got assigned task 3 for framework c9852a23-bc07-422d-8= d69-23c167a1924d-0001 Aug 8 15:49:38 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:38.512176 3= 0995 slave.cpp:1614] Launching task 3 for framework c9852a23-bc07-422d-8d69= -23c167a1924d-0001 Aug 8 15:49:38 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:38.512511 3= 0995 slave.cpp:5674] Launching executor 3 of framework c9852a23-bc07-422d-8= d69-23c167a1924d-0001 with resources cpus(*):0.1; mem(*):32 in work directo= ry '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9= 852a23-bc07-422d-8d69-23c167a1924d-0001/executors/3/runs/eb20f642-bb90-4293= -8eec-6f1576ccaeb1' Aug 8 15:49:38 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:38.512737 3= 0995 slave.cpp:1840] Queuing task '3' for executor '3' of framework c9852a2= 3-bc07-422d-8d69-23c167a1924d-0001 Aug 8 15:49:38 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:38.512948 3= 0998 containerizer.cpp:781] Starting container 'eb20f642-bb90-4293-8eec-6f1= 576ccaeb1' for executor '3' of framework 'c9852a23-bc07-422d-8d69-23c167a19= 24d-0001' Aug 8 15:49:38 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:38.514993 3= 0996 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner= /containers/eb20f642-bb90-4293-8eec-6f1576ccaeb1/backends/copy/rootfses/8d2= e66c8-f0d6-4891-86e3-7e5c222adfd2' for container eb20f642-bb90-4293-8eec-6f= 1576ccaeb1 Aug 8 15:49:44 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:44.524807 3= 1000 slave.cpp:1495] Got assigned task 4 for framework c9852a23-bc07-422d-8= d69-23c167a1924d-0001 Aug 8 15:49:44 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:44.525184 3= 1000 slave.cpp:1614] Launching task 4 for framework c9852a23-bc07-422d-8d69= -23c167a1924d-0001 Aug 8 15:49:44 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:44.525995 3= 1000 slave.cpp:5674] Launching executor 4 of framework c9852a23-bc07-422d-8= d69-23c167a1924d-0001 with resources cpus(*):0.1; mem(*):32 in work directo= ry '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9= 852a23-bc07-422d-8d69-23c167a1924d-0001/executors/4/runs/d248d565-e9d1-438e= -8524-d71f601ba981' Aug 8 15:49:44 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:44.526221 3= 1000 slave.cpp:1840] Queuing task '4' for executor '4' of framework c9852a2= 3-bc07-422d-8d69-23c167a1924d-0001 Aug 8 15:49:44 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:44.526448 3= 0995 containerizer.cpp:781] Starting container 'd248d565-e9d1-438e-8524-d71= f601ba981' for executor '4' of framework 'c9852a23-bc07-422d-8d69-23c167a19= 24d-0001' Aug 8 15:49:44 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:44.528301 3= 0998 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner= /containers/d248d565-e9d1-438e-8524-d71f601ba981/backends/copy/rootfses/d05= cefcb-6f83-49b6-a1c1-8e53d463712c' for container d248d565-e9d1-438e-8524-d7= 1f601ba981 Aug 8 15:49:50 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:50.531419 3= 0996 slave.cpp:1495] Got assigned task 5 for framework c9852a23-bc07-422d-8= d69-23c167a1924d-0001 Aug 8 15:49:50 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:50.531810 3= 0996 slave.cpp:1614] Launching task 5 for framework c9852a23-bc07-422d-8d69= -23c167a1924d-0001 Aug 8 15:49:50 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:50.532166 3= 0996 slave.cpp:5674] Launching executor 5 of framework c9852a23-bc07-422d-8= d69-23c167a1924d-0001 with resources cpus(*):0.1; mem(*):32 in work directo= ry '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9= 852a23-bc07-422d-8d69-23c167a1924d-0001/executors/5/runs/2f372338-3c40-4463= -b61f-4f7e9e42766a' Aug 8 15:49:50 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:50.532412 3= 0996 slave.cpp:1840] Queuing task '5' for executor '5' of framework c9852a2= 3-bc07-422d-8d69-23c167a1924d-0001 Aug 8 15:49:50 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:50.532740 3= 0998 containerizer.cpp:781] Starting container '2f372338-3c40-4463-b61f-4f7= e9e42766a' for executor '5' of framework 'c9852a23-bc07-422d-8d69-23c167a19= 24d-0001' Aug 8 15:49:50 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:50.534970 3= 0993 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner= /containers/2f372338-3c40-4463-b61f-4f7e9e42766a/backends/copy/rootfses/691= d6235-1596-4a5d-aeef-21bf43b2cc03' for container 2f372338-3c40-4463-b61f-4f= 7e9e42766a Aug 8 15:49:56 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:56.540551 3= 0999 slave.cpp:1495] Got assigned task 6 for framework c9852a23-bc07-422d-8= d69-23c167a1924d-0001 Aug 8 15:49:56 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:56.540967 3= 0999 slave.cpp:1614] Launching task 6 for framework c9852a23-bc07-422d-8d69= -23c167a1924d-0001 Aug 8 15:49:56 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:56.541805 3= 0999 slave.cpp:5674] Launching executor 6 of framework c9852a23-bc07-422d-8= d69-23c167a1924d-0001 with resources cpus(*):0.1; mem(*):32 in work directo= ry '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9= 852a23-bc07-422d-8d69-23c167a1924d-0001/executors/6/runs/b77f9aa9-b2e5-488d= -9fdd-fcbd2e5c8455' Aug 8 15:49:56 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:56.542049 3= 0999 slave.cpp:1840] Queuing task '6' for executor '6' of framework c9852a2= 3-bc07-422d-8d69-23c167a1924d-0001 Aug 8 15:49:56 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:56.542263 3= 0996 containerizer.cpp:781] Starting container 'b77f9aa9-b2e5-488d-9fdd-fcb= d2e5c8455' for executor '6' of framework 'c9852a23-bc07-422d-8d69-23c167a19= 24d-0001' Aug 8 15:49:56 ip-172-31-23-17 mesos-slave[30961]: I0808 15:49:56.543998 3= 0996 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisioner= /containers/b77f9aa9-b2e5-488d-9fdd-fcbd2e5c8455/backends/copy/rootfses/360= 1630b-5b57-4a45-9e2c-cfa79eacf2fd' for container b77f9aa9-b2e5-488d-9fdd-fc= bd2e5c8455 Aug 8 15:50:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:50:32.664178 3= 0995 slave.cpp:4591] Current disk usage 31.17%. Max allowed age: 4.11797297= 3602755days Aug 8 15:51:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:51:32.665262 3= 0997 slave.cpp:4591] Current disk usage 41.95%. Max allowed age: 3.36366928= 1005290days Aug 8 15:52:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:52:32.510210 3= 0999 slave.cpp:4539] Terminating executor ''2' of framework c9852a23-bc07-4= 22d-8d69-23c167a1924d-0001' because it did not register within 3mins Aug 8 15:52:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:52:32.510499 3= 1000 containerizer.cpp:1622] Destroying container 'a58a1b2c-14a4-4ada-a4b3-= 666d8d077597' Aug 8 15:52:32 ip-172-31-23-17 mesos-slave[30961]: I0808 15:52:32.665740 3= 0994 slave.cpp:4591] Current disk usage 52.57%. Max allowed age: 2.62012071= 2155822days Aug 8 15:52:38 ip-172-31-23-17 mesos-slave[30961]: I0808 15:52:38.512807 3= 0995 slave.cpp:4539] Terminating executor ''3' of framework c9852a23-bc07-4= 22d-8d69-23c167a1924d-0001' because it did not register within 3mins Aug 8 15:52:38 ip-172-31-23-17 mesos-slave[30961]: I0808 15:52:38.513015 3= 0994 containerizer.cpp:1622] Destroying container 'eb20f642-bb90-4293-8eec-= 6f1576ccaeb1' Aug 8 15:52:44 ip-172-31-23-17 mesos-slave[30961]: I0808 15:52:44.526844 3= 0996 slave.cpp:4539] Terminating executor ''4' of framework c9852a23-bc07-4= 22d-8d69-23c167a1924d-0001' because it did not register within 3mins Aug 8 15:52:44 ip-172-31-23-17 mesos-slave[30961]: I0808 15:52:44.527053 3= 0996 containerizer.cpp:1622] Destroying container 'd248d565-e9d1-438e-8524-= d71f601ba981' Aug 8 15:52:50 ip-172-31-23-17 mesos-slave[30961]: I0808 15:52:50.532806 3= 1000 slave.cpp:4539] Terminating executor ''5' of framework c9852a23-bc07-4= 22d-8d69-23c167a1924d-0001' because it did not register within 3mins Aug 8 15:52:50 ip-172-31-23-17 mesos-slave[30961]: I0808 15:52:50.532981 3= 1000 containerizer.cpp:1622] Destroying container '2f372338-3c40-4463-b61f-= 4f7e9e42766a' Aug 8 15:52:56 ip-172-31-23-17 mesos-slave[30961]: I0808 15:52:56.542345 3= 0995 slave.cpp:4539] Terminating executor ''6' of framework c9852a23-bc07-4= 22d-8d69-23c167a1924d-0001' because it did not register within 3mins Aug 8 15:52:56 ip-172-31-23-17 mesos-slave[30961]: I0808 15:52:56.542580 3= 0998 containerizer.cpp:1622] Destroying container 'b77f9aa9-b2e5-488d-9fdd-= fcbd2e5c8455' Aug 8 15:53:09 ip-172-31-23-17 mesos-slave[30961]: E0808 15:53:09.691315 3= 0996 slave.cpp:3976] Container 'eb20f642-bb90-4293-8eec-6f1576ccaeb1' for e= xecutor '3' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0001 failed t= o start: Container is being destroyed during provisioning Aug 8 15:53:09 ip-172-31-23-17 mesos-slave[30961]: W0808 15:53:09.695711 3= 0996 composing.cpp:541] Container 'eb20f642-bb90-4293-8eec-6f1576ccaeb1' is= already destroyed Aug 8 15:53:11 ip-172-31-23-17 mesos-slave[30961]: E0808 15:53:11.270226 3= 0993 slave.cpp:3976] Container 'a58a1b2c-14a4-4ada-a4b3-666d8d077597' for e= xecutor '2' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0001 failed t= o start: Container is being destroyed during provisioning Aug 8 15:53:11 ip-172-31-23-17 mesos-slave[30961]: W0808 15:53:11.270447 3= 0999 composing.cpp:541] Container 'a58a1b2c-14a4-4ada-a4b3-666d8d077597' is= already destroyed Aug 8 15:53:11 ip-172-31-23-17 mesos-slave[30961]: E0808 15:53:11.689219 3= 0995 slave.cpp:3976] Container 'd248d565-e9d1-438e-8524-d71f601ba981' for e= xecutor '4' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0001 failed t= o start: Container is being destroyed during provisioning Aug 8 15:53:11 ip-172-31-23-17 mesos-slave[30961]: W0808 15:53:11.689400 3= 1000 composing.cpp:541] Container 'd248d565-e9d1-438e-8524-d71f601ba981' is= already destroyed Aug 8 15:53:12 ip-172-31-23-17 mesos-slave[30961]: E0808 15:53:12.400029 3= 0999 slave.cpp:3976] Container '2f372338-3c40-4463-b61f-4f7e9e42766a' for e= xecutor '5' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0001 failed t= o start: Container is being destroyed during provisioning Aug 8 15:53:12 ip-172-31-23-17 mesos-slave[30961]: W0808 15:53:12.400218 3= 0994 composing.cpp:541] Container '2f372338-3c40-4463-b61f-4f7e9e42766a' is= already destroyed Aug 8 15:53:12 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:12.799772 3= 0998 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provis= ioner/containers/eb20f642-bb90-4293-8eec-6f1576ccaeb1/backends/copy/rootfse= s/8d2e66c8-f0d6-4891-86e3-7e5c222adfd2' for container eb20f642-bb90-4293-8e= ec-6f1576ccaeb1 Aug 8 15:53:12 ip-172-31-23-17 mesos-slave[30961]: E0808 15:53:12.799855 3= 0995 slave.cpp:3976] Container 'b77f9aa9-b2e5-488d-9fdd-fcbd2e5c8455' for e= xecutor '6' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0001 failed t= o start: Container is being destroyed during provisioning Aug 8 15:53:12 ip-172-31-23-17 mesos-slave[30961]: W0808 15:53:12.810742 3= 0999 composing.cpp:541] Container 'b77f9aa9-b2e5-488d-9fdd-fcbd2e5c8455' is= already destroyed Aug 8 15:53:12 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:12.811067 3= 0998 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provis= ioner/containers/a58a1b2c-14a4-4ada-a4b3-666d8d077597/backends/copy/rootfse= s/a688d80b-3eb9-4301-abfc-3e6a742cc8be' for container a58a1b2c-14a4-4ada-a4= b3-666d8d077597 Aug 8 15:53:12 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:12.812954 3= 0998 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provis= ioner/containers/d248d565-e9d1-438e-8524-d71f601ba981/backends/copy/rootfse= s/d05cefcb-6f83-49b6-a1c1-8e53d463712c' for container d248d565-e9d1-438e-85= 24-d71f601ba981 Aug 8 15:53:12 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:12.814383 3= 0998 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provis= ioner/containers/2f372338-3c40-4463-b61f-4f7e9e42766a/backends/copy/rootfse= s/691d6235-1596-4a5d-aeef-21bf43b2cc03' for container 2f372338-3c40-4463-b6= 1f-4f7e9e42766a Aug 8 15:53:12 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:12.815577 3= 0998 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provis= ioner/containers/b77f9aa9-b2e5-488d-9fdd-fcbd2e5c8455/backends/copy/rootfse= s/3601630b-5b57-4a45-9e2c-cfa79eacf2fd' for container b77f9aa9-b2e5-488d-9f= dd-fcbd2e5c8455 Aug 8 15:53:15 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:15.995162 3= 1000 slave.cpp:4082] Executor '4' of framework c9852a23-bc07-422d-8d69-23c1= 67a1924d-0001 has terminated with unknown status Aug 8 15:53:15 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:15.995355 3= 1000 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 4f428b5a-153= 8-4f78-8383-6c9b68f67f3f) for task 4 of framework c9852a23-bc07-422d-8d69-2= 3c167a1924d-0001 from @0.0.0.0:0 Aug 8 15:53:15 ip-172-31-23-17 mesos-slave[30961]: W0808 15:53:15.995841 3= 1000 containerizer.cpp:1451] Ignoring update for unknown container: d248d56= 5-e9d1-438e-8524-d71f601ba981 Aug 8 15:53:15 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:15.996011 3= 1000 status_update_manager.cpp:320] Received status update TASK_FAILED (UUI= D: 4f428b5a-1538-4f78-8383-6c9b68f67f3f) for task 4 of framework c9852a23-b= c07-422d-8d69-23c167a1924d-0001 Aug 8 15:53:15 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:15.996346 3= 1000 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 4f428b5a-1538= -4f78-8383-6c9b68f67f3f) for task 4 of framework c9852a23-bc07-422d-8d69-23= c167a1924d-0001 to master@172.31.19.240:5050 Aug 8 15:53:15 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:15.998983 3= 0993 status_update_manager.cpp:392] Received status update acknowledgement = (UUID: 4f428b5a-1538-4f78-8383-6c9b68f67f3f) for task 4 of framework c9852a= 23-bc07-422d-8d69-23c167a1924d-0001 Aug 8 15:53:15 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:15.999192 3= 0993 slave.cpp:4193] Cleaning up executor '4' of framework c9852a23-bc07-42= 2d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.000690 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c16= 7a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0001/executors/4= /runs/d248d565-e9d1-438e-8524-d71f601ba981' for gc 6.9999884190637days in t= he future Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.000754 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c16= 7a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0001/executors/4= ' for gc 6.99998841775407days in the future Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.344812 3= 0995 slave.cpp:4082] Executor '3' of framework c9852a23-bc07-422d-8d69-23c1= 67a1924d-0001 has terminated with unknown status Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.344981 3= 0995 slave.cpp:3211] Handling status update TASK_FAILED (UUID: c33d9357-e74= 9-4e05-82d2-cb305d1ff0d2) for task 3 of framework c9852a23-bc07-422d-8d69-2= 3c167a1924d-0001 from @0.0.0.0:0 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: W0808 15:53:16.346043 3= 0994 containerizer.cpp:1451] Ignoring update for unknown container: eb20f64= 2-bb90-4293-8eec-6f1576ccaeb1 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.346206 3= 0998 status_update_manager.cpp:320] Received status update TASK_FAILED (UUI= D: c33d9357-e749-4e05-82d2-cb305d1ff0d2) for task 3 of framework c9852a23-b= c07-422d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.346482 3= 0997 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: c33d9357-e749= -4e05-82d2-cb305d1ff0d2) for task 3 of framework c9852a23-bc07-422d-8d69-23= c167a1924d-0001 to master@172.31.19.240:5050 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.348764 3= 0998 status_update_manager.cpp:392] Received status update acknowledgement = (UUID: c33d9357-e749-4e05-82d2-cb305d1ff0d2) for task 3 of framework c9852a= 23-bc07-422d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.348863 3= 0998 slave.cpp:4193] Cleaning up executor '3' of framework c9852a23-bc07-42= 2d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.348984 3= 0998 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c16= 7a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0001/executors/3= /runs/eb20f642-bb90-4293-8eec-6f1576ccaeb1' for gc 6.99999596136593days in = the future Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.349020 3= 0998 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c16= 7a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0001/executors/3= ' for gc 6.99999596097185days in the future Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.498633 3= 0996 slave.cpp:4082] Executor '2' of framework c9852a23-bc07-422d-8d69-23c1= 67a1924d-0001 has terminated with unknown status Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.498803 3= 0996 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 3d818e6a-f60= 0-412c-b07d-f21794a0af30) for task 2 of framework c9852a23-bc07-422d-8d69-2= 3c167a1924d-0001 from @0.0.0.0:0 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.498921 3= 0996 slave.cpp:4082] Executor '5' of framework c9852a23-bc07-422d-8d69-23c1= 67a1924d-0001 has terminated with unknown status Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.498991 3= 0996 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 16048628-611= 2-42ed-b3db-5b418a29d40d) for task 5 of framework c9852a23-bc07-422d-8d69-2= 3c167a1924d-0001 from @0.0.0.0:0 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: W0808 15:53:16.499202 3= 0996 containerizer.cpp:1451] Ignoring update for unknown container: a58a1b2= c-14a4-4ada-a4b3-666d8d077597 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: W0808 15:53:16.499289 3= 0996 containerizer.cpp:1451] Ignoring update for unknown container: 2f37233= 8-3c40-4463-b61f-4f7e9e42766a Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.499400 3= 0997 status_update_manager.cpp:320] Received status update TASK_FAILED (UUI= D: 3d818e6a-f600-412c-b07d-f21794a0af30) for task 2 of framework c9852a23-b= c07-422d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.499665 3= 0995 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 3d818e6a-f600= -412c-b07d-f21794a0af30) for task 2 of framework c9852a23-bc07-422d-8d69-23= c167a1924d-0001 to master@172.31.19.240:5050 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.499678 3= 0997 status_update_manager.cpp:320] Received status update TASK_FAILED (UUI= D: 16048628-6112-42ed-b3db-5b418a29d40d) for task 5 of framework c9852a23-b= c07-422d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.499928 3= 0997 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 16048628-6112= -42ed-b3db-5b418a29d40d) for task 5 of framework c9852a23-bc07-422d-8d69-23= c167a1924d-0001 to master@172.31.19.240:5050 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.504577 3= 0994 status_update_manager.cpp:392] Received status update acknowledgement = (UUID: 3d818e6a-f600-412c-b07d-f21794a0af30) for task 2 of framework c9852a= 23-bc07-422d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.504712 3= 0994 slave.cpp:4193] Cleaning up executor '2' of framework c9852a23-bc07-42= 2d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.505421 3= 0999 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c16= 7a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0001/executors/2= /runs/a58a1b2c-14a4-4ada-a4b3-666d8d077597' for gc 6.99999415104296days in = the future Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.505486 3= 0994 status_update_manager.cpp:392] Received status update acknowledgement = (UUID: 16048628-6112-42ed-b3db-5b418a29d40d) for task 5 of framework c9852a= 23-bc07-422d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.505493 3= 0999 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c16= 7a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0001/executors/2= ' for gc 6.99999415021926days in the future Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.505623 3= 0994 slave.cpp:4193] Cleaning up executor '5' of framework c9852a23-bc07-42= 2d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.505748 3= 0993 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c16= 7a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0001/executors/5= /runs/2f372338-3c40-4463-b61f-4f7e9e42766a' for gc 6.99999414686222days in = the future Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.505798 3= 0993 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c16= 7a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0001/executors/5= ' for gc 6.9999941464237days in the future Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.598873 3= 0999 slave.cpp:4082] Executor '6' of framework c9852a23-bc07-422d-8d69-23c1= 67a1924d-0001 has terminated with unknown status Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.599030 3= 0999 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 62805972-05e= 7-419f-8ff3-f80a88cf8199) for task 6 of framework c9852a23-bc07-422d-8d69-2= 3c167a1924d-0001 from @0.0.0.0:0 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: W0808 15:53:16.599324 3= 0996 containerizer.cpp:1451] Ignoring update for unknown container: b77f9aa= 9-b2e5-488d-9fdd-fcbd2e5c8455 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.599454 3= 0996 status_update_manager.cpp:320] Received status update TASK_FAILED (UUI= D: 62805972-05e7-419f-8ff3-f80a88cf8199) for task 6 of framework c9852a23-b= c07-422d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.599701 3= 0993 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 62805972-05e7= -419f-8ff3-f80a88cf8199) for task 6 of framework c9852a23-bc07-422d-8d69-23= c167a1924d-0001 to master@172.31.19.240:5050 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.602051 3= 0996 status_update_manager.cpp:392] Received status update acknowledgement = (UUID: 62805972-05e7-419f-8ff3-f80a88cf8199) for task 6 of framework c9852a= 23-bc07-422d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.602171 3= 0996 slave.cpp:4193] Cleaning up executor '6' of framework c9852a23-bc07-42= 2d-8d69-23c167a1924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.602313 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c16= 7a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0001/executors/6= /runs/b77f9aa9-b2e5-488d-9fdd-fcbd2e5c8455' for gc 6.99999302943407days in = the future Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.602334 3= 0996 slave.cpp:4281] Cleaning up framework c9852a23-bc07-422d-8d69-23c167a1= 924d-0001 Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.602388 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c16= 7a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0001/executors/6= ' for gc 6.99999302868148days in the future Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.602442 3= 1000 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c16= 7a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0001' for gc 6.9= 9999302741926days in the future Aug 8 15:53:16 ip-172-31-23-17 mesos-slave[30961]: I0808 15:53:16.602445 3= 0999 status_update_manager.cpp:282] Closing status update streams for frame= work c9852a23-bc07-422d-8d69-23c167a1924d-0001 {noformat} > Tasks fail when provisioning multiple containers with large docker image= s using copy backend > -------------------------------------------------------------------------= -------------------- > > Key: MESOS-6004 > URL: https://issues.apache.org/jira/browse/MESOS-6004 > Project: Mesos > Issue Type: Bug > Components: containerization, docker > Affects Versions: 0.28.2, 1.0.0 > Environment: h4. Agent Platform > - Ubuntu 16.04 > - AWS g2.x2large instance > - Nvidia support enabled > h4. Agent Configuration > -{noformat} > --containerizers=3Dmesos,docker > --docker_config=3D > --docker_store_dir=3D/mnt/mesos/store/docker > --executor_registration_timeout=3D3mins > --hostname=3D > --image_providers=3Ddocker > --image_provisioner_backend=3Dcopy > --isolation=3Dfilesystem/linux,docker/runtime,cgroups/devices,gpu/nvidia > --switch_user=3Dfalse > --work_dir=3D/mnt/mesos > {noformat} > h4. Framework > - custom framework written in python > - using unified containerizer with docker images > h4. Test Setup > * 1 master > * 1 agent > * 5 tasks scheduled at the same time: > ** resources: cpus: 0.1, mem: 128 > ** command: `echo test` > ** docker image: custom docker image, based on nvidia/cuda ~5gb > ** the same docker image was for all tasks, already pulled. > Reporter: Michael Thomas > Labels: containerizer, docker, performance > > When scheduling more than one task on the same agent, all tasks fail a as= containers seem to be destroyed during provisioning. > Specifically, the errors on the agent logs are: > {noformat} > E0808 15:53:09.691315 30996 slave.cpp:3976] Container 'eb20f642-bb90-429= 3-8eec-6f1576ccaeb1' for executor '3' of framework c9852a23-bc07-422d-8d69-= 23c167a1924d-0001 failed to start: Container is being destroyed during prov= isioning > {noformat} > and=20 > {noformat} > I0808 15:52:32.510210 30999 slave.cpp:4539] Terminating executor ''2' of = framework c9852a23-bc07-422d-8d69-23c167a1924d-0001' because it did not reg= ister within 3mins > {noformat} > As the default provisioning method {{copy}} is being used, I assume this = is due to the provisioning of multiple containers taking too long and the a= gent will not wait. For large images, this method is simply not performant. > The issue did not occur, when only one tasks was scheduled. > Increasing the {{executor_registration_timeout}} parameter, seemed to hel= p a bit as it allowed to schedule at least 2 tasks at the same time. But st= ill fails with more (5 in this case) > h4. Complete logs > (with GLOG_v=3D1) > {noformat} > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.800375 = 3738 slave.cpp:198] Agent started on 1)@172.31.23.17:5051 > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.800403 = 3738 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix= =3D"http://" --appc_store_dir=3D"/tmp/mesos/store/appc" --authenticate_http= _readonly=3D"false" --authenticate_http_readwrite=3D"false" --authenticatee= =3D"crammd5" --authentication_backoff_factor=3D"1secs" --authorizer=3D"loca= l" --cgroups_cpu_enable_pids_and_tids_count=3D"false" --cgroups_enable_cfs= =3D"false" --cgroups_hierarchy=3D"/sys/fs/cgroup" --cgroups_limit_swap=3D"f= alse" --cgroups_root=3D"mesos" --container_disk_watch_interval=3D"15secs" -= -containerizers=3D"mesos,docker" --default_role=3D"*" --disk_watch_interval= =3D"1mins" --docker=3D"docker" --docker_config=3D"{"auths":{"https:\/\/inde= x.docker.io\/v1\/":{"auth":"dGVycmFsb3VwZTpUYWxFWUFOSXR5","email":"sebastia= n.gerke@terraloupe.com"}}}" --docker_kill_orphans=3D"true" --docker_registr= y=3D"https://registry-1.docker.io" --docker_remove_delay=3D"6hrs" --docker_= socket=3D"/var/run/docker.sock" --docker_stop_timeout=3D"0ns" --docker_stor= e_dir=3D"/mnt/t" --docker_volume_checkp > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: oint_dir=3D"/var/run/m= esos/isolators/docker/volume" --enforce_container_disk_quota=3D"false" --ex= ecutor_registration_timeout=3D"1mins" --executor_shutdown_grace_period=3D"5= secs" --fetcher_cache_dir=3D"/tmp/mesos/fetch" --fetcher_cache_size=3D"2GB"= --frameworks_home=3D"" --gc_delay=3D"1weeks" --gc_disk_headroom=3D"0.1" --= hadoop_home=3D"" --help=3D"false" --hostname=3D"ec2-52-59-113-0.eu-central-= 1.compute.amazonaws.com" --hostname_lookup=3D"true" --http_authenticators= =3D"basic" --http_command_executor=3D"false" --image_providers=3D"docker" -= -image_provisioner_backend=3D"copy" --initialize_driver_logging=3D"true" --= isolation=3D"filesystem/linux,docker/runtime,cgroups/devices,gpu/nvidia" --= launcher_dir=3D"/usr/libexec/mesos" --log_dir=3D"/var/log/mesos" --logbufse= cs=3D"0" --logging_level=3D"INFO" --master=3D"zk://172.31.19.240:2181/mesos= " --oversubscribed_resources_interval=3D"15secs" --perf_duration=3D"10secs"= --perf_interval=3D"1mins" --port=3D"5051" --qos_correction_interval_min=3D= "0ns" --quiet=3D"false" --recover=3D"reconnect" --re > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: covery_timeout=3D"15mi= ns" --registration_backoff_factor=3D"1secs" --revocable_cpu_low_priority=3D= "true" --sandbox_directory=3D"/mnt/mesos/sandbox" --strict=3D"true" --switc= h_user=3D"false" --systemd_enable_support=3D"true" --systemd_runtime_direct= ory=3D"/run/systemd/system" --version=3D"false" --work_dir=3D"/mnt/mesos" > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: 2016-08-09 10:11:41,80= 0:3738(0x7f71172a1700):ZOO_INFO@check_events@1728: initiated connection to = server [172.31.19.240:2181] > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801007 = 3738 slave.cpp:519] Agent resources: gpus(*):1; cpus(*):8; mem(*):14014; d= isk(*):60257; ports(*):[31000-32000] > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801059 = 3738 slave.cpp:527] Agent attributes: [ ] > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801077 = 3738 slave.cpp:532] Agent hostname: ec2-52-59-113-0.eu-central-1.compute.a= mazonaws.com > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.802232 = 3772 state.cpp:57] Recovering state from '/mnt/mesos/meta' > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803539 = 3774 status_update_manager.cpp:200] Recovering status update manager > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803745 = 3772 containerizer.cpp:522] Recovering containerizer > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803788 = 3770 docker.cpp:775] Recovering Docker containers > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.808550 = 3777 metadata_manager.cpp:251] Successfully loaded 2 Docker images > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.808688 = 3771 provisioner.cpp:253] Provisioner recovery complete > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: 2016-08-09 10:11:41,81= 6:3738(0x7f71172a1700):ZOO_INFO@check_events@1775: session establishment co= mplete on server [172.31.19.240:2181], sessionId=3D0x1566a66ab9b0010, negot= iated timeout=3D10000 > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817003 = 3774 group.cpp:349] Group process (group(1)@172.31.23.17:5051) connected t= o ZooKeeper > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817116 = 3774 group.cpp:837] Syncing group operations: queue size (joins, cancels, = datas) =3D (0, 0, 0) > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817147 = 3774 group.cpp:427] Trying to create path '/mesos' in ZooKeeper > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.818857 = 3774 detector.cpp:152] Detected a new leader: (id=3D'64') > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.819005 = 3776 group.cpp:706] Trying to get '/mesos/json.info_0000000064' in ZooKeep= er > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.819855 = 3775 zookeeper.cpp:259] A new leading master (UPID=3Dmaster@172.31.19.240:= 5050) is detected > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.899876 = 3777 slave.cpp:4782] Finished recovery > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900573 = 3777 slave.cpp:4815] Garbage collecting old agent 524105e7-de7a-43f0-8b28-= d3ff3e0c4a44-S2 > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900687 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff= 3e0c4a44-S2' for gc 6.99998957623704days in the future > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900729 = 3777 slave.cpp:4815] Garbage collecting old agent c9852a23-bc07-422d-8d69-= 23c167a1924d-S0 > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900779 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28= -d3ff3e0c4a44-S2' for gc 6.99998957520889days in the future > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900817 = 3777 slave.cpp:4815] Garbage collecting old agent 524105e7-de7a-43f0-8b28-= d3ff3e0c4a44-S1 > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900825 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c1= 67a1924d-S0' for gc 6.99998957451259days in the future > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900899 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/c9852a23-bc07-422d-8d69= -23c167a1924d-S0' for gc 6.99998957412148days in the future > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900933 = 3777 slave.cpp:4815] Garbage collecting old agent 524105e7-de7a-43f0-8b28-= d3ff3e0c4a44-S3 > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900944 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff= 3e0c4a44-S1' for gc 6.99998957332148days in the future > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901010 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28= -d3ff3e0c4a44-S1' for gc 6.99998957291556days in the future > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901051 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff= 3e0c4a44-S3' for gc 6.99998957196741days in the future > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901101 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28= -d3ff3e0c4a44-S3' for gc 6.99998957156148days in the future > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901201 = 3775 status_update_manager.cpp:174] Pausing sending status updates > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901203 = 3777 slave.cpp:895] New master detected at master@172.31.19.240:5050 > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901278 = 3777 slave.cpp:916] No credentials provided. Attempting to register withou= t authentication > Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901317 = 3777 slave.cpp:927] Detecting new master > Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267201 = 3774 slave.cpp:1197] Re-registered with master master@172.31.19.240:5050 > Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267302 = 3771 status_update_manager.cpp:181] Resuming sending status updates > Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267302 = 3774 slave.cpp:1233] Forwarding total oversubscribed resources > Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267577 = 3774 slave.cpp:2526] Updated checkpointed resources from to > Aug 9 10:11:48 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:48.269104 = 3777 slave.cpp:2218] Asked to shut down framework c9852a23-bc07-422d-8d69-= 23c167a1924d-0002 by master@172.31.19.240:5050 > Aug 9 10:11:48 ip-172-31-23-17 mesos-slave[3738]: W0809 10:11:48.269186 = 3777 slave.cpp:2233] Cannot shut down unknown framework c9852a23-bc07-422d= -8d69-23c167a1924d-0002 > Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.969521 = 3772 slave.cpp:1495] Got assigned task 0 for framework c9852a23-bc07-422d-= 8d69-23c167a1924d-0003 > Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.970026 = 3772 slave.cpp:1614] Launching task 0 for framework c9852a23-bc07-422d-8d6= 9-23c167a1924d-0003 > Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.970826 = 3772 slave.cpp:5674] Launching executor 0 of framework c9852a23-bc07-422d-= 8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work direct= ory '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c= 9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0/runs/09a77a80-f904-458= 1-966a-734d24ae96f3' > Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.971051 = 3772 slave.cpp:1840] Queuing task '0' for executor '0' of framework c9852a= 23-bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.971212 = 3773 containerizer.cpp:781] Starting container '09a77a80-f904-4581-966a-73= 4d24ae96f3' for executor '0' of framework 'c9852a23-bc07-422d-8d69-23c167a1= 924d-0003' > Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.973953 = 3774 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisione= r/containers/09a77a80-f904-4581-966a-734d24ae96f3/backends/copy/rootfses/ec= 9bb10e-6726-4872-be2b-4a72dfc0fafc' for container 09a77a80-f904-4581-966a-7= 34d24ae96f3 > Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978344 = 3774 slave.cpp:1495] Got assigned task 1 for framework c9852a23-bc07-422d-= 8d69-23c167a1924d-0003 > Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978687 = 3774 slave.cpp:1614] Launching task 1 for framework c9852a23-bc07-422d-8d6= 9-23c167a1924d-0003 > Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978976 = 3774 slave.cpp:5674] Launching executor 1 of framework c9852a23-bc07-422d-= 8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work direct= ory '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c= 9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1/runs/0cea1dc7-b523-4b6= 8-851a-531ed101de5a' > Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.979173 = 3774 slave.cpp:1840] Queuing task '1' for executor '1' of framework c9852a= 23-bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.979387 = 3777 containerizer.cpp:781] Starting container '0cea1dc7-b523-4b68-851a-53= 1ed101de5a' for executor '1' of framework 'c9852a23-bc07-422d-8d69-23c167a1= 924d-0003' > Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.981374 = 3776 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisione= r/containers/0cea1dc7-b523-4b68-851a-531ed101de5a/backends/copy/rootfses/e3= af89fe-c889-45fc-929b-f008b18e5d4e' for container 0cea1dc7-b523-4b68-851a-5= 31ed101de5a > Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.983610 = 3776 slave.cpp:1495] Got assigned task 2 for framework c9852a23-bc07-422d-= 8d69-23c167a1924d-0003 > Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.983947 = 3776 slave.cpp:1614] Launching task 2 for framework c9852a23-bc07-422d-8d6= 9-23c167a1924d-0003 > Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984236 = 3776 slave.cpp:5674] Launching executor 2 of framework c9852a23-bc07-422d-= 8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work direct= ory '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c= 9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2/runs/a4c972da-b12d-4b4= b-8481-79ca320ddab7' > Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984450 = 3776 slave.cpp:1840] Queuing task '2' for executor '2' of framework c9852a= 23-bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984778 = 3777 containerizer.cpp:781] Starting container 'a4c972da-b12d-4b4b-8481-79= ca320ddab7' for executor '2' of framework 'c9852a23-bc07-422d-8d69-23c167a1= 924d-0003' > Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.986330 = 3776 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisione= r/containers/a4c972da-b12d-4b4b-8481-79ca320ddab7/backends/copy/rootfses/fe= af70a4-df43-45c9-b6be-a654c9210a54' for container a4c972da-b12d-4b4b-8481-7= 9ca320ddab7 > Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.996104 = 3771 slave.cpp:1495] Got assigned task 3 for framework c9852a23-bc07-422d-= 8d69-23c167a1924d-0003 > Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.996462 = 3771 slave.cpp:1614] Launching task 3 for framework c9852a23-bc07-422d-8d6= 9-23c167a1924d-0003 > Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997241 = 3771 slave.cpp:5674] Launching executor 3 of framework c9852a23-bc07-422d-= 8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work direct= ory '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c= 9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3/runs/d9363f0d-645b-468= f-947a-8bf97d15a630' > Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997406 = 3771 slave.cpp:1840] Queuing task '3' for executor '3' of framework c9852a= 23-bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997640 = 3777 containerizer.cpp:781] Starting container 'd9363f0d-645b-468f-947a-8b= f97d15a630' for executor '3' of framework 'c9852a23-bc07-422d-8d69-23c167a1= 924d-0003' > Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.999593 = 3777 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisione= r/containers/d9363f0d-645b-468f-947a-8bf97d15a630/backends/copy/rootfses/ee= b1b066-94b8-4880-a696-090f830cf2d6' for container d9363f0d-645b-468f-947a-8= bf97d15a630 > Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.003924 = 3775 slave.cpp:1495] Got assigned task 4 for framework c9852a23-bc07-422d-= 8d69-23c167a1924d-0003 > Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.004278 = 3775 slave.cpp:1614] Launching task 4 for framework c9852a23-bc07-422d-8d6= 9-23c167a1924d-0003 > Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005383 = 3775 slave.cpp:5674] Launching executor 4 of framework c9852a23-bc07-422d-= 8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32 in work direct= ory '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c= 9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4/runs/58c398e2-38f9-438= 9-87ba-8696c9a523eb' > Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005681 = 3775 slave.cpp:1840] Queuing task '4' for executor '4' of framework c9852a= 23-bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005971 = 3772 containerizer.cpp:781] Starting container '58c398e2-38f9-4389-87ba-86= 96c9a523eb' for executor '4' of framework 'c9852a23-bc07-422d-8d69-23c167a1= 924d-0003' > Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.007786 = 3777 provisioner.cpp:294] Provisioning image rootfs '/mnt/mesos/provisione= r/containers/58c398e2-38f9-4389-87ba-8696c9a523eb/backends/copy/rootfses/78= b7bf3a-6f3b-4f8a-b4fa-98a6ea4bf8f5' for container 58c398e2-38f9-4389-87ba-8= 696c9a523eb > Aug 9 10:12:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:41.801560 = 3777 slave.cpp:4591] Current disk usage 43.28%. Max allowed age: 3.2700919= 66898553days > Aug 9 10:12:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:54.972057 = 3772 slave.cpp:4539] Terminating executor ''0' of framework c9852a23-bc07-= 422d-8d69-23c167a1924d-0003' because it did not register within 1mins > Aug 9 10:12:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:54.972322 = 3775 containerizer.cpp:1622] Destroying container '09a77a80-f904-4581-966a= -734d24ae96f3' > Aug 9 10:13:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:00.979907 = 3777 slave.cpp:4539] Terminating executor ''1' of framework c9852a23-bc07-= 422d-8d69-23c167a1924d-0003' because it did not register within 1mins > Aug 9 10:13:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:00.980058 = 3775 containerizer.cpp:1622] Destroying container '0cea1dc7-b523-4b68-851a= -531ed101de5a' > Aug 9 10:13:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:06.985003 = 3777 slave.cpp:4539] Terminating executor ''2' of framework c9852a23-bc07-= 422d-8d69-23c167a1924d-0003' because it did not register within 1mins > Aug 9 10:13:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:06.985255 = 3774 containerizer.cpp:1622] Destroying container 'a4c972da-b12d-4b4b-8481= -79ca320ddab7' > Aug 9 10:13:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:12.998438 = 3777 slave.cpp:4539] Terminating executor ''3' of framework c9852a23-bc07-= 422d-8d69-23c167a1924d-0003' because it did not register within 1mins > Aug 9 10:13:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:12.998641 = 3776 containerizer.cpp:1622] Destroying container 'd9363f0d-645b-468f-947a= -8bf97d15a630' > Aug 9 10:13:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:19.006058 = 3773 slave.cpp:4539] Terminating executor ''4' of framework c9852a23-bc07-= 422d-8d69-23c167a1924d-0003' because it did not register within 1mins > Aug 9 10:13:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:19.006228 = 3773 containerizer.cpp:1622] Destroying container '58c398e2-38f9-4389-87ba= -8696c9a523eb' > Aug 9 10:13:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:41.802209 = 3775 slave.cpp:4591] Current disk usage 57.59%. Max allowed age: 2.2690068= 05055764days > Aug 9 10:14:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:14:41.803022 = 3776 slave.cpp:4591] Current disk usage 68.70%. Max allowed age: 1.4912041= 18572176days > Aug 9 10:15:01 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:01.516207 = 3775 slave.cpp:3976] Container '58c398e2-38f9-4389-87ba-8696c9a523eb' for = executor '4' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed = to start: Container is being destroyed during provisioning > Aug 9 10:15:01 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:01.518277 = 3772 composing.cpp:541] Container '58c398e2-38f9-4389-87ba-8696c9a523eb' i= s already destroyed > Aug 9 10:15:02 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:02.291764 = 3775 slave.cpp:3976] Container '09a77a80-f904-4581-966a-734d24ae96f3' for = executor '0' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed = to start: Container is being destroyed during provisioning > Aug 9 10:15:02 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:02.291877 = 3775 composing.cpp:541] Container '09a77a80-f904-4581-966a-734d24ae96f3' i= s already destroyed > Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:03.022406 = 3774 slave.cpp:3976] Container 'd9363f0d-645b-468f-947a-8bf97d15a630' for = executor '3' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed = to start: Container is being destroyed during provisioning > Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:03.022595 = 3771 composing.cpp:541] Container 'd9363f0d-645b-468f-947a-8bf97d15a630' i= s already destroyed > Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:03.461519 = 3775 slave.cpp:3976] Container '0cea1dc7-b523-4b68-851a-531ed101de5a' for = executor '1' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed = to start: Container is being destroyed during provisioning > Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:03.461643 = 3774 composing.cpp:541] Container '0cea1dc7-b523-4b68-851a-531ed101de5a' i= s already destroyed > Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.190901 = 3776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provi= sioner/containers/58c398e2-38f9-4389-87ba-8696c9a523eb/backends/copy/rootfs= es/78b7bf3a-6f3b-4f8a-b4fa-98a6ea4bf8f5' for container 58c398e2-38f9-4389-8= 7ba-8696c9a523eb > Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:04.190904 = 3777 slave.cpp:3976] Container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' for = executor '2' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed = to start: Container is being destroyed during provisioning > Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:04.191128 = 3770 composing.cpp:541] Container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' i= s already destroyed > Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197221 = 3776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provi= sioner/containers/09a77a80-f904-4581-966a-734d24ae96f3/backends/copy/rootfs= es/ec9bb10e-6726-4872-be2b-4a72dfc0fafc' for container 09a77a80-f904-4581-9= 66a-734d24ae96f3 > Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197410 = 3776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provi= sioner/containers/d9363f0d-645b-468f-947a-8bf97d15a630/backends/copy/rootfs= es/eeb1b066-94b8-4880-a696-090f830cf2d6' for container d9363f0d-645b-468f-9= 47a-8bf97d15a630 > Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197510 = 3776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provi= sioner/containers/0cea1dc7-b523-4b68-851a-531ed101de5a/backends/copy/rootfs= es/e3af89fe-c889-45fc-929b-f008b18e5d4e' for container 0cea1dc7-b523-4b68-8= 51a-531ed101de5a > Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197619 = 3776 provisioner.cpp:434] Destroying container rootfs at '/mnt/mesos/provi= sioner/containers/a4c972da-b12d-4b4b-8481-79ca320ddab7/backends/copy/rootfs= es/feaf70a4-df43-45c9-b6be-a654c9210a54' for container a4c972da-b12d-4b4b-8= 481-79ca320ddab7 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.390928 = 3776 slave.cpp:4082] Executor '3' of framework c9852a23-bc07-422d-8d69-23c= 167a1924d-0003 has terminated with unknown status > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.395370 = 3776 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 543e1e1e-ae= 47-48ab-ad64-ee82fdd27418) for task 3 of framework c9852a23-bc07-422d-8d69-= 23c167a1924d-0003 from @0.0.0.0:0 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.402472 = 3776 slave.cpp:4082] Executor '2' of framework c9852a23-bc07-422d-8d69-23c= 167a1924d-0003 has terminated with unknown status > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.402556 = 3776 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 1a54664d-41= 57-42a3-a97f-7189c137a769) for task 2 of framework c9852a23-bc07-422d-8d69-= 23c167a1924d-0003 from @0.0.0.0:0 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.406177 = 3772 containerizer.cpp:1451] Ignoring update for unknown container: d9363f= 0d-645b-468f-947a-8bf97d15a630 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.407325 = 3777 status_update_manager.cpp:320] Received status update TASK_FAILED (UU= ID: 543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework c9852a23-= bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.407598 = 3773 containerizer.cpp:1451] Ignoring update for unknown container: a4c972= da-b12d-4b4b-8481-79ca320ddab7 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.409890 = 3774 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 543e1e1e-ae4= 7-48ab-ad64-ee82fdd27418) for task 3 of framework c9852a23-bc07-422d-8d69-2= 3c167a1924d-0003 to master@172.31.19.240:5050 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.409893 = 3777 status_update_manager.cpp:320] Received status update TASK_FAILED (UU= ID: 1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework c9852a23-= bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.414727 = 3774 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 1a54664d-415= 7-42a3-a97f-7189c137a769) for task 2 of framework c9852a23-bc07-422d-8d69-2= 3c167a1924d-0003 to master@172.31.19.240:5050 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.418180 = 3777 status_update_manager.cpp:392] Received status update acknowledgement= (UUID: 543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework c9852= a23-bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419615 = 3777 status_update_manager.cpp:392] Received status update acknowledgement= (UUID: 1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework c9852= a23-bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419636 = 3774 slave.cpp:4193] Cleaning up executor '3' of framework c9852a23-bc07-4= 22d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419924 = 3772 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c1= 67a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/= 3/runs/d9363f0d-645b-468f-947a-8bf97d15a630' for gc 6.9999951410637days in = the future > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420703 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c1= 67a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/= 3' for gc 6.99999513154074days in the future > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420722 = 3774 slave.cpp:4193] Cleaning up executor '2' of framework c9852a23-bc07-4= 22d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420897 = 3774 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c1= 67a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/= 2/runs/a4c972da-b12d-4b4b-8481-79ca320ddab7' for gc 6.99999512931852days in= the future > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420940 = 3774 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c1= 67a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/= 2' for gc 6.99999512869333days in the future > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490155 = 3772 slave.cpp:4082] Executor '4' of framework c9852a23-bc07-422d-8d69-23c= 167a1924d-0003 has terminated with unknown status > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490289 = 3772 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 9c5528ec-07= 25-42f2-a2e2-465d3c73b8d5) for task 4 of framework c9852a23-bc07-422d-8d69-= 23c167a1924d-0003 from @0.0.0.0:0 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490414 = 3772 slave.cpp:4082] Executor '0' of framework c9852a23-bc07-422d-8d69-23c= 167a1924d-0003 has terminated with unknown status > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490489 = 3772 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 3f801210-b0= 29-427f-af3f-e51b1dfa5cc8) for task 0 of framework c9852a23-bc07-422d-8d69-= 23c167a1924d-0003 from @0.0.0.0:0 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.490687 = 3773 containerizer.cpp:1451] Ignoring update for unknown container: 58c398= e2-38f9-4389-87ba-8696c9a523eb > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.490774 = 3773 containerizer.cpp:1451] Ignoring update for unknown container: 09a77a= 80-f904-4581-966a-734d24ae96f3 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490867 = 3773 status_update_manager.cpp:320] Received status update TASK_FAILED (UU= ID: 9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework c9852a23-= bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491082 = 3773 status_update_manager.cpp:320] Received status update TASK_FAILED (UU= ID: 3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework c9852a23-= bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491082 = 3772 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 9c5528ec-072= 5-42f2-a2e2-465d3c73b8d5) for task 4 of framework c9852a23-bc07-422d-8d69-2= 3c167a1924d-0003 to master@172.31.19.240:5050 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491303 = 3774 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 3f801210-b02= 9-427f-af3f-e51b1dfa5cc8) for task 0 of framework c9852a23-bc07-422d-8d69-2= 3c167a1924d-0003 to master@172.31.19.240:5050 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494438 = 3775 status_update_manager.cpp:392] Received status update acknowledgement= (UUID: 9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework c9852= a23-bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494572 = 3775 slave.cpp:4193] Cleaning up executor '4' of framework c9852a23-bc07-4= 22d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494757 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c1= 67a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/= 4/runs/58c398e2-38f9-4389-87ba-8696c9a523eb' for gc 6.99999427436444days in= the future > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494817 = 3775 status_update_manager.cpp:392] Received status update acknowledgement= (UUID: 3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework c9852= a23-bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494937 = 3775 slave.cpp:4193] Cleaning up executor '0' of framework c9852a23-bc07-4= 22d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494823 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c1= 67a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/= 4' for gc 6.99999427361185days in the future > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.495107 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c1= 67a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/= 0/runs/09a77a80-f904-4581-966a-734d24ae96f3' for gc 6.99999427055407days in= the future > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.495149 = 3773 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c1= 67a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/= 0' for gc 6.99999427024296days in the future > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591338 = 3775 slave.cpp:4082] Executor '1' of framework c9852a23-bc07-422d-8d69-23c= 167a1924d-0003 has terminated with unknown status > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591469 = 3775 slave.cpp:3211] Handling status update TASK_FAILED (UUID: 85b7c18a-c5= 9c-4449-827c-a3bfaa2a7de8) for task 1 of framework c9852a23-bc07-422d-8d69-= 23c167a1924d-0003 from @0.0.0.0:0 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.591841 = 3776 containerizer.cpp:1451] Ignoring update for unknown container: 0cea1d= c7-b523-4b68-851a-531ed101de5a > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591974 = 3770 status_update_manager.cpp:320] Received status update TASK_FAILED (UU= ID: 85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework c9852a23-= bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.592157 = 3772 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID: 85b7c18a-c59= c-4449-827c-a3bfaa2a7de8) for task 1 of framework c9852a23-bc07-422d-8d69-2= 3c167a1924d-0003 to master@172.31.19.240:5050 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594518 = 3771 status_update_manager.cpp:392] Received status update acknowledgement= (UUID: 85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework c9852= a23-bc07-422d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594636 = 3771 slave.cpp:4193] Cleaning up executor '1' of framework c9852a23-bc07-4= 22d-8d69-23c167a1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594792 = 3770 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c1= 67a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/= 1/runs/0cea1dc7-b523-4b68-851a-531ed101de5a' for gc 6.99999311635556days in= the future > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594820 = 3771 slave.cpp:4281] Cleaning up framework c9852a23-bc07-422d-8d69-23c167a= 1924d-0003 > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594861 = 3770 gc.cpp:55] Scheduling '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c1= 67a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/= 1' for gc 6.99999311568296days in the future > Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594897 = 3770 status_update_manager.cpp:282] Closing status update streams for fram= ework c9852a23-bc07-422d-8d69-23c167a1924d-0003 > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)