aurora-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zameer Manji <zma...@apache.org>
Subject Re: Review Request 51874: Change framework_name default value from 'TwitterScheduler' to 'Aurora'
Date Thu, 15 Sep 2016 18:34:36 GMT


> On Sept. 14, 2016, 3:48 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java,
line 82
> > <https://reviews.apache.org/r/51874/diff/5/?file=1498686#file1498686line82>
> >
> >     Did you try to rollback to pre 0.15 scheduler while changing the framework name?
Trying to see if we can drop this 'backwards incompatible' statement now.
> 
> Santhosh Kumar Shanmugham wrote:
>     Tested "roll-forward" (to Aurora) and "roll-back" (via release and config change)
(to TwitterScheduler) on Aurora-0.14 (depends on Mesos-0.27.2) and Aurora-0.15(dependes on
Mesos-0.28.2). The master was able to re-register the framework with the same "id" and the
running tasks were continuing to make progress. (See details in testing section)
>     
>     However I could not rollback the scheduler from 0.15 to 0.14 from source inside vagrant.
Started to on "aurorabuild all" complain with message,
>     "Could not satisfy all requirements for mesos.native==0.27.2"
> 
> Santhosh Kumar Shanmugham wrote:
>     Tested changing the framework_name on Aurora 0.14, 0.15 and master. Dropping the
comment about 'backward incompatible'.

Just to be clear, you tested this change against a single Mesos master verison right? Could
you share which version of Mesos that was?


- Zameer


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51874/#review148988
-----------------------------------------------------------


On Sept. 14, 2016, 5:33 p.m., Santhosh Kumar Shanmugham wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51874/
> -----------------------------------------------------------
> 
> (Updated Sept. 14, 2016, 5:33 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Maxim Khutornenko.
> 
> 
> Bugs: AURORA-1688
>     https://issues.apache.org/jira/browse/AURORA-1688
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> Change framework_name default value from 'TwitterScheduler' to 'Aurora'
> 
> 
> Diffs
> -----
> 
>   RELEASE-NOTES.md ad2c68a6defe07c94480d7dee5b1496b50dc34e5 
>   src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
8a386bd208956eb0c8c2f48874b0c6fb3af58872 
>   src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 97677f24a50963178a123b420d7ac136e4fde3fe

> 
> Diff: https://reviews.apache.org/r/51874/diff/
> 
> 
> Testing
> -------
> 
> ./build-support/jenkins/build.sh
> ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 
> Testing to make sure backward compatibility:
> 
> # HEAD of master:
> 
> # Case 1: Rolling forward does not impact running tasks:
> Renaming framework from 'TwitterScheduler' to 'Aurora':
> 
> The framework re-registers after restart (treated by master as failover) and gets the
same framework-id. Running task remain unaffected.
> 
> Master log:
> I0914 16:48:28.408182  9815 master.cpp:1297] Giving framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
(TwitterScheduler) at scheduler-75517c8f-5913-49e9-8cc4-342a78c9bbcb@192.168.33.7:8083 3weeks
to failover
> I0914 16:48:28.408226  9815 hierarchical.cpp:382] Deactivated framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
> E0914 16:48:28.408617  9819 process.cpp:2105] Failed to shutdown socket with fd 28: Transport
endpoint is not connected
> I0914 16:48:43.722126  9813 master.cpp:2424] Received SUBSCRIBE call for framework 'Aurora'
at scheduler-dfad8309-de4b-47d8-a8f8-82828ea40a12@192.168.33.7:8083
> I0914 16:48:43.722190  9813 master.cpp:2500] Subscribing framework Aurora with checkpointing
enabled and capabilities [ REVOCABLE_RESOURCES, GPU_RESOURCES ]
> I0914 16:48:43.722225  9813 master.cpp:2564] Updating info for framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
> I0914 16:48:43.722256  9813 master.cpp:2577] Framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
(Aurora) at scheduler-75517c8f-5913-49e9-8cc4-342a78c9bbcb@192.168.33.7:8083 failed over
> I0914 16:48:43.722429  9813 hierarchical.cpp:348] Activated framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
> I0914 16:48:43.722595  9813 master.cpp:5709] Sending 1 offers to framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
(Aurora) at scheduler-dfad8309-de4b-47d8-a8f8-82828ea40a12@192.168.33.7:8083
> 
> Scheduler log:
> I0914 16:48:44.157 [Thread-10, MesosSchedulerImpl:151] Registered with ID value: "071c44a1-b4d4-4339-a727-03a79f725851-0000"
> , master: id: "461b98b8-63e1-40e3-96fd-cb62420945ae"
> ip: 119646400
> port: 5050
> pid: "master@192.168.33.7:5050"
> hostname: "aurora.local"
> version: "1.0.0"
> address {
>   hostname: "aurora.local"
>   ip: "192.168.33.7"
>   port: 5050
> }
> 
> # Case 2: Rolling backward does not impact running tasks:
> Rolling back framework name from 'Aurora' to 'TwitterScheduler':
> 
> The framework re-registers after restart (treated by master as failover) and gets the
same framework-id. Running task remain unaffected.
> 
> Master log:
> I0914 16:51:33.203495  9812 master.cpp:1297] Giving framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
(Aurora) at scheduler-dfad8309-de4b-47d8-a8f8-82828ea40a12@192.168.33.7:8083 3weeks to failover
> I0914 16:51:33.203526  9812 hierarchical.cpp:382] Deactivated framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
> I0914 16:51:49.614074  9813 master.cpp:2424] Received SUBSCRIBE call for framework 'TwitterScheduler'
at scheduler-6fa8b819-aed9-42e1-9c6c-3e4be2f62500@192.168.33.7:8083
> I0914 16:51:49.614215  9813 master.cpp:2500] Subscribing framework TwitterScheduler with
checkpointing enabled and capabilities [ REVOCABLE_RESOURCES, GPU_RESOURCES ]
> I0914 16:51:49.614312  9813 master.cpp:2564] Updating info for framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
> I0914 16:51:49.614359  9813 master.cpp:2577] Framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
(TwitterScheduler) at scheduler-dfad8309-de4b-47d8-a8f8-82828ea40a12@192.168.33.7:8083 failed
over
> I0914 16:51:49.614977  9813 hierarchical.cpp:348] Activated framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
> I0914 16:51:49.615170  9813 master.cpp:5709] Sending 1 offers to framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
(TwitterScheduler) at scheduler-6fa8b819-aed9-42e1-9c6c-3e4be2f62500@192.168.33.7:8083
> 
> Scheduler log:
> I0914 16:51:50.249 [Thread-10, MesosSchedulerImpl:151] Registered with ID value: "071c44a1-b4d4-4339-a727-03a79f725851-0000"
> , master: id: "461b98b8-63e1-40e3-96fd-cb62420945ae"
> ip: 119646400
> port: 5050
> pid: "master@192.168.33.7:5050"
> hostname: "aurora.local"
> version: "1.0.0"
> address {
>   hostname: "aurora.local"
>   ip: "192.168.33.7"
>   port: 5050
> }
> 
> # Case 3: Restarting with old framework_name (rolling back config) does not impact running
tasks:
> Restarting the scheduler after updating the config from 'Aurora' to 'TwitterScheduler':
> 
> Rename takes effect. The master re-registered the framework to the same id. Running task
remain unaffected.
> 
> Master log:
> I0914 20:34:58.059640 28176 master.cpp:1297] Giving framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
(Aurora) at scheduler-4a7c21b7-5d90-4218-936e-4142051b3444@192.168.33.7:8083 3weeks to failover
> I0914 20:34:58.059675 28176 hierarchical.cpp:382] Deactivated framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
> I0914 20:35:23.447479 28175 master.cpp:2424] Received SUBSCRIBE call for framework 'TwitterScheduler'
at scheduler-cea31751-7cb5-46b2-8208-f9ab1d4fe86c@192.168.33.7:8083
> I0914 20:35:23.447573 28175 master.cpp:2500] Subscribing framework TwitterScheduler with
checkpointing enabled and capabilities [ REVOCABLE_RESOURCES, GPU_RESOURCES ]
> I0914 20:35:23.447592 28175 master.cpp:2564] Updating info for framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
> I0914 20:35:23.447615 28175 master.cpp:2577] Framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
(TwitterScheduler) at scheduler-4a7c21b7-5d90-4218-936e-4142051b3444@192.168.33.7:8083 failed
over
> I0914 20:35:23.447777 28175 hierarchical.cpp:348] Activated framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
> I0914 20:35:23.447968 28175 master.cpp:5709] Sending 1 offers to framework 071c44a1-b4d4-4339-a727-03a79f725851-0000
(TwitterScheduler) at scheduler-cea31751-7cb5-46b2-8208-f9ab1d4fe86c@192.168.33.7:8083
> 
> Scheduler log:
> I0914 20:35:24.000 [Thread-10, MesosSchedulerImpl:151] Registered with ID value: "071c44a1-b4d4-4339-a727-03a79f725851-0000
> "
> , master: id: "848618fb-714d-4b00-ad80-950f6bdc70c6"
> ip: 119646400
> port: 5050
> pid: "master@192.168.33.7:5050"
> hostname: "aurora.local"
> version: "1.0.0"
> address {
>   hostname: "aurora.local"
>   ip: "192.168.33.7"
>   port: 5050
> }
> 
> # Testing on olders versions that uses Mesos 0.28 (Aurora 0.15) and Mesos 0.27 (Aurora
0.14)
> 
> # Aurora Version: 0.14
> # Initial version (TwitterScheduler)
> # https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=commit;h=b0b598088847630f37c3f995db98a8edf9520b7e
> 
> git reset —hard b0b598088847630f37c3f995db98a8edf9520b7e # reset HEAD to v0.14
> vagrant destroy
> vagrant up
> 
> vagrant ssh -c "aurora job create devcluster/www-data/prod/hello aurora/examples/jobs/hello_world.aurora"
# start some job
> 
> Verify the framework name (TwitterScheduler) and id (some id - XXX) http://192.168.33.7:5050/#/frameworks
> Verify the task is running http://192.168.33.7:8081/scheduler/www-data/prod/hello
> 
> vagrant@aurora:~$ sudo grep 'Registered with ID value' /var/log/upstart/aurora-scheduler.log
> I0914 23:26:52.095 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> 
> # Roll forward
> git apply -3 ~/Downloads/rb51874.patch # apply the framework name default change
> vagrant ssh -c “aurorabuild scheduler” # rebuild
> 
> Verify the framework name (Aurora) and id (same id - XXX) http://192.168.33.7:5050/#/frameworks
> Verify the task is still running http://192.168.33.7:8081/scheduler/www-data/prod/hello
> 
> vagrant@aurora:~$ sudo grep 'Registered with ID value' /var/log/upstart/aurora-scheduler.log
> I0914 23:26:52.095 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> I0914 23:33:19.336 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> 
> # Roll backward
> git stash
> vagrant ssh -c “aurorabuild scheduler” # rebuild
> 
> Verify the framework name (TwitterScheduler) and id (same id - XXX) http://192.168.33.7:5050/#/frameworks
> Verify the task is still running http://192.168.33.7:8081/scheduler/www-data/prod/hello
> 
> vagrant@aurora:~$ sudo grep 'Registered with ID value' /var/log/upstart/aurora-scheduler.log
> I0914 23:26:52.095 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> I0914 23:33:19.336 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> I0914 23:35:28.734 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> 
> # Roll forward again
> git apply -3 ~/Downloads/rb51874.patch # apply the framework name default change
> vagrant ssh -c “aurorabuild scheduler” # rebuild
> 
> Verify the framework name (Aurora) and id (same id - XXX) http://192.168.33.7:5050/#/frameworks
> Verify the task is still running http://192.168.33.7:8081/scheduler/www-data/prod/hello
> 
> vagrant@aurora:~$ sudo grep 'Registered with ID value' /var/log/upstart/aurora-scheduler.log
> I0914 23:26:52.095 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> I0914 23:33:19.336 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> I0914 23:35:28.734 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> I0914 23:36:29.195 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> 
> # Restart with old framework name
> vagrant ssh
> sudo vim /etc/init/aurora-scheduler.conf
> # add -framework_name=TwitterScheduler after "exec bin/aurora-scheduler” and save
> sudo stop aurora-scheduler
> sudo start aurora-scheduler
> 
> vagrant@aurora:~$ sudo grep 'Registered with ID value' /var/log/upstart/aurora-scheduler.log
> I0914 23:26:52.095 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> I0914 23:33:19.336 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> I0914 23:35:28.734 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> I0914 23:36:29.195 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> I0914 23:39:46.118 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "317cd38a-edc1-4168-8c20-ca81d8306e04-0000"
> 
> Verify the framework name (TwitterScheduler) and id (same id - XXX) http://192.168.33.7:5050/#/frameworks
> Verify the task is still running http://192.168.33.7:8081/scheduler/www-data/prod/hello
> 
> # Aurora Version: 0.15
> 
> # Initial Version
> # https://git-wip-us.apache.org/repos/asf?p=aurora.git;a=commit;h=e870884fb30bc4d960aa5ed4901df679edbafb34
> git reset —hard e870884fb30bc4d960aa5ed4901df679edbafb34
> 
> vagrant destroy
> vagrant up
> 
> # start some job
> vagrant ssh -c "aurora job create devcluster/www-data/prod/hello aurora/examples/jobs/hello_world.aurora"
> 
> vagrant@aurora:~$ sudo grep 'Registered with ID value' /var/log/upstart/aurora-scheduler.log
> I0915 00:08:11.136 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> 
> vagrant@aurora:~$ aurora job status  devcluster
>  INFO] Retrieving jobs for role None
>  INFO] Checking status of devcluster/www-data/prod/hello
> Active tasks (1):
>            Task role: www-data, env: prod, name: hello, instance: 0, status: RUNNING
on 192.168.33.7
>              CPU: 1.0 core(s), RAM: 128 MB, Disk: 128 MB
>              events:
>               2016-09-15 00:08:36 PENDING: None
>               2016-09-15 00:08:37 ASSIGNED: None
>               2016-09-15 00:08:39 STARTING: Initializing sandbox.
>               2016-09-15 00:08:39 RUNNING: None
> Inactive tasks (0):
> 
> # Roll forward:
> git stash pop
> vagrant ssh -c "aurorabuild scheduler"
> 
> vagrant@aurora:~$ sudo grep 'Registered with ID value' /var/log/upstart/aurora-scheduler.log
> I0915 00:08:11.136 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> I0915 00:12:33.395 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> vagrant@aurora:~$ aurora job status  devcluster
>  INFO] Retrieving jobs for role None
>  INFO] Checking status of devcluster/www-data/prod/hello
> Active tasks (1):
>            Task role: www-data, env: prod, name: hello, instance: 0, status: RUNNING
on 192.168.33.7
>              CPU: 1.0 core(s), RAM: 128 MB, Disk: 128 MB
>              events:
>               2016-09-15 00:08:36 PENDING: None
>               2016-09-15 00:08:37 ASSIGNED: None
>               2016-09-15 00:08:39 STARTING: Initializing sandbox.
>               2016-09-15 00:08:39 RUNNING: None
> Inactive tasks (0):
> 
> # Rollback
> git stash
> vagrant ssh -c "aurorabuild scheduler"
> 
> vagrant@aurora:~$ sudo grep 'Registered with ID value' /var/log/upstart/aurora-scheduler.log
> I0915 00:08:11.136 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> I0915 00:12:33.395 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> I0915 00:14:49.374 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> vagrant@aurora:~$ aurora job status  devcluster
>  INFO] Retrieving jobs for role None
>  INFO] Checking status of devcluster/www-data/prod/hello
> Active tasks (1):
>            Task role: www-data, env: prod, name: hello, instance: 0, status: RUNNING
on 192.168.33.7
>              CPU: 1.0 core(s), RAM: 128 MB, Disk: 128 MB
>              events:
>               2016-09-15 00:08:36 PENDING: None
>               2016-09-15 00:08:37 ASSIGNED: None
>               2016-09-15 00:08:39 STARTING: Initializing sandbox.
>               2016-09-15 00:08:39 RUNNING: None
> Inactive tasks (0):
> 
> # Roll forward:
> git stash pop
> vagrant ssh -c "aurorabuild scheduler"
> 
> vagrant@aurora:~$ sudo grep 'Registered with ID value' /var/log/upstart/aurora-scheduler.log
> I0915 00:08:11.136 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> I0915 00:12:33.395 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> I0915 00:14:49.374 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> I0915 00:16:14.004 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> vagrant@aurora:~$ aurora job status  devcluster
>  INFO] Retrieving jobs for role None
>  INFO] Checking status of devcluster/www-data/prod/hello
> Active tasks (1):
>            Task role: www-data, env: prod, name: hello, instance: 0, status: RUNNING
on 192.168.33.7
>              CPU: 1.0 core(s), RAM: 128 MB, Disk: 128 MB
>              events:
>               2016-09-15 00:08:36 PENDING: None
>               2016-09-15 00:08:37 ASSIGNED: None
>               2016-09-15 00:08:39 STARTING: Initializing sandbox.
>               2016-09-15 00:08:39 RUNNING: None
> Inactive tasks (0):
> 
> # Restart with old framework name
> vagrant ssh
> sudo vim /etc/init/aurora-scheduler.conf
> # add -framework_name=TwitterScheduler after "exec bin/aurora-scheduler” and save
> sudo stop aurora-scheduler
> sudo start aurora-scheduler
> 
> vagrant@aurora:~$ sudo grep 'Registered with ID value' /var/log/upstart/aurora-scheduler.log
> I0915 00:08:11.136 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> I0915 00:12:33.395 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> I0915 00:14:49.374 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> I0915 00:16:14.004 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> I0915 00:18:16.200 [Thread-11, MesosSchedulerImpl:151] Registered with ID value: "308d7661-6bb1-4936-86b4-a01158bfa06b-0000"
> vagrant@aurora:~$ aurora job status  devcluster
>  INFO] Retrieving jobs for role None
>  INFO] Checking status of devcluster/www-data/prod/hello
> Active tasks (1):
>            Task role: www-data, env: prod, name: hello, instance: 0, status: RUNNING
on 192.168.33.7
>              CPU: 1.0 core(s), RAM: 128 MB, Disk: 128 MB
>              events:
>               2016-09-15 00:08:36 PENDING: None
>               2016-09-15 00:08:37 ASSIGNED: None
>               2016-09-15 00:08:39 STARTING: Initializing sandbox.
>               2016-09-15 00:08:39 RUNNING: None
> Inactive tasks (0):
> 
> 
> Thanks,
> 
> Santhosh Kumar Shanmugham
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message