kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ewe...@apache.org
Subject [kafka] branch 1.0 updated: MINOR: Switch to use AWS spot instances
Date Fri, 05 Oct 2018 17:22:36 GMT
This is an automated email from the ASF dual-hosted git repository.

ewencp pushed a commit to branch 1.0
in repository https://gitbox.apache.org/repos/asf/kafka.git


The following commit(s) were added to refs/heads/1.0 by this push:
     new b88b582  MINOR: Switch to use AWS spot instances
b88b582 is described below

commit b88b5823d013f3c64e0a516fead3b2e8f0d22e80
Author: Max Zheng <maxzheng.os@gmail.com>
AuthorDate: Fri Oct 5 10:21:25 2018 -0700

    MINOR: Switch to use AWS spot instances
    
    Pricing for m3.xlarge: On-Demand is at $0.266. Reserved is at about $0.16 (40% discount).
And Spot is at $0.0627 (76% discount relative to On-Demand, or 60% discount relative to Reserved).
Insignificant fluctuation in the past 3 months.
    
    Ran on branch builder and works as expected -- each worker is created using spot instances
(https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1982/console)
    
    This can be safely backported to 0.10.2 (tested using https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1983/)
    
    Author: Max Zheng <maxzheng.os@gmail.com>
    
    Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
    
    Closes #5707 from maxzheng/minor-switch@trunk
    
    (cherry picked from commit 50ec82940d4af61f12300235b7553bd5cf231894)
    Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
---
 Vagrantfile                               |  6 ++++++
 tests/README.md                           |  1 +
 vagrant/aws/aws-example-Vagrantfile.local |  1 +
 vagrant/aws/aws-init.sh                   | 11 +++++++----
 4 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/Vagrantfile b/Vagrantfile
index 3636076..88f2028 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -42,6 +42,8 @@ ec2_region = "us-east-1"
 ec2_az = nil # Uses set by AWS
 ec2_ami = "ami-905730e8"
 ec2_instance_type = "m3.medium"
+ec2_spot_instance = ENV['SPOT_INSTANCE'] ? ENV['SPOT_INSTANCE'] == 'true' : true
+ec2_spot_max_price = "0.113"  # On-demand price for instance type
 ec2_user = "ubuntu"
 ec2_instance_name_prefix = "kafka-vagrant"
 ec2_security_groups = nil
@@ -133,6 +135,10 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
     else
       aws.associate_public_ip = ec2_associate_public_ip
     end
+    aws.region_config ec2_region do |region|
+      region.spot_instance = ec2_spot_instance
+      region.spot_max_price = ec2_spot_max_price
+    end
 
     # Exclude some directories that can grow very large from syncing
     override.vm.synced_folder ".", "/vagrant", type: "rsync", rsync__exclude: ['.git', 'core/data/',
'logs/', 'tests/results/', 'results/']
diff --git a/tests/README.md b/tests/README.md
index f0ffdf5..f42b28a 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -461,6 +461,7 @@ the test driver machine.
         ec2_instance_type = "..." # Pick something appropriate for your
                                   # test. Note that the default m3.medium has
                                   # a small disk.
+        ec2_spot_max_price = "0.123"  # On-demand price for instance type
         enable_hostmanager = false
         num_zookeepers = 0
         num_kafka = 0
diff --git a/vagrant/aws/aws-example-Vagrantfile.local b/vagrant/aws/aws-example-Vagrantfile.local
index ee9db9a..23187a0 100644
--- a/vagrant/aws/aws-example-Vagrantfile.local
+++ b/vagrant/aws/aws-example-Vagrantfile.local
@@ -17,6 +17,7 @@
 # To use it, move it to the base kafka directory and rename
 # it to Vagrantfile.local, and adjust variables as needed.
 ec2_instance_type = "m3.xlarge"
+ec2_spot_max_price = "0.266"  # On-demand price for instance type
 enable_hostmanager = false
 num_zookeepers = 0
 num_brokers = 0
diff --git a/vagrant/aws/aws-init.sh b/vagrant/aws/aws-init.sh
index 7517626..54092c8 100755
--- a/vagrant/aws/aws-init.sh
+++ b/vagrant/aws/aws-init.sh
@@ -31,15 +31,18 @@ base_dir=`dirname $0`/../..
 
 if [ -z `which vagrant` ]; then
     echo "Installing vagrant..."
-    wget https://releases.hashicorp.com/vagrant/1.9.3/vagrant_1.9.3_x86_64.deb
-    sudo dpkg -i vagrant_1.9.3_x86_64.deb
-    rm -f vagrant_1.9.3_x86_64.deb
+    wget https://releases.hashicorp.com/vagrant/2.1.5/vagrant_2.1.5_x86_64.deb
+    sudo dpkg -i vagrant_2.1.5_x86_64.deb
+    rm -f vagrant_2.1.5_x86_64.deb
 fi
 
 # Install necessary vagrant plugins
 # Note: Do NOT install vagrant-cachier since it doesn't work on AWS and only
 # adds log noise
-vagrant_plugins="vagrant-aws vagrant-hostmanager"
+
+# Custom vagrant-aws with spot instance support. See https://github.com/mitchellh/vagrant-aws/issues/32
+wget -nv https://s3-us-west-2.amazonaws.com/confluent-packaging-tools/vagrant-aws-0.7.2.spot.gem
-P /tmp
+vagrant_plugins="/tmp/vagrant-aws-0.7.2.spot.gem vagrant-hostmanager"
 existing=`vagrant plugin list`
 for plugin in $vagrant_plugins; do
     echo $existing | grep $plugin > /dev/null


Mime
View raw message