airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shameera Rathnayaka <shameerai...@gmail.com>
Subject Re: List of Devops Scripts for Apache Airavata
Date Fri, 10 Mar 2017 23:36:30 GMT
Hi,

I second Marlon's point about burden on CI/CD by having totally two
repositories, also we should have good discipline to keep two separate
repos in sync which is kind of hard IMO. Moving all scripts to one place
and organizing them in different subdirectories make more sense. That is
why I created dev-tools and dev-tools/ansible in the first place.

We only need to maintain two different scripts, one for development and one
for master, that is the only thing we will officially maintain.(feature
developers can have their own variation of scripts, that is not our
concern). We need to have proper code review sessions to avoid our scripts
being obsolete. As all in one repo help us to do this in same PR. We can
tag dev-tools along with new release branch to have proper versioning.

Thank you,
Shameera.

On Fri, Mar 10, 2017 at 1:14 PM Shenoy, Gourav Ganesh <goshenoy@indiana.edu>
wrote:

Hi Marcus,



Although it is not being actively used, but it still is in working shape.



Thanks and Regards,

Gourav Shenoy



*From: *"Christie, Marcus Aaron" <machrist@iu.edu>


*Date: *Friday, March 10, 2017 at 11:54 AM
*To: *"dev@airavata.apache.org" <dev@airavata.apache.org>, "Shenoy, Gourav
Ganesh" <goshenoy@indiana.edu>
*Subject: *Re: List of Devops Scripts for Apache Airavata



Dev,



Eric and I met this morning to discuss how to consolidate these various
efforts.  Here’s what we came up with:



1. (Marcus) Create an airavata-devops repo for holding all of the Ansible
scripts.  To start with, we’ll use the dev-tools/ansible scripts.

2. (Eric) Once the airavata-devops repo is in place, Eric will move his
Jetstream provisioning scripts onto a branch there

3. (Eric) Eric is also going to start working on Jetstream scripts for
other servers besides PGA



A question for Gourav: are the modules/cloud/ansible-playbooks still being
actively used?



Some other things we discussed:

* How to securely store secret information?  We discussed using
ansible-vault versus having a separate, private repo for Ansible
inventories.  We settled on using ansible-vault since it has the advantage
of keeping everything in one repo. The disadvantage is in needing to share
a password file out-of-band with the rest of the team.  However, we felt
the sharing of the password would be mitigated as we move toward more
automated systems for running deployments (for example, users who only need
to run a deploy won’t need the password file if we use Jenkins to run
deploys; only Jenkins would really need to have the password file).

* One advantage to NOT having a separate repo for devops scripts is that it
keeps the devops scripts versioned together with the code those scripts can
build and deploy.  Having a separate repo means we will need to have some
discipline when updating devops scripts.  I think it makes sense to have
the devops repo branch structure reflect the airavata branch structure.
That is, the ‘develop’ branch of the devops repo should be good for
deploying the ‘develop’ branch of airavata and likewise for ‘master’.  If
code is updated in airavata on a separate branch that needs corresponding
devops script updates, that should likewise be done on a same-named
separate branch in the devops repo.



Eric, let me know if I missed anything.





Thanks,



Marcus





On Mar 9, 2017, at 3:37 PM, Anuj Bhandar <bhandar.anuj@gmail.com> wrote:



Gourav,



It is a great initiative, I vote for a separate repository for devops, the
Airavata repository plays many roles already.



A separate repository would aid active development and bring modularity.



Thanks,

Anuj Bhandar



On 3/9/17 3:14 PM, Coulter, John Eric wrote:

Thanks for starting this, Gourav!

I'm inclined to vote for a separate DevOps repo, to keep things more
modular.



I know Marcus and I have done some work/testing starting from scripts that
I think Shameera created in dev-tools/ansible. I've got a side-repo up
which contains playbooks for provisioning and deploying the PGA from
scratch as a proof of concept, using dynamic inventory instead of a static
file (so, on an empty jetstream allocation, you get working instance
running the PGA, with router, public ip with a single ansible-playbook
command). Next step there is to add provisioning for VMs to run airavata,
etc.


*Script location*

*Branch*

*Purpose of Script*

modules/cloud/ansible-playbooks​​​

develop

1. Provisions instances on EC2 and OpenStack (Jetstream)

2. Deploys a Mesos/Marathon cluster on the provisioned i​nstances

 ​

​dev-tools/ansible/

​develop

​1. Deploy/update airavata services on existing machines​/instances
2. todo - include provisioning scripts for cloud resources

https://github.com/ECoulter/airavata-vms

​

​1. POC for provisioning and deployment of PGA on Jetstream, with dynamic
inventory. (intended to merge with dev-tools/ansible when complete/tested
by others)





---------------



Cheers,

---------------------
Eric Coulter        jecoulte@iu.edu
XSEDE Capabilities and Resource Integration Engineer
IU Campus Bridging & Research Infrastructure
RT/PTI/UITS
812-856-3250 <(812)%20856-3250>





-- 
Shameera Rathnayaka

Mime
View raw message