airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vidya Sagar Kalvakunta <vkalv...@umail.iu.edu>
Subject Re: UI Portal - Load Balancing & High availability
Date Fri, 27 Jan 2017 17:21:00 GMT
On Jan 27, 2017 8:53 AM, "Christie, Marcus Aaron" <machrist@iu.edu> wrote:

Vidya,

Thanks. Response are below.

On Jan 26, 2017, at 4:01 PM, Vidya Sagar Kalvakunta <vkalvaku@umail.iu.edu>
wrote:

Hi Marcus,

Using a service registry such as Consul as the backend for the load
balancer automatically ensures the service that is currently down (in this
case being updated) is removed from the routing table of the load balancer
and the service is added when it is back online.


In a way, yes that would work, but I wouldn’t want to disrupt ongoing
requests when taking a portal instance down during the deployment.  On
systems I’ve worked on in the past what we did was:
1. Take the portal instance out of the load balancer
2. At this point the portal instance may still be processing requests but
it shouldn’t be getting any new ones
3. Either monitor the number of processing requests and wait till it drops
to zero or just wait a predetermined amount of time (or both, for example,
wait until current requests finish or some max amount of time passes).
4. Then proceed with taking down the portal instance and updating it, etc.



If you want more fine-grained control, i.e there are multiple instances
with only a part of them currently updated to the latest version and you
want the load balancer to give more priority to the latest instances until
all the instances are done being updated.

A service registered in Consul can have tags associated with it, so each
instance of the Laravel portal can register a tag with its version number.
We can write the load balancer config file, such that it assigns more
weight to instances with "Version 2" tags and less weight to instances with
"Version 1" tags.


That’s interesting, I think that might work.  What I was thinking of is
take half of the instances out of the load balancer, update them, bring
them back up, put them back into the load balancer, then repeat with the
other half. That way you don’t have a mix of versions deployed.


This scenario will work with both Consul Template + HAProxy
<https://github.com/hashicorp/consul-template/blob/master/examples/haproxy.md>
or Fabio <https://github.com/eBay/fabio>.


Thanks,
Vidya




Marcus,

I think we can include the functionality within the Laravel portal so that
when it receives a shutdown signal, it deregisters itself from Consul.
Thereby it will not receive any further requests, and will finish
processing the currently running requests.

Your idea of updating half of the instances at once would be a better
solution than maintaining multiple versions at once.

Mime
View raw message