livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vivek Suvarna <vikk...@gmail.com>
Subject Re: Multiple Livy instances and load balancing
Date Thu, 27 Jul 2017 00:24:28 GMT
Thanks Meisam. 


Is this code in github? 
Is there a PR for this?

Regards
Vivek


Sent from my iPhone

> On 27 Jul 2017, at 12:51 AM, Meisam Fathi <meisam.fathi@gmail.com> wrote:
> 
> Hi Vivek,
> 
> We are running multiple instances of Livy on our clusters. Our users can create and access
jobs on any of the Livy instances.
> 
> To answer your particular question
> 
> 1. Is this feature available in the 0.3 release? No. The feature is not available out
of the box–we modified Livy to add this feature. 
> 2. How would I name/number the multiple instances I bring up? The load balancer knows
all the instances. Users only interact with the load balancer.
> 3. How does one load balance and send requests across the multiple instances? Livy instances
do not know that requests come from a load balancer. Each processes requests it receives and
updates a shared "session store", which we implemented on top of ZooKeeper.
> 4. Does Livy have a heartbeat mechanism to understand which or how many instances are
up? No. There is a heartbeat mechanism in Livy, but we do not using it to detect live/dead
Livy instances. Each instance only reacts to the updates to the ZooKeeper "session store",
which is shared by all instances of Livy.
> 
> Thanks,
> Meisam
> 
>> On Wed, Jul 26, 2017 at 1:36 AM Vivek <viveksuvarna@yahoo.co.in> wrote:
>> Ok.so if I start multiple instances how will I know which instance to send the request
to?
>> And multiple instances would then be only controlled by the port id? 
>> 
>> 
>> Sent from my iPhone
>> 
>>> On 26 Jul 2017, at 4:04 PM, Saisai Shao <sai.sai.shao@gmail.com> wrote:
>>> 
>>> Current Livy doesn't support the things you mentioned here. You can start multiple
Livy in the cluster, but each LivyServer is a standalone service doesn't aware the existence
of others.
>>> 
>>>> On Wed, Jul 26, 2017 at 10:27 AM, Vivek <viveksuvarna@yahoo.co.in>
wrote:
>>>> Hi,
>>>> 
>>>> We are now considering moving into a uat environment using Livy at my company.
>>>> 
>>>> Has anyone implemented multiple Livy instances on a single cluster with load
balancing?
>>>> 
>>>> A few questions.
>>>> 1. Is this feature available in the 0.3 release?
>>>> 2. How would I name/number the multiple instances I bring up?
>>>> 3. How does one load balance and send requests across the multiple instances?
>>>> 4. Does Livy have a heartbeat mechanism to understand which or how many instances
are up?
>>>> 
>>>> Any answers would be appreciated.
>>>> 
>>>> Regards
>>>> Vivek
>>>> 
>>>> 
>>>> Sent from my iPhone
>>> 

Mime
View raw message