zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jordan Zimmerman <jor...@jordanzimmerman.com>
Subject Re: InterProcessSemaphoreMutex not cleaning up old znodes
Date Thu, 17 Sep 2015 14:45:15 GMT
Jens,

Reaper and ChildReaper are now deprecated. Moving forward, Curator will automatically use
container nodes which are now supported directly by ZooKeeper (I wrote the feature). Reaper/ChildReaper
was always a hack to work around the situation.

-JZ



On September 17, 2015 at 9:43:03 AM, Jens Rantil (jens.rantil@tink.se) wrote:

Hi again Jordan,

Your input is highly appreciated. Thanks!

I assume you mean I would do something like
myReaper = new ChildReaper(client, "/mylock")
myReaper.start()
mutex = new InterProcessSemaphoreMutex(client, "/mylock/user123");
myReaper.addPath("/mylock/user123")
// Use the mutex...
However, when reading the implementation of ChildReaper (version 2.7.1) it looks like the
paths member field never removes "/mylock/user123" if it has been reaped by the reaper for "/mylock".
This would essentially become a memory leak if I have a lot of users, right? Or have I misunderstood
something?

Generally this isn't a pretty abstraction since a ChildReaper user (me) must essentially know
how the internals of a recipe to know if it creates additional hierarchies or not. A different
approach would be to add a `Reapable` interface to all recipe classes and have them implement
how they would reap themselves. Possibly they could simply delegate to a RecursiveChildReaper
that supports a configurable cleanup level (1 for basic locks, 2 for InterProcessSemaphoreMutexes
that has two level znode hierarchies).

Thanks,
Jens

On Thu, Sep 17, 2015 at 3:38 PM, Jordan Zimmerman <jordan@jordanzimmerman.com> wrote:
>
> You’d need to add each level to the ChildReaper. You can also move to ZooKeeper 3.5.1-alpha
and Curator 2.9.0 and you’ll get automatic cleanup (via Container nodes).
>
> -Jordan
>
>
>
> On September 17, 2015 at 8:11:49 AM, Jens Rantil (jens.rantil@tink.se) wrote:
>
> Hi again,
>
> A follow-up regarding this: I also have a couple of InterProcessSemaphoreMutexes that
I'd like to clean up and since they are using using a nested hierachy (containing "locks"
and "leases"), they aren't being cleaned up my ChildReaper. Is there a way to clean these
up automatically?
>
> Thanks,
> Jens
>
> On Mon, Sep 14, 2015 at 3:47 PM, Jordan Zimmerman <jordan@jordanzimmerman.com>
wrote:
>>
>> ChildReaper uses a Reaper internally. So, just use whichever one works for you.
>>
>> -Jordan
>>
>>
>>
>> On September 14, 2015 at 8:30:19 AM, Jens Rantil (jens.rantil@tink.se) wrote:
>>
>> Hi again,
>>
>> A small follow-up question. What are the pros and cons of using Reaper vs. ChildReaper?
It seems to me that using ChildReaper will involve a lot less intrusive code. Are there performance
implications of one or the other?
>>
>> Thanks,
>> Jens
>>
>> On Sun, Sep 13, 2015 at 10:27 PM, Jordan Zimmerman <jordan@jordanzimmerman.com>
wrote:
>>>
>>> This has long been an issue in ZooKeeper. The good news is that there are solutions:
>>>
>>> * Current releases: Curator has utilities for cleaning up old parent nodes. Look
at Reaper and ChildReaper
>>> * New releases: ZooKeeper 3.5.1 + Curator 2.9.0 (just released) support the new
Container node feature. Curator 2.9.0 will automatically create container nodes which are
automatically cleaned up when empty.
>>>
>>> -Jordan
>>>
>>>
>>> On September 13, 2015 at 11:41:41 AM, Jens Rantil (jens.rantil@tink.se) wrote:
>>>
>>> Hi again,
>>>
>>> I have a distributed application which has InterProcessSemaphoreMutexes which
occasionally are taken per user. To do this, the mutex path is appended with a user's unique
ID (example: /mylock/{userid}). All is fine and dandy and this generally works. However, I've
noticed that the paths given to the InterProcessSemaphoreMutexes aren't cleaned up after I'm
done with the locks.
>>>
>>> This means, my Zookeeper ensemble has every single one of my previous mutexes
held in memory. When I list my znodes I see:
>>> /mylock/1
>>> /mylock/2
>>> ...
>>> /mylock/XXX
>>>
>>> Given that I generally only take ~20 locks at a time it feels unnecessary to
keep all these in memory. We have had to increase initLimit in Zookeeper configuration, most
certainly due to all of these znodes.
>>>
>>> Two questions:
>>>
>>> Is there a reason for keeping these znodes hanging around?
>>> Given that I need to use mutexes, what are the best practise for taking locks
per user like this? Hash the user IDs and bucket them into a limitted number of mutexes (enough
to avoid contention)? Have a separate cleaning thread that cleans up old znodes? How are you
guys handling this?
>>>
>>> Thanks,
>>> Jens
>>>
>>> --
>>> Jens Rantil
>>> Backend engineer
>>> Tink AB
>>>
>>> Email: jens.rantil@tink.se
>>> Phone: +46 708 84 18 32
>>> Web: www.tink.se
>>>
>>> Facebook Linkedin Twitter
>>
>>
>>
>>
>> --
>> Jens Rantil
>> Backend engineer
>> Tink AB
>>
>> Email: jens.rantil@tink.se
>> Phone: +46 708 84 18 32
>> Web: www.tink.se
>>
>> Facebook Linkedin Twitter
>
>
>
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.rantil@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook Linkedin Twitter




--
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message