zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maarten Koopmans <maar...@vrijheid.net>
Subject Re: Zookeeper on 60+Gb mem
Date Tue, 05 Oct 2010 22:02:24 GMT
Good point. And Cassandra is a no-go for me for now. I get the model, but I don't like, check,
dislike, things like Thrift.  



Op 5 okt. 2010 om 23:54 heeft Dave Wright <wrightd@gmail.com> het volgende geschreven:

> I think the issue of having to write a full ~60GB snapshot file at
> intervals would make this prohibitive, particularly on EC2 via EBS. At
> a scale like that I think you'd be better off with a traditional
> database or a nosql database like Cassandra, possibly using Zookeeper
> for transaction locking/coordination on top.
> 
> 
> -Dave Wright
> 
> On Tue, Oct 5, 2010 at 5:27 PM, Patrick Hunt <phunt@apache.org> wrote:
>> Tuning GC is going to be critical, otw all the sessions will timeout (and
>> potentially expire) during GC pauses.
>> 
>> Patrick
>> 
>> On Tue, Oct 5, 2010 at 1:18 PM, Maarten Koopmans <maarten@vrijheid.net>wrote:
>> 
>>> Yes, and syncing after a crash will be interesting as well. Off note; I am
>>> running it with a 6GB heap now, but it's not filled yet. I do have smoke
>>> tests thoug, so maybe I'll give it a try.
>>> 
>>> 
>>> 
>>> Op 5 okt. 2010 om 21:13 heeft Benjamin Reed <breed@yahoo-inc.com> het
>>> volgende geschreven:
>>> 
>>>> 
>>>> you will need to time how long it takes to read all that state back in
>>> and adjust the initTime accordingly. it will probably take a while to pull
>>> all that data into memory.
>>>> 
>>>> ben
>>>> 
>>>> On 10/05/2010 11:36 AM, Avinash Lakshman wrote:
>>>>> I have run it over 5 GB of heap with over 10M znodes. We will definitely
>>> run
>>>>> it with over 64 GB of heap. Technically I do not see any limitiation.
>>>>> However I will the experts chime in.
>>>>> 
>>>>> Avinash
>>>>> 
>>>>> On Tue, Oct 5, 2010 at 11:14 AM, Mahadev Konar<mahadev@yahoo-inc.com
>>>> wrote:
>>>>> 
>>>>>> Hi Maarteen,
>>>>>>  I definitely know of a group which uses around 3GB of memory heap
for
>>>>>> zookeeper but never heard of someone with such huge requirements.
I
>>> would
>>>>>> say it definitely would be a learning experience with such high memory
>>>>>> which
>>>>>> I definitely think would be very very useful for others in the
>>> community as
>>>>>> well.
>>>>>> 
>>>>>> Thanks
>>>>>> mahadev
>>>>>> 
>>>>>> 
>>>>>> On 10/5/10 11:03 AM, "Maarten Koopmans"<maarten@vrijheid.net>
 wrote:
>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> I just wondered: has anybody ever ran zookeeper "to the max"
on a 68GB
>>>>>>> quadruple extra large high memory EC2 instance? With, say, 60GB
>>> allocated
>>>>>> or
>>>>>>> so?
>>>>>>> 
>>>>>>> Because EC2 with EBS is a nice way to grow your zookeeper cluster
>>> (data
>>>>>> on the
>>>>>>> ebs columes, upgrade as your memory utilization grows....)  -
I just
>>>>>> wonder
>>>>>>> what the limits are there, or if I am foing where angels fear
to
>>> tread...
>>>>>>> 
>>>>>>> --Maarten
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>> 
> 

Mime
View raw message