zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: counter with zookeeper
Date Thu, 02 Dec 2010 18:15:16 GMT
Regarding #4, post something on github and approve a few committers who
agree to keep things kosher Apache-style.

The cleanup should happen as if by magic and you can post the patch from
git.

On Thu, Dec 2, 2010 at 10:00 AM, David Rosenstrauch <darose@darose.net>wrote:

> Let me try to bump up the prio on getting this out soon.
>
> Blockers were:
> 1) waiting on getting this proved out in production - no longer an issue;
> we've been running this for weeks now
> 2) getting approval from management to release the code - also no longer an
> issue; the approved this a while ago
> 3) cleaning up the code to remove dependencies on our other code - this
> still needs to be done, but won't take long
> 4) getting this into shape for inclusion in the apache source tree (see:
> http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-user/201008.mbox/%3C4C618801.6050701@apache.org%3E)
> - this will actually take a while
>
> If I tackle #3, I can at least put out an "unofficial version" for people
> to play with.
>
> Tackling #4 will take longer though.  (Perhaps if someone wants to
> volunteer to help, this could get sped up.)
>
> Will email back soon with a link to grab the unofficial RC.
>
> DR
>
>
> On 12/02/2010 12:54 PM, Claudio Martella wrote:
>
>> Mhm, that's smart, i like it.
>>
>> looking forward to see your published code.
>>
>>
>> On 12/2/10 6:51 PM, David Rosenstrauch wrote:
>>
>>> Yes.  E.g., current state is:
>>>
>>>
>>> 2122691627-2122691654,2122691754-2122691786,2122719881-2122719915,2122720271-2122720368,2122720488-2122720510,2122721282-2122721368,2122721369-2122721391,2122733020-2122733095,2122733212-2122733330,2122733489-2122733546,2122733932-2122734000,2122734001-2122734013,2122734256-2122734300,2122738380-2122738419,2122738420-2122738516,2122738949-2122739091,2122739207-2122739212,2122739298-2122739324,2122739960-2122740122,2122740641-2122740651,2122740769-2122740789,2122744412-2122744449,2122745502-2122745554,2122745668-2122745734,2122745735-2122745801,2122745939-2122745959,2122746069-2122746105,2122746811-2122746845,2122746965-2122747044,2122747452-2122747477,2122747580-2122747602,2122747603-2122747656,2122747774-2122747844,2122747963-2122748009,2122748864-2122748935,2122749312-2122749331,2122749545-2122749577,2122749956-2122750030,2122750281-2122750592,2122750593-2122750593,2122753024-2122753079,2122753080-2122753084,2122753211-2122753214,2122753343-2122753455,2122753782-21227538
>>>
>> 11,
>
>>
>>>
>>> 2122754275-2122754352,2122755379-2122755527,2122755528-2122755611,2122755801-2122755849,2122755850-2122755889,2122756248-2122756275,2122756493-2122756586,2122756685-2122756737,2122756738-2122756861,2122756862-2122756887,2122757251-2122757360,2122757593-2122757715,2122757716-2122757736,2122757867-2122757941,2122758063-2122758079,2122758347-2122758359,2122758576-2122758651,2122758914-2122759041,2122759175-2122759191,2122759292-2122759319,2122759320-2122759334,2122759467-2122759524,2122759747-2122759765,2122759871-2122759952,2122759953-2122759991,2122760096-2122760191,2122760192-2122760199,2122760200-2122760227,2122760547-2122760601,2122760700-2122760701,2122760702-2122760713,2122760714-2122760830,2122761222-2122761316,2122761416-2122761439,2122762805-2122762987,2122763112-2122763148,2122763269-2122763336,2122764382-2122764445,2122764446-2122764463,2122764587-2122764613,2122764846-2122765138,2122765511-2122765511,2122765738-2122765744,2122766347-2122766398,2122766520-21227665
>>>
>> 44,
>
>>
>>>
>>> 2122766653-2122766756,2122766757-2122766820,2122767104-2122767148,2122767149-2122767185,2122767311-2122767359,2122767360-2122767398,2122767533-2122767596,2122767597-2122767680,2122767924-2122767939,2122768199-2122768340,2122768849-2122768927,2122769064-2122769314,2122769545-2122769587,2122770173-2122770362,2122770497-2122770512,2122770621-2122770664,2122770972-2122771104,2122771213-2122771311,2122771312-2122771531,2122771655-2122771673,2122771785-2122771806,2122771807-2122771853,2122771854-2122771889,2122771890-2122771917,2122777516-2122777676,2122777778-2122777789,2122779043-2122779098,2122779206-2122779405,2122779670-2122779698,2122779808-2122779872,2122779873-2122779875,2122780679-2122780755,2122780869-2122780957,2122780958-2122781066,2122781185-2122781339,2122781463-2122781484,2122781485-2122781504,2122781739-2122781740,2122781863-2122781878,2122781879-2122782019,2122782121-2122782244,2122783688-2122783812,2122783916-2122783936,2122784051-2122784122,2122784260-21227842
>>>
>> 95,
>
>>
>>>
>>> 2122784296-2122784530,2122784678-2122784812,2122784911-2122784913,2122785039-2122785220,2122785324-2122785412,2122785525-2122785662,2122786051-2122786341,2122786457-2122786503,2122786504-2122786787,2122787039-2122787093,2122787237-2122787431,2122787669-2122787812,2122787925-2122788059,2122788176-2122788295,2122788516-2122788579,2122788681-2122788812,2122788813-2122788914,2122789037-2122789229,2122789230-2122789459,2122789460-2122789502,2122791353-2122791552,2122791553-2122791614,2122791725-2122791824,2122791825-2122791949,2122792069-2122792213,2122792330-2122792341,2122792342-2122792383,2122792507-2122792851,2122793085-2122793089,2122793090-2122793224,2122793225-2122793347,2122793460-2122793516,2122793517-2122793643,2122793745-2122793782,2122793783-2122793949,2122793950-2122794070,2122794071-2122794085,2122794086-2122794245,2122794571-2122794662,2122794910-2122795296,2122795432-2122795701,2122795825-2122795891,2122796002-2122796074,2122796180-2122796269,2122796500-21227969
>>>
>> 28,
>
>>
>>>
>>> 2122797169-2122797296,2122797297-2122797544,2122797545-2122797548,2122797549-2122797626,2122799193-2122799247,2122799248-2122799406,2122799529-2122799658,2122799659-2122799703,2122799822-2122800053,2122800179-2122800309,2122800310-2122800427,2122800678-2122800703,2122800859-2122801458,2122801586-2122801679,2122801680-2122801703,2122801844-2122801857,2122801858-2122801916,2122802042-2122802098,2122802218-2122802703,2122802800-2122802938,2122803202-2122803333,2122803470-2122803703,2122803704-2122804012,2122804116-2122804148,2122804149-2122804316,2122804317-2122804436,2122804545-2122804561,2122805924-2122806012,2122806135-2122806667,2122806786-2122807012,2122807137-2122807159,2122807280-2122807791,2122807897-2122808012,2122808138-2122808389,2122809758-2122810047,2122810161-2122810240,2122810359-2122810880,2122811003-2122811165,2122811166-2122811240,2122811355-2122811381,2122811507-2122812119,2122813558-2122814389,2122814514-2122814568,2122815795-2122816389,2122816525-21228168
>>>
>> 22,
>
>>
>>>
>>> 2122819142-2122819389,2122819390-2122819484,2122819485-2122819820,2122819821-2122819937,2122820042-2122820134,2122821337-2122821937,2122821938-2122822223,2122823559-2122824464,2122824465-4000000000
>>>
>>>
>>> (~5K characters)
>>>
>>> DR
>>>
>>> On 12/02/2010 12:40 PM, Claudio Martella wrote:
>>>
>>>> so you keep a list of unused ID ranges in the ZK node's data?
>>>>
>>>>
>>>> On 12/2/10 6:21 PM, David Rosenstrauch wrote:
>>>>
>>>>> I don't, for several reasons:
>>>>>
>>>>> a) We request ID's frequently enough that old ones do eventually get
>>>>> used up.
>>>>>
>>>>> b) Even with a largely fragmented ID space, the largest we've seen the
>>>>> bytes contents of that ZK node get up to is 5-6KBytes.  So no real
>>>>> worries about either storage space or network I/O when reading/writing
>>>>> to/from the ZK node.
>>>>>
>>>>> DR
>>>>>
>>>>> On 12/02/2010 10:58 AM, Claudio Martella wrote:
>>>>>
>>>>>> I like Ted's idea too.
>>>>>>
>>>>>> David, how do you handle re-compaction of your fragmented ID space?
>>>>>>
>>>>>>
>>>>>> On 12/2/10 4:55 PM, David Rosenstrauch wrote:
>>>>>>
>>>>>>> On 12/02/2010 10:47 AM, Ted Dunning wrote:
>>>>>>>
>>>>>>>> I would recommend that you increment the counter by 100 or
1000 and
>>>>>>>> then
>>>>>>>> increment a local counter over the implied range.  This will
>>>>>>>> drive the
>>>>>>>> amortized ZK overhead down to tens of microseconds which
should be
>>>>>>>> good for
>>>>>>>> almost any application. Your final ids will still be almost
entirely
>>>>>>>> contiguous.  You could implement a fancier counter in ZK
that
>>>>>>>> remembers
>>>>>>>> returned chunks for re-use to get perfect contiguity if you
really
>>>>>>>> wanted
>>>>>>>> that.
>>>>>>>>
>>>>>>>
>>>>>>> This is what our library does.  You request chunks of, say, 1000
>>>>>>> ID's,
>>>>>>> and then push back any remaining unused ID's in the chunk you
took.
>>>>>>>
>>>>>>> DR
>>>>>>>
>>>>>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message