zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Junqueira <...@apache.org>
Subject Re: How to extend ZooKeeper data structure without breaking the backward compatibility
Date Wed, 11 May 2016 21:13:35 GMT
We currently put a version in the directory name and afaict the version isn't used anywhere
else. We also have a version in FileHeader. 

The options I see are:

- We write a tool to convert the data from one format to another. In the case of upgrade,
we run the tool before starting the upgraded server. If we do it, then we will be simply moving
all the data from the old version to the new version.
- We can do the same thing as in the first bullet without the tool in the case the server
detects that the current data is in the old format. In this case, the serves converts it all
before it fully starts. This procedure will of course introduce some delay when starting an
upgraded server.
- Another way is to mix files with different versions and then let the server figure out how
to process the data with a given version.

Option 3 is a bit messier because we will be mixing files of different versions, but I'd be
interested in hearing opinions. Does it make sense at all?

-Flavio


> On 11 May 2016, at 18:06, Flavio P JUNQUEIRA <fpj@apache.org> wrote:
> 
> We need a file format version in this case.  Let me have look and get back
> to you.
> 
> -Flavio
> On 11 May 2016 15:49, "Mohammad arshad" <mohammad.arshad@huawei.com> wrote:
> 
> Loading snapshot file into memory(deserialization) does not involve any
> protocol version.
> are you suggesting version and dbId from FileHeader?
> Which field (version, dbId) would be more appropriate to decide which
> version ,StatPersisted or StatPersistedV0, to be deserialized. I think it
> should be the dbId.
> 
> Best Regards
> Mohammad Arshad
> HUAWEI TECHNOLOGIES CO.LTD.
> Huawei Tecnologies India Pvt. Ltd.
> Near EPIP Industrial Area, Kundalahalli Village
> Whitefield, Bangalore-560066
> www.huawei.com
> -----------------------------------------------------------------------------------------------------------------
> This e-mail and its attachments contain confidential information from
> HUAWEI, which
> is intended only for the person or entity whose address is listed above.
> Any use of the
> information contained herein in any way (including, but not limited to,
> total or partial
> disclosure, reproduction, or dissemination) by persons other than the
> intended
> recipient(s) is prohibited. If you receive this e-mail in error, please
> notify the sender by
> phone or email immediately and delete it!
> 
> 
> -----Original Message-----
> From: Flavio Junqueira [mailto:fpj@apache.org]
> Sent: 11 May 2016 18:18
> To: user@zookeeper.apache.org
> Cc: dev@zookeeper.apache.org
> Subject: Re: How to extend ZooKeeper data structure without breaking the
> backward compatibility
> 
> In this case, I guess we have to check the protocol version and depending
> on the version deserialize differently.
> 
> -Flavio
> 
>> On 11 May 2016, at 13:21, Mohammad arshad <mohammad.arshad@huawei.com>
> wrote:
>> 
>> Thanks Flavio,
>> 
>> I think CreateTxn kind of solution cannot be applied in this scenario
>> I can case of transaction the records are separable but in case of
>> snapshot records are continuous In this scenario, cuser
>> de-serialization eats into the next record which causes bigger problem
>> 
>> Here is exception I got after adding cuser and running the ZooKeeper
>> on old data
>> java.io.IOException: Unreasonable length = 796553071
>>      at
> org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
>>      at
> org.apache.jute.BinaryInputArchive.readString(BinaryInputArchive.java:81)
>>      at
> org.apache.zookeeper.data.StatPersisted.deserialize(StatPersisted.java:143)
>>      at
> org.apache.zookeeper.server.DataNode.deserialize(DataNode.java:173)
>>      at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
>>      at
>> org.apache.zookeeper.server.DataTree.deserialize(DataTree.java:1278)
>> 
>> 
>> Best Regards
>> Mohammad Arshad
>> HUAWEI TECHNOLOGIES CO.LTD.
>> Huawei Tecnologies India Pvt. Ltd.
>> Near EPIP Industrial Area, Kundalahalli Village Whitefield,
>> Bangalore-560066 www.huawei.com
>> ----------------------------------------------------------------------
>> -------------------------------------------
>> This e-mail and its attachments contain confidential information from
>> HUAWEI, which is intended only for the person or entity whose address
>> is listed above. Any use of the information contained herein in any
>> way (including, but not limited to, total or partial disclosure,
>> reproduction, or dissemination) by persons other than the intended
>> recipient(s) is prohibited. If you receive this e-mail in error,
>> please notify the sender by phone or email immediately and delete it!
>> 
>> -----Original Message-----
>> From: Flavio Junqueira [mailto:fpj@apache.org]
>> Sent: 11 May 2016 16:02
>> To: user@zookeeper.apache.org
>> Cc: Zookeeper
>> Subject: Re: How to extend ZooKeeper data structure without breaking
>> the backward compatibility
>> 
>> In the past, we simply created two versions of the data structure:
>> 
>>   class CreateTxnV0 {
>>       ustring path;
>>       buffer data;
>>       vector<org.apache.zookeeper.data.ACL> acl;
>>       boolean ephemeral;
>>   }
>>   class CreateTxn {
>>       ustring path;
>>       buffer data;
>>       vector<org.apache.zookeeper.data.ACL> acl;
>>       boolean ephemeral;
>>       int parentCVersion;
>>   }
>> 
>> and deal with it in the code. It is not ideal and the serialization
> framework is actually a pretty old one.
>> 
>> -Flavio
>> 
>>> On 11 May 2016, at 11:14, Mohammad arshad <mohammad.arshad@huawei.com>
> wrote:
>>> 
>>> Hello Everyone
>>> 
>>> is there any way to extend ZooKeeper data structure without breaking the
> backward compatibility.
>>> Suppose I want to add cuser field in StatPersisted class
>>> StatPersisted {
>>>      ........
>>>      ustring cuser;    //user who create the node
>>>  }
>>> This is fine with fresh ZooKeeper server installation, where
>>> ZooKeeper data is created freshly But this causes problem while
> de-serializing when server is new but points to old ZooKeeper data.
>>> Here by ZooKeeper data I mean snapshot and transaction log
>>> 
>>> In protocol buffers a field can be optional but in jute optional field
> is not supported.
>>> is there any way to overcome this jute limitation? any work around?
>>> 
>>> Best Regards
>>> Mohammad Arshad
>>> HUAWEI TECHNOLOGIES CO.LTD.
>>> Huawei Tecnologies India Pvt. Ltd.
>>> Near EPIP Industrial Area, Kundalahalli Village Whitefield,
>>> Bangalore-560066 www.huawei.com<http://www.huawei.com/>
>>> ---------------------------------------------------------------------
>>> -
>>> -------------------------------------------
>>> This e-mail and its attachments contain confidential information from
>>> HUAWEI, which is intended only for the person or entity whose address
>>> is listed above. Any use of the information contained herein in any
>>> way (including, but not limited to, total or partial disclosure,
>>> reproduction, or dissemination) by persons other than the intended
>>> recipient(s) is prohibited. If you receive this e-mail in error,
>>> please notify the sender by phone or email immediately and delete it!
>>> 
>> 


Mime
View raw message