asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Till Westmann" <>
Subject Re: Metadata changes
Date Mon, 14 Dec 2015 19:53:41 GMT
Let me prefix this with that statement that I haven’t looked into our 
metadata storage implementation.
But looking at the relatively frequent changes to the metadata format it 
seems that we should either
a) make the types for metadata records open or
b) be very careful in designing the next (and future) revision(s) of it.
I don’t know why the metadata storage was done the way it is now and 
what the considerations were when we did that.

Does anybody remember those (or is there even a document that contains 
the design and rationale)?


On 14 Dec 2015, at 11:35, Steven Jacobs wrote:

> It could easily be done as reverse-compatible, but my thinking was 
> that
> this is the "wrong" choice. I can easily make the datatype dataverse 
> an
> open field for the metadata. The question is, why do we have open vs 
> closed
> fields for metadata at all? If it is okay for them to be open, should 
> we
> get rid of the schema entirely? if it's not okay, then shouldn't this 
> field
> be closed? From a design standpoint it seems that if reverse 
> compatibility
> were not an issue than the field should be closed.
> Steven
> On Mon, Dec 14, 2015 at 11:30 AM, Ildar Absalyamov <
>> wrote:
>> If fix for 2) will break the backwards-compatibility 1) might do that 
>> as
>> well and be submitted together.
>> Now 2) was a long overdue problem, I don’t think there is any 
>> reason even
>> to try make changes backwards-compatible, because it was broken in 
>> the
>> first place.
>>> On Dec 14, 2015, at 11:16, Steven Jacobs <> wrote:
>>> It's a new attribute, but it's a closed field, which means it isn't
>>> backwards compatible.
>>> Steven
>>> On Mon, Dec 14, 2015 at 11:13 AM, Ian Maxon <> wrote:
>>>> For 1), I guess the question is whether it would be a backwards
>>>> compatible change. Since it's just a new attribute (right?...), and 
>>>> it
>>>> is also sort of a new feature rather than a fix for something that 
>>>> was
>>>> critically broken, I would tend toward putting it on master. If 
>>>> it's
>>>> not backwards compatible though maybe it needs more careful
>>>> consideration.
>>>> -Ian
>>>> On Mon, Dec 14, 2015 at 10:12 AM, Steven Jacobs <>
>> wrote:
>>>>> Hi all,
>>>>> I'm implementing a change so that datasets can use datatypes from
>>>> alternate
>>>>> data verses (previously the type and set had to be from the same
>>>>> dataverse). Unfortunately this means another change for Dataset
>> Metadata
>>>>> (which will now store the dataverse for its type).
>>>>> As such, I had a couple of questions:
>>>>> 1) Should this change be thrown into the release branch, as it is
>> another
>>>>> Metadata change?
>>>>> 2) In implementing this change, I've been looking at the Metadata
>>>> secondary
>>>>> indexes. I had a discussion with Ildar, and it seems the thread on
>>>> Metadata
>>>>> secondary indexes being "hacked" has been lost. Is this also 
>>>>> something
>>>> that
>>>>> should get into the release? Is there anyone currently looking at 
>>>>> it?
>>>>> Steven
>> Best regards,
>> Ildar

View raw message