Mailing-List: contact dev-help@asterixdb.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.incubator.apache.org
Subject: Re: Metadata changes
To: dev@asterixdb.incubator.apache.org
References: 
 <CAE1xAnspZx_HGgUUqT0PrQEftNTsm5L9JQNbNbo-2WDTq-LuMA@mail.gmail.com>
 <CAN_YF5yjaAUb-LkghiVeVTHiN5Mf2Jiy4HhOTMM+iC4LF81P9g@mail.gmail.com>
 <CAE1xAnvC0oGzvuOqkuOd+jF-Yt0pPU1WmKSKKMTJY7UxD+M7YQ@mail.gmail.com>
 <81DE960F-C90C-4912-87EF-3E3C533EE3D2@gmail.com>
 <CAE1xAnuGGat1K5rg3jZ3c_sG=MXp_3foAytmummcY+=eN_aYYQ@mail.gmail.com>
 <19C5337A-A9D1-4D04-910E-0FE6E477F0A8@apache.org>
From: Mike Carey <dtabass@gmail.com>
Message-ID: <566F3272.1030801@gmail.com>
Date: Mon, 14 Dec 2015 13:19:46 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.4.0
MIME-Version: 1.0
In-Reply-To: <19C5337A-A9D1-4D04-910E-0FE6E477F0A8@apache.org>
Content-Type: multipart/alternative;
 boundary="------------070503000109000906080606"

--------------070503000109000906080606
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

I think we started out as a bunch of relation-heads, modeling things 
after the system catalogs found in our favorite RDBMSs.  Their record 
types are open AFAIK - but I think Steven was saying that it feels like 
this is a use case whose field(s) should be predeclared (and using the 
term "closed field" to mean predeclared field).  I agree.  I think to 
the extent that we can anticipate the need for a given bit of metadata, 
it's useful to have it be there in the known schema; but, when we move 
into the era when we cannot make changes to the known schema part of the 
catalogs, because we have dependent customers who'd be hurt by that, we 
will start adding more fields that are not predeclared but only 
documented.  The schema for the metadata is really there more as 
documentation than as being "truly necessary"...

When we first started out we were just not thinking in an "open-minded" 
way on the catalog side of things.....  :-)

Cheers,
Mike

On 12/14/15 11:53 AM, Till Westmann wrote:
> Let me prefix this with that statement that I haven’t looked into our 
> metadata storage implementation.
> But looking at the relatively frequent changes to the metadata format 
> it seems that we should either
> a) make the types for metadata records open or
> b) be very careful in designing the next (and future) revision(s) of it.
> I don’t know why the metadata storage was done the way it is now and 
> what the considerations were when we did that.
>
> Does anybody remember those (or is there even a document that contains 
> the design and rationale)?
>
> Thanks,
> Till
>
> On 14 Dec 2015, at 11:35, Steven Jacobs wrote:
>
>> It could easily be done as reverse-compatible, but my thinking was that
>> this is the "wrong" choice. I can easily make the datatype dataverse an
>> open field for the metadata. The question is, why do we have open vs 
>> closed
>> fields for metadata at all? If it is okay for them to be open, should we
>> get rid of the schema entirely? if it's not okay, then shouldn't this 
>> field
>> be closed? From a design standpoint it seems that if reverse 
>> compatibility
>> were not an issue than the field should be closed.
>> Steven
>>
>> On Mon, Dec 14, 2015 at 11:30 AM, Ildar Absalyamov <
>> ildar.absalyamov@gmail.com> wrote:
>>
>>> If fix for 2) will break the backwards-compatibility 1) might do 
>>> that as
>>> well and be submitted together.
>>>
>>> Now 2) was a long overdue problem, I don’t think there is any reason 
>>> even
>>> to try make changes backwards-compatible, because it was broken in the
>>> first place.
>>>
>>>> On Dec 14, 2015, at 11:16, Steven Jacobs <sjaco002@ucr.edu> wrote:
>>>>
>>>> It's a new attribute, but it's a closed field, which means it isn't
>>>> backwards compatible.
>>>> Steven
>>>>
>>>> On Mon, Dec 14, 2015 at 11:13 AM, Ian Maxon <imaxon@uci.edu> wrote:
>>>>
>>>>> For 1), I guess the question is whether it would be a backwards
>>>>> compatible change. Since it's just a new attribute (right?...), 
>>>>> and it
>>>>> is also sort of a new feature rather than a fix for something that 
>>>>> was
>>>>> critically broken, I would tend toward putting it on master. If it's
>>>>> not backwards compatible though maybe it needs more careful
>>>>> consideration.
>>>>>
>>>>> -Ian
>>>>>
>>>>> On Mon, Dec 14, 2015 at 10:12 AM, Steven Jacobs <sjaco002@ucr.edu>
>>> wrote:
>>>>>> Hi all,
>>>>>> I'm implementing a change so that datasets can use datatypes from
>>>>> alternate
>>>>>> data verses (previously the type and set had to be from the same
>>>>>> dataverse). Unfortunately this means another change for Dataset
>>> Metadata
>>>>>> (which will now store the dataverse for its type).
>>>>>>
>>>>>> As such, I had a couple of questions:
>>>>>>
>>>>>> 1) Should this change be thrown into the release branch, as it is
>>> another
>>>>>> Metadata change?
>>>>>>
>>>>>> 2) In implementing this change, I've been looking at the Metadata
>>>>> secondary
>>>>>> indexes. I had a discussion with Ildar, and it seems the thread on
>>>>> Metadata
>>>>>> secondary indexes being "hacked" has been lost. Is this also 
>>>>>> something
>>>>> that
>>>>>> should get into the release? Is there anyone currently looking at 
>>>>>> it?
>>>>>>
>>>>>> Steven
>>>>>
>>>
>>> Best regards,
>>> Ildar
>>>
>>>


--------------070503000109000906080606--