Mailing-List: contact dev-help@asterixdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.apache.org
From: "Till Westmann" <tillw@apache.org>
To: dev@asterixdb.apache.org
Subject: Re: Temporary Datasets
Date: Mon, 20 Nov 2017 13:13:50 -0800
Message-ID: <2702DF13-3B8F-4DA9-8F4D-CAD175B48FFA@apache.org>
In-Reply-To: <CAN_YF5xduCuBOXOd-jA7MTE+ioVE3RpYDZkT=+ExtVbpWnFbeQ@mail.gmail.com>
References: <F4F7762C-7FD1-4355-B37D-5D19A15CEA79@gmail.com>
 <CAFUTFeaKG0-fYgVNPJJV_dRoJ7avPN8VToY6vzmT-wBUdA2NSw@mail.gmail.com>
 <CAN_YF5xduCuBOXOd-jA7MTE+ioVE3RpYDZkT=+ExtVbpWnFbeQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
archived-at: Mon, 20 Nov 2017 21:13:54 -0000

Good question - we might. If we take it out now, we could break future
applications if we reintroduce it. Now I’m torn between not keeping
unused stuff and compatibility.
In this case I’m leaning slightly towards compatibility …

Cheers,
Till

On 20 Nov 2017, at 12:41, Ian Maxon wrote:

> Are we going to preserve the keyword?
>
> On Mon, Nov 20, 2017 at 12:06 PM, Michael Blow 
> <mblow.apache@gmail.com> wrote:
>>>> removing temporary datasets?
>>
>> +1
>>
>>
>> On Mon, Nov 20, 2017 at 1:52 PM Murtadha Hubail <hubailmor@gmail.com> 
>> wrote:
>>
>>> Hi all,
>>>
>>>
>>>
>>> As you might be aware, we have a feature in AsterixDB to create 
>>> temporary
>>> datasets that differ from regular datasets in some ways such as:
>>> Their existence is not persisted in metadata, but only in the CC 
>>> metadata
>>> cache.
>>> They don’t’ generate any transaction logs
>>> Their files are deleted on NC restart.
>>> If they are not accessed for some period of time, their metadata 
>>> records
>>> are removed from the CC metadata cache.
>>>
>>>
>>> Temporary datasets were originally introduced to serve as a staging 
>>> area
>>> between AsterixDB and external systems such as Perglix. However, as 
>>> the
>>> system evolved over the years, the assumptions they were built on 
>>> don’t
>>> hold anymore and they could lead to undesired consequences such as 
>>> leaking
>>> files after a CC restart or inability to access the dataset files on 
>>> a
>>> restarted NC. Therefore, I’m proposing to remove the support for 
>>> the
>>> current temporary datasets and we may add the feature with a careful 
>>> design
>>> at a later stage.
>>>
>>>
>>>
>>> Any thoughts or concerns on removing temporary datasets?
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Murtadha
>>>
>>>