Mailing-List: contact dev-help@asterixdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.apache.org
From: "Till Westmann" <tillw@apache.org>
To: dev@asterixdb.apache.org
Subject: Re: Temporary Datasets
Date: Mon, 20 Nov 2017 13:10:57 -0800
Message-ID: <E8D83849-8470-401A-A724-1FC6010E8992@apache.org>
In-Reply-To: <8e405afc-3cde-7afb-1756-fe57101d7ed7@gmail.com>
References: <F4F7762C-7FD1-4355-B37D-5D19A15CEA79@gmail.com>
 <8e405afc-3cde-7afb-1756-fe57101d7ed7@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
archived-at: Mon, 20 Nov 2017 21:11:02 -0000

+1

Till

On 20 Nov 2017, at 12:05, Mike Carey wrote:

> +1 to remove them and then re-create them later - based on the state 
> of the AsterixDB storage world and cluster dynamics at that time.  (I 
> think we'll have a better chance of getting them perfect if we re-do 
> them then - I don't remember that it took Yingyi very long to do them 
> the first time, so I think the re-do path will beat the fix-up path if 
> we want them again.)  As far as I know, since we don't document them, 
> nobody is using them - and I think the engineering cost of maintaining 
> orphaned code is too high (not worth it).
>
> Any thoughts to the contrary?
>
> Cheers,
>
> Mike
>
>
> On 11/20/17 10:52 AM, Murtadha Hubail wrote:
>> Hi all,
>>
>>
>> As you might be aware, we have a feature in AsterixDB to create 
>> temporary datasets that differ from regular datasets in some ways 
>> such as:
>> Their existence is not persisted in metadata, but only in the CC 
>> metadata cache.
>> They don’t’ generate any transaction logs
>> Their files are deleted on NC restart.
>> If they are not accessed for some period of time, their metadata 
>> records are removed from the CC metadata cache.
>>
>> Temporary datasets were originally introduced to serve as a staging 
>> area between AsterixDB and external systems such as Perglix. However, 
>> as the system evolved over the years, the assumptions they were built 
>> on don’t hold anymore and they could lead to undesired consequences 
>> such as leaking files after a CC restart or inability to access the 
>> dataset files on a restarted NC. Therefore, I’m proposing to remove 
>> the support for the current temporary datasets and we may add the 
>> feature with a careful design at a later stage.
>>
>>
>> Any thoughts or concerns on removing temporary datasets?
>>
>>
>> Cheers,
>>
>> Murtadha
>>
>>