asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Till Westmann" <ti...@apache.org>
Subject Re: Temporary Datasets
Date Mon, 20 Nov 2017 21:10:57 GMT
+1

Till

On 20 Nov 2017, at 12:05, Mike Carey wrote:

> +1 to remove them and then re-create them later - based on the state 
> of the AsterixDB storage world and cluster dynamics at that time.  (I 
> think we'll have a better chance of getting them perfect if we re-do 
> them then - I don't remember that it took Yingyi very long to do them 
> the first time, so I think the re-do path will beat the fix-up path if 
> we want them again.)  As far as I know, since we don't document them, 
> nobody is using them - and I think the engineering cost of maintaining 
> orphaned code is too high (not worth it).
>
> Any thoughts to the contrary?
>
> Cheers,
>
> Mike
>
>
> On 11/20/17 10:52 AM, Murtadha Hubail wrote:
>> Hi all,
>>
>>
>> As you might be aware, we have a feature in AsterixDB to create 
>> temporary datasets that differ from regular datasets in some ways 
>> such as:
>> Their existence is not persisted in metadata, but only in the CC 
>> metadata cache.
>> They don’t’ generate any transaction logs
>> Their files are deleted on NC restart.
>> If they are not accessed for some period of time, their metadata 
>> records are removed from the CC metadata cache.
>>
>> Temporary datasets were originally introduced to serve as a staging 
>> area between AsterixDB and external systems such as Perglix. However, 
>> as the system evolved over the years, the assumptions they were built 
>> on don’t hold anymore and they could lead to undesired consequences 
>> such as leaking files after a CC restart or inability to access the 
>> dataset files on a restarted NC. Therefore, I’m proposing to remove 
>> the support for the current temporary datasets and we may add the 
>> feature with a careful design at a later stage.
>>
>>
>> Any thoughts or concerns on removing temporary datasets?
>>
>>
>> Cheers,
>>
>> Murtadha
>>
>>

Mime
View raw message