asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wail Alkowaileet <wael....@gmail.com>
Subject Re: Force LSM component flush & NC-CC messaging ACK
Date Sat, 21 Jan 2017 16:51:15 GMT
I remember one reason to enforce flush is for Preglix connector [1][2][3].

For the messaging framework, I believe that you probably have the same
issue I had. I did what Till has suggested as it is guaranteed by the
robustness of AsterixDB and not the user who might kill the process anyway.

[1]
https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/servlet/ConnectorAPIServlet.java
[2]
https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-app/src/main/java/org/apache/asterix/util/FlushDatasetUtils.java
[3]
https://github.com/apache/asterixdb/blob/2f9d4c3ab4d55598fe9a14fbf28faef12bed208b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/std/FlushDatasetOperatorDescriptor.java

On Sat, Jan 21, 2017 at 7:17 PM, Mike Carey <dtabass@gmail.com> wrote:

> I believe Ildar is just looking for a way to ensure, in doing experiments,
> that things are all in disk components.  His stats-gathering extensions
> camp on the LSM lifecycle - flushes in particular - and he wants to finish
> that process in his testing and experiments.  Wail's schema inference stuff
> has a similar flavor.  So the goal is to flush any lingering memory
> components to disk for a given dataset at the end of the "experiment
> lifecycle".
>
> We have DDL to compact a dataset - which flushes AND compacts - it might
> also be useful to have DDL to flush a dataset without also forcing
> compaction - as a way for an administrator to release that dataset's
> in-memory component related resources.  (Not that it's "necessary" for any
> correctness reason - just might be nice to be able to do that.  That could
> also be useful in scripting more user-level-oriented recovery tests.)
>
> Thus, I'd likely vote for adding a harmless new DDL statement - another
> arm of the one that supports compaction - for this.
>
> Cheers,
>
> Mike
>
>
>
> On 1/21/17 6:21 AM, Till Westmann wrote:
>
>> Hi Ildar,
>>
>> On 19 Jan 2017, at 4:02, Ildar Absalyamov wrote:
>>
>> Since I was out for quite a while and a lot of things happened in a
>>> meantime in a codebase I wanted to clarify couple of things.
>>>
>>> I was wondering if there is any legitimate way to force the data of
>>> in-memory components to be flushed, other then stop the whole instance?
>>> It used to be that choosing a different default dataverse with “use”
>>> statement did that trick, but that is not the case anymore.
>>>
>>
>> Just wondering, why do you want to flush the in-memory components to disk?
>>
>> Another question is regarding CC<->NC & NC<->NC messaging. Does the
>>> sender get some kind of ACK that the message was received by the addressee?
>>> Say if I send a message just before the instance shutdown will the shutdown
>>> hook wait until the message is delivered and processed?
>>>
>>
>> I agree with Murtadha, that I can certainly be done. However, we also
>> need to assume that some shutdowns won’t be clean and so the messages might
>> not be received. So it might be easier to just be able to recover from
>> missing messages than to be able to recover *and* to synchronize on
>> shutdown. Just a thought - maybe that’s not even an issue for your use-case.
>>
>> Cheers,
>> Till
>>
>
>


-- 

*Regards,*
Wail Alkowaileet

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message