asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abdullah alamoudi <bamou...@gmail.com>
Subject Re: Weekly status
Date Mon, 25 Jan 2016 20:41:24 GMT
Answers Inlined.

Cheers,
Abdullah.

Amoudi, Abdullah.

On Tue, Jan 26, 2016 at 12:15 AM, Mike Carey <dtabass@gmail.com> wrote:

> So my remaining Q's (before we e-mail anyone) would be:
> 1. Do we possibly drop some data?  (Still trying to
> make sure I understand the implications of your first
> point below.)
>
No but we don't know when the record was actually persisted and so, there
is a chance that we record a file as has been read completely while there
is a small window that allows a crash before persistence.

> 2. Do we (silently) ignore faulty records for now?
>
Yes.

> 3. Do we (silently) ignore duplicate records for now?
>
Yes.

> Thx!
> Mike
>
> On 1/24/16 5:48 AM, abdullah alamoudi wrote:
>
>> Yes, those two are fixed at this stage.
>> I am not sure I understand well what you mean by semantics but I will
>> explain how the filesystem feed will behave.
>> The user will have to provide the following (directories' locations,
>> expression to match files names against, formats of records). Once the
>> adapter is connected, it will start reading from the specified directories
>> until the directories are deleted or the feed is disconnected.
>> duplicate records will be skipped.
>>
>> When there are no more files in the specified directories, the feed
>> adapter
>> will push the buffered records to storage then wait for more events from
>> the file system.
>>
>> Things that are not yet done properly:
>> 1. at-least-once semantics.
>> 2. logging of faulty records.
>> 3. logging of duplicate records.
>>
>> 2 &3 are easy to implement correctly while 1 will take about a week to get
>> right with a good design and implementation.
>> Should I draft a new email?
>>
>>
>> Amoudi, Abdullah.
>>
>> On Sun, Jan 24, 2016 at 3:14 AM, Mike Carey <dtabass@gmail.com> wrote:
>>
>> Q:  So are both of those (1 & 2) fixed?  And could you clarify the
>>> semantics that file feeds will have at this stage?  :-)
>>> Thx!
>>>
>>> On 1/22/16 11:40 PM, abdullah alamoudi wrote:
>>>
>>> I think we are ready if they don't care much about at-least-once
>>>> semantics.
>>>>
>>>> Other than that, everything is ready.
>>>> The last communication we had with them was a promise to fix:
>>>> 1. Duplicate key exceptions
>>>> 2. Pushing last few records to storage.
>>>>
>>>> Cheers
>>>> Abdullah
>>>> Cool.  Are we ready to ping Wisconsin (Condor) again soon...?
>>>> (When did you last interact w/them - and where were things left...?)
>>>>
>>>> On 1/22/16 8:11 AM, abdullah alamoudi wrote:
>>>>
>>>> It looks like I will not be able to attend the weekly meeting today so I
>>>>
>>>>> am
>>>>> sending my status here:
>>>>> 1. Addressed Young-Seok's comments on the Upsert change.
>>>>> 2. Completed streaming of Couchbase inserts into AsterixDB.
>>>>> 3. Completed the implementation of the flush () operation on all
>>>>> IFrameWriters.
>>>>> 4. Removed the ExternalLookup operator and merged it with
>>>>> UnnestMapOperator
>>>>> 5. Created a proposal for Feed changes.
>>>>>
>>>>> Cheers,
>>>>> Abdullah
>>>>>
>>>>>
>>>>>
>>>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message