asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abdullah alamoudi <bamou...@gmail.com>
Subject Re: Weekly status
Date Sun, 24 Jan 2016 13:48:31 GMT
Yes, those two are fixed at this stage.
I am not sure I understand well what you mean by semantics but I will
explain how the filesystem feed will behave.
The user will have to provide the following (directories' locations,
expression to match files names against, formats of records). Once the
adapter is connected, it will start reading from the specified directories
until the directories are deleted or the feed is disconnected.
duplicate records will be skipped.

When there are no more files in the specified directories, the feed adapter
will push the buffered records to storage then wait for more events from
the file system.

Things that are not yet done properly:
1. at-least-once semantics.
2. logging of faulty records.
3. logging of duplicate records.

2 &3 are easy to implement correctly while 1 will take about a week to get
right with a good design and implementation.
Should I draft a new email?


Amoudi, Abdullah.

On Sun, Jan 24, 2016 at 3:14 AM, Mike Carey <dtabass@gmail.com> wrote:

> Q:  So are both of those (1 & 2) fixed?  And could you clarify the
> semantics that file feeds will have at this stage?  :-)
> Thx!
>
> On 1/22/16 11:40 PM, abdullah alamoudi wrote:
>
>> I think we are ready if they don't care much about at-least-once
>> semantics.
>>
>> Other than that, everything is ready.
>> The last communication we had with them was a promise to fix:
>> 1. Duplicate key exceptions
>> 2. Pushing last few records to storage.
>>
>> Cheers
>> Abdullah
>> Cool.  Are we ready to ping Wisconsin (Condor) again soon...?
>> (When did you last interact w/them - and where were things left...?)
>>
>> On 1/22/16 8:11 AM, abdullah alamoudi wrote:
>>
>> It looks like I will not be able to attend the weekly meeting today so I
>>> am
>>> sending my status here:
>>> 1. Addressed Young-Seok's comments on the Upsert change.
>>> 2. Completed streaming of Couchbase inserts into AsterixDB.
>>> 3. Completed the implementation of the flush () operation on all
>>> IFrameWriters.
>>> 4. Removed the ExternalLookup operator and merged it with
>>> UnnestMapOperator
>>> 5. Created a proposal for Feed changes.
>>>
>>> Cheers,
>>> Abdullah
>>>
>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message