asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Carey <dtab...@gmail.com>
Subject Re: Weekly status
Date Mon, 25 Jan 2016 20:15:18 GMT
So my remaining Q's (before we e-mail anyone) would be:
1. Do we possibly drop some data?  (Still trying to
make sure I understand the implications of your first
point below.)
2. Do we (silently) ignore faulty records for now?
3. Do we (silently) ignore duplicate records for now?
Thx!
Mike

On 1/24/16 5:48 AM, abdullah alamoudi wrote:
> Yes, those two are fixed at this stage.
> I am not sure I understand well what you mean by semantics but I will
> explain how the filesystem feed will behave.
> The user will have to provide the following (directories' locations,
> expression to match files names against, formats of records). Once the
> adapter is connected, it will start reading from the specified directories
> until the directories are deleted or the feed is disconnected.
> duplicate records will be skipped.
>
> When there are no more files in the specified directories, the feed adapter
> will push the buffered records to storage then wait for more events from
> the file system.
>
> Things that are not yet done properly:
> 1. at-least-once semantics.
> 2. logging of faulty records.
> 3. logging of duplicate records.
>
> 2 &3 are easy to implement correctly while 1 will take about a week to get
> right with a good design and implementation.
> Should I draft a new email?
>
>
> Amoudi, Abdullah.
>
> On Sun, Jan 24, 2016 at 3:14 AM, Mike Carey <dtabass@gmail.com> wrote:
>
>> Q:  So are both of those (1 & 2) fixed?  And could you clarify the
>> semantics that file feeds will have at this stage?  :-)
>> Thx!
>>
>> On 1/22/16 11:40 PM, abdullah alamoudi wrote:
>>
>>> I think we are ready if they don't care much about at-least-once
>>> semantics.
>>>
>>> Other than that, everything is ready.
>>> The last communication we had with them was a promise to fix:
>>> 1. Duplicate key exceptions
>>> 2. Pushing last few records to storage.
>>>
>>> Cheers
>>> Abdullah
>>> Cool.  Are we ready to ping Wisconsin (Condor) again soon...?
>>> (When did you last interact w/them - and where were things left...?)
>>>
>>> On 1/22/16 8:11 AM, abdullah alamoudi wrote:
>>>
>>> It looks like I will not be able to attend the weekly meeting today so I
>>>> am
>>>> sending my status here:
>>>> 1. Addressed Young-Seok's comments on the Upsert change.
>>>> 2. Completed streaming of Couchbase inserts into AsterixDB.
>>>> 3. Completed the implementation of the flush () operation on all
>>>> IFrameWriters.
>>>> 4. Removed the ExternalLookup operator and merged it with
>>>> UnnestMapOperator
>>>> 5. Created a proposal for Feed changes.
>>>>
>>>> Cheers,
>>>> Abdullah
>>>>
>>>>
>>>>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message