Mailing-List: contact dev-help@asterixdb.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <56A68256.2000700@gmail.com>
References: 
 <CAOQkTyMWOV4Mdp-TYS-KyLp5Yp1oqAi16-TOsTFz_M6djCzM0A@mail.gmail.com>
	<CAOQkTyPxuzwMUS0gxFbO3K4p+PCWD90crBcLf8gS_7ticXwO0Q@mail.gmail.com>
	<56A2703E.6090105@gmail.com>
	<CAOQkTyPwCOLr6hj16jPvamoyLzC3VVyWZJ+wCqVrD-EGk==TkQ@mail.gmail.com>
	<56A4093F.6000702@gmail.com>
	<CAOQkTyNuQKtc7XrH3fKfyM-A8b9erXGB=hEdDKqoiCrHjYv4_A@mail.gmail.com>
	<56A68256.2000700@gmail.com>
Date: Tue, 26 Jan 2016 00:41:24 +0400
Message-ID: 
 <CAOQkTyMXt85uyiziwK79kxLs8=2E0sX9zmO71fmmyZj03CS1TQ@mail.gmail.com>
Subject: Re: Weekly status
From: abdullah alamoudi <bamousaa@gmail.com>
To: dev@asterixdb.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a1146ed2c9efe57052a2e9720

--001a1146ed2c9efe57052a2e9720
Content-Type: text/plain; charset=UTF-8

Answers Inlined.

Cheers,
Abdullah.

Amoudi, Abdullah.

On Tue, Jan 26, 2016 at 12:15 AM, Mike Carey <dtabass@gmail.com> wrote:

> So my remaining Q's (before we e-mail anyone) would be:
> 1. Do we possibly drop some data?  (Still trying to
> make sure I understand the implications of your first
> point below.)
>
No but we don't know when the record was actually persisted and so, there
is a chance that we record a file as has been read completely while there
is a small window that allows a crash before persistence.

> 2. Do we (silently) ignore faulty records for now?
>
Yes.

> 3. Do we (silently) ignore duplicate records for now?
>
Yes.

> Thx!
> Mike
>
> On 1/24/16 5:48 AM, abdullah alamoudi wrote:
>
>> Yes, those two are fixed at this stage.
>> I am not sure I understand well what you mean by semantics but I will
>> explain how the filesystem feed will behave.
>> The user will have to provide the following (directories' locations,
>> expression to match files names against, formats of records). Once the
>> adapter is connected, it will start reading from the specified directories
>> until the directories are deleted or the feed is disconnected.
>> duplicate records will be skipped.
>>
>> When there are no more files in the specified directories, the feed
>> adapter
>> will push the buffered records to storage then wait for more events from
>> the file system.
>>
>> Things that are not yet done properly:
>> 1. at-least-once semantics.
>> 2. logging of faulty records.
>> 3. logging of duplicate records.
>>
>> 2 &3 are easy to implement correctly while 1 will take about a week to get
>> right with a good design and implementation.
>> Should I draft a new email?
>>
>>
>> Amoudi, Abdullah.
>>
>> On Sun, Jan 24, 2016 at 3:14 AM, Mike Carey <dtabass@gmail.com> wrote:
>>
>> Q:  So are both of those (1 & 2) fixed?  And could you clarify the
>>> semantics that file feeds will have at this stage?  :-)
>>> Thx!
>>>
>>> On 1/22/16 11:40 PM, abdullah alamoudi wrote:
>>>
>>> I think we are ready if they don't care much about at-least-once
>>>> semantics.
>>>>
>>>> Other than that, everything is ready.
>>>> The last communication we had with them was a promise to fix:
>>>> 1. Duplicate key exceptions
>>>> 2. Pushing last few records to storage.
>>>>
>>>> Cheers
>>>> Abdullah
>>>> Cool.  Are we ready to ping Wisconsin (Condor) again soon...?
>>>> (When did you last interact w/them - and where were things left...?)
>>>>
>>>> On 1/22/16 8:11 AM, abdullah alamoudi wrote:
>>>>
>>>> It looks like I will not be able to attend the weekly meeting today so I
>>>>
>>>>> am
>>>>> sending my status here:
>>>>> 1. Addressed Young-Seok's comments on the Upsert change.
>>>>> 2. Completed streaming of Couchbase inserts into AsterixDB.
>>>>> 3. Completed the implementation of the flush () operation on all
>>>>> IFrameWriters.
>>>>> 4. Removed the ExternalLookup operator and merged it with
>>>>> UnnestMapOperator
>>>>> 5. Created a proposal for Feed changes.
>>>>>
>>>>> Cheers,
>>>>> Abdullah
>>>>>
>>>>>
>>>>>
>>>>>
>

--001a1146ed2c9efe57052a2e9720--