Mailing-List: contact dev-help@asterixdb.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.incubator.apache.org
Subject: Re: Feeds UDF
To: dev@asterixdb.incubator.apache.org
References: <8CEA091E-1E8A-46AE-98F7-9EBE5FF9AEE0@gmail.com>
 <CAOQkTyP1d5wvvy2jLT9TTCOoYg1Ox97hJ31-z8iJEwH-kSZi4Q@mail.gmail.com>
From: Mike Carey <dtabass@gmail.com>
Message-ID: <5667DC12.3020800@gmail.com>
Date: Tue, 8 Dec 2015 23:45:22 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.4.0
MIME-Version: 1.0
In-Reply-To: 
 <CAOQkTyP1d5wvvy2jLT9TTCOoYg1Ox97hJ31-z8iJEwH-kSZi4Q@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------080405010800010105080706"

--------------080405010800010105080706
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

I'm confused:  Why is "self-join" (or any join) an issue?  I think the 
alledged case (:-)) against self-join is equivalent to a case against 
ever doing any queries against any data set under any circumstances 
where data is being inserted....  I don't think we want to restrict the 
system to only querying read-only datasets...

A feed lets you run a query against the system based on the contents of 
a current incoming record R.  Unless I am missing something (which is 
not unlikely because it's been a long day and I just got home from 
traveling :-)), this is equivalent to:

     let $r = ... (picture a constant constructor that yields the same 
content as R) ...
     return feed_processor ($r)

Right?  I.e., the new record R is not yet in the dataset - so - what's 
the issue?  What's special about this?

Cheers,
Mike

PS - Again, apologies if a long day has led to extra cluelessness on my 
part...


On 12/8/15 9:52 PM, abdullah alamoudi wrote:
> I think that we probably should restrict feed applied functions somehow
> (needs further thoughts and discussions) and I know for sure that we don't.
> As for the case you present, I would imagine that it could be allowed
> theoretically but I think everyone sees why it should be disallowed.
>
> One thing to keep in mind is that we introduce a materialize if the dataset
> was part of an insert pipeline. Now think about how this would work with a
> continuous feed. One choice would be that the feed will materialize all
> records to be inserted and once the feed stops, it would start inserting
> them but I still think we should not allow it.
>
> My 2c,
> Any opposing argument?
>
>
> Amoudi, Abdullah.
>
> On Tue, Dec 8, 2015 at 6:28 PM, Ildar Absalyamov <ildar.absalyamov@gmail.com
>> wrote:
>> Hi All,
>>
>> As a part of feed ingestion we do allow preprocessing incoming data with
>> AQL UDFs.
>> I was wondering if we somehow restrict the kind of UDFs that could be
>> used? Do we allow joins in these UDFs? Especially joins with the same
>> dataset, which is used for intake. Ex:
>>
>> create type TweetType as open {
>>    id: string,
>>    username : string,
>>    location : string,
>>    text : string,
>>    timestamp : string
>> }
>> create dataset Tweets(TweetType)
>> primary key id;
>> create function feed_processor($x) {
>> for $y in dataset Tweets
>> // self-join with Tweets dataset on some predicate($x, $y)
>> return $y
>> }
>> create feed TweetFeed
>> apply function feed_processor;
>>
>> The query above fails in runtime, but I was wondering if that
>> theoretically could work at all.
>>
>> Best regards,
>> Ildar
>>
>>


--------------080405010800010105080706--