asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ildar Absalyamov <ildar.absalya...@gmail.com>
Subject Re: Feeds UDF
Date Wed, 09 Dec 2015 06:11:37 GMT
Abdullah,

OK, now I see what problems it will cause.
Kinda related question: could the feed implement “upsert” semantics, that you’ve been
working on, instead of “insert” semantics?

> On Dec 8, 2015, at 21:52, abdullah alamoudi <bamousaa@gmail.com> wrote:
> 
> I think that we probably should restrict feed applied functions somehow
> (needs further thoughts and discussions) and I know for sure that we don't.
> As for the case you present, I would imagine that it could be allowed
> theoretically but I think everyone sees why it should be disallowed.
> 
> One thing to keep in mind is that we introduce a materialize if the dataset
> was part of an insert pipeline. Now think about how this would work with a
> continuous feed. One choice would be that the feed will materialize all
> records to be inserted and once the feed stops, it would start inserting
> them but I still think we should not allow it.
> 
> My 2c,
> Any opposing argument?
> 
> 
> Amoudi, Abdullah.
> 
> On Tue, Dec 8, 2015 at 6:28 PM, Ildar Absalyamov <ildar.absalyamov@gmail.com
>> wrote:
> 
>> Hi All,
>> 
>> As a part of feed ingestion we do allow preprocessing incoming data with
>> AQL UDFs.
>> I was wondering if we somehow restrict the kind of UDFs that could be
>> used? Do we allow joins in these UDFs? Especially joins with the same
>> dataset, which is used for intake. Ex:
>> 
>> create type TweetType as open {
>>  id: string,
>>  username : string,
>>  location : string,
>>  text : string,
>>  timestamp : string
>> }
>> create dataset Tweets(TweetType)
>> primary key id;
>> create function feed_processor($x) {
>> for $y in dataset Tweets
>> // self-join with Tweets dataset on some predicate($x, $y)
>> return $y
>> }
>> create feed TweetFeed
>> apply function feed_processor;
>> 
>> The query above fails in runtime, but I was wondering if that
>> theoretically could work at all.
>> 
>> Best regards,
>> Ildar
>> 
>> 

Best regards,
Ildar


Mime
View raw message