asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abdullah alamoudi <bamou...@gmail.com>
Subject Re: Feeds UDF
Date Wed, 09 Dec 2015 05:52:11 GMT
I think that we probably should restrict feed applied functions somehow
(needs further thoughts and discussions) and I know for sure that we don't.
As for the case you present, I would imagine that it could be allowed
theoretically but I think everyone sees why it should be disallowed.

One thing to keep in mind is that we introduce a materialize if the dataset
was part of an insert pipeline. Now think about how this would work with a
continuous feed. One choice would be that the feed will materialize all
records to be inserted and once the feed stops, it would start inserting
them but I still think we should not allow it.

My 2c,
Any opposing argument?


Amoudi, Abdullah.

On Tue, Dec 8, 2015 at 6:28 PM, Ildar Absalyamov <ildar.absalyamov@gmail.com
> wrote:

> Hi All,
>
> As a part of feed ingestion we do allow preprocessing incoming data with
> AQL UDFs.
> I was wondering if we somehow restrict the kind of UDFs that could be
> used? Do we allow joins in these UDFs? Especially joins with the same
> dataset, which is used for intake. Ex:
>
> create type TweetType as open {
>   id: string,
>   username : string,
>   location : string,
>   text : string,
>   timestamp : string
> }
> create dataset Tweets(TweetType)
> primary key id;
> create function feed_processor($x) {
> for $y in dataset Tweets
> // self-join with Tweets dataset on some predicate($x, $y)
> return $y
> }
> create feed TweetFeed
> apply function feed_processor;
>
> The query above fails in runtime, but I was wondering if that
> theoretically could work at all.
>
> Best regards,
> Ildar
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message