Mailing-List: contact dev-help@asterixdb.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.incubator.apache.org
Subject: Re: Feeds UDF
To: dev@asterixdb.incubator.apache.org
References: <8CEA091E-1E8A-46AE-98F7-9EBE5FF9AEE0@gmail.com>
From: Mike Carey <dtabass@gmail.com>
Message-ID: <5667D9FD.4040202@gmail.com>
Date: Tue, 8 Dec 2015 23:36:29 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0)
 Gecko/20100101 Thunderbird/38.4.0
MIME-Version: 1.0
In-Reply-To: <8CEA091E-1E8A-46AE-98F7-9EBE5FF9AEE0@gmail.com>
Content-Type: multipart/alternative;
 boundary="------------030507060203000001020800"

--------------030507060203000001020800
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit

I have the impression that AQL UDFs are poorly tested right now, so this 
failure may be unrelated to the sameness of the dataset.
We don't have any restrictions currently, and I'm not sure we need to - 
joins are an important use case, actually.
(Not self joins specifically, but joins for sure.)

Cheers,
Mike


On 12/8/15 6:28 PM, Ildar Absalyamov wrote:
> Hi All,
>
> As a part of feed ingestion we do allow preprocessing incoming data with AQL UDFs.
> I was wondering if we somehow restrict the kind of UDFs that could be used? Do we allow joins in these UDFs? Especially joins with the same dataset, which is used for intake. Ex:
>
> create type TweetType as open {
>    id: string,
>    username : string,
>    location : string,
>    text : string,
>    timestamp : string
> }
> create dataset Tweets(TweetType)
> primary key id;
> create function feed_processor($x) {
> for $y in dataset Tweets
> // self-join with Tweets dataset on some predicate($x, $y)
> return $y
> }
> create feed TweetFeed
> apply function feed_processor;
>
> The query above fails in runtime, but I was wondering if that theoretically could work at all.
>
> Best regards,
> Ildar
>


--------------030507060203000001020800--