Return-Path: X-Original-To: apmail-asterixdb-users-archive@minotaur.apache.org Delivered-To: apmail-asterixdb-users-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F0018181A7 for ; Fri, 30 Oct 2015 07:30:22 +0000 (UTC) Received: (qmail 89625 invoked by uid 500); 30 Oct 2015 07:30:22 -0000 Delivered-To: apmail-asterixdb-users-archive@asterixdb.apache.org Received: (qmail 89597 invoked by uid 500); 30 Oct 2015 07:30:22 -0000 Mailing-List: contact users-help@asterixdb.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@asterixdb.incubator.apache.org Delivered-To: mailing list users@asterixdb.incubator.apache.org Received: (qmail 89577 invoked by uid 99); 30 Oct 2015 07:30:22 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Oct 2015 07:30:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 2F11DC0098; Fri, 30 Oct 2015 07:30:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.879 X-Spam-Level: ** X-Spam-Status: No, score=2.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id qj9ud7GMDGZC; Fri, 30 Oct 2015 07:30:21 +0000 (UTC) Received: from mail-lb0-f181.google.com (mail-lb0-f181.google.com [209.85.217.181]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 40D6923086; Fri, 30 Oct 2015 07:30:20 +0000 (UTC) Received: by lbjm5 with SMTP id m5so44510327lbj.3; Fri, 30 Oct 2015 00:30:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=l5jClrIdKgWPp/RAcIVN1LUeiTlIpvRdMUxH0C4itzo=; b=nZrUcquNX+0t5KLaClUAGCWgcvVUv97Lv932MVvWlFH2Qpk7GzKyd9JlPRMwSgIvtF gq/et94QU984SMMqfvG/NcaVXzR5ABdCUWb0ckKHKllSC5eRqePm4CTXS13wYSLjmirD WtThEGcsCFwFRdrhzTyMqtVF4NMVXGH1vR5LCE4RtwT+wF/uuzPECZPLv7RUWPTZ3nPL GULiI3/Vv63GnIjvmY5XDk4BZImbWDmAj2Q3GebzITqz2LCFAECwW1Qb6ap7udxeseHz S+EPonXR9VD+Eu6wyqRldQhv5faxVH9Xj/MwYGwKyuod0kP6yaL9kPr9QYsbHwdrA23z wu/Q== X-Received: by 10.112.129.6 with SMTP id ns6mr3220345lbb.109.1446190218603; Fri, 30 Oct 2015 00:30:18 -0700 (PDT) Received: from dhcp-10-20-56-58.wlan.ntnu.no (129-241-228-220-gw.cgn.ntnu.no. [129.241.228.220]) by smtp.gmail.com with ESMTPSA id e63sm1015344lfi.1.2015.10.30.00.30.17 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 30 Oct 2015 00:30:17 -0700 (PDT) Content-Type: multipart/alternative; boundary="Apple-Mail=_889D6A8F-D637-469D-8143-AAB6C01472E4" Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\)) Subject: Re: Socket feed questions From: Heri Ramampiaro In-Reply-To: Date: Fri, 30 Oct 2015 08:30:16 +0100 Cc: users@asterixdb.incubator.apache.org Message-Id: <50BCB062-E638-4165-B009-A01E300908E1@gmail.com> References: <792997416F795F469B4A87A6FEC9F9766F54CF29@VTTMAIL3.ad.vtt.fi> <56290A20.8030106@gmail.com> <563054EB.5000405@ics.uci.edu> <2AD1BA54-BD3F-4CBB-BE67-77A72C03D839@gmail.com> To: dev@asterixdb.incubator.apache.org X-Mailer: Apple Mail (2.3096.5) --Apple-Mail=_889D6A8F-D637-469D-8143-AAB6C01472E4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Although I haven=E2=80=99t tested this (I have mostly used & created = java-based UDFs),=20 one can execute AQL calls from within a UDF. Feeds allows functions to=20= execute arbitrary AQL statements (DDL, DMLs etc). I.e., I believe what you are trying to do is possible. =20 For example you could do: (Given that you have a rec. type called Tweet, and a dataset = ProcessedTweets create feed CoordTwitterFeed if not exists using =E2=80=9Cpush_twitter" (("type-name"=3D"Tweet=E2=80=9D)) apply function find-intersection; (Here "find-intersection=E2=80=9D is an AQL function that does the step = specified under 2)). To do nr. 3 the easiest way is to have a modified version of the = =E2=80=9ChashTag=E2=80=9D=20 (let=E2=80=99s call this =E2=80=9Ctweetlocator=E2=80=9D). Java-based = UDF (see my previous message with the=20 TweetLib example). You can then connect this as a secondary feed, = connected to=20 =E2=80=9CCoordTwitterFeed=E2=80=9D. E.g.: create secondary feed ProcessedTwitterFeed from feed CoordTwitterFeed apply function =E2=80=9Ctweelib#tweetlocator"; Thereafter you can connect the feeds to appropriate datasets. (Since the find-intersection is an AQL function, you can also call = =E2=80=9Ctweelib#tweetlocator=E2=80=9D inside this function). Best, -heri > On Oct 30, 2015, at 12:18 AM, Young-Seok Kim = wrote: >=20 > I'm not sure whether the following UDF is possible or not, but = hopefully it > is. > What we're trying to do is to have the following UDF. >=20 > The UDF > 1) accepts an incoming tweet record through a feed job as an input and = then > 2) takes a field, more specifically, coordinate field value from the = tweet > record and sends a spatial-intersect query using the coordinate in = order to > find out the corresponding county of the coordinates (we have created > AsterixDB instance which stores US county shapes records into a = dataset and > created R-tree index on the polygon field of it, so the query will = return > the county effectively using the R-tree index) > 3) creates a new tweet record consisting of the original record's = fields + > the returned county value > 4) ingests to a tweet dataset in the AsterixDB instance. >=20 > Can we have such an UDF? >=20 > Best, > Young-Seok >=20 >=20 > On Thu, Oct 29, 2015 at 3:53 PM, Heri Ramampiaro = wrote: >=20 >>>=20 >>> 2. Can we use AQL function in those kind of feed UDFs? >>=20 >> Can you give an example of what you are trying to do? >> I.e do you want to run an AQL inside a UDF or use an AQL function as = a >> UDF connected to a running feed? >>=20 >> -heri --Apple-Mail=_889D6A8F-D637-469D-8143-AAB6C01472E4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Although I haven=E2=80=99t tested this (I have mostly used = & created java-based UDFs), 
one can execute AQL = calls from within a UDF. Feeds allows functions = to 
execute arbitrary AQL statements (DDL, = DMLs etc). I.e., I believe what
you are trying to do is possible. =  

For example you could = do:
(Given that = you have a rec. type called Tweet, and a dataset = ProcessedTweets

create feed CoordTwitterFeed = if not exists
using =E2=80=9Cpush_twitter" = (("type-name"=3D"Tweet=E2=80=9D))
apply = function find-intersection;

(Here "find-intersection=E2=80=9D is an AQL function that = does the step specified under 2)).

To do nr. 3 the easiest way is to have = a modified version of the =E2=80=9ChashTag=E2=80=9D 
(let=E2=80=99s call this =E2=80=9Ctweetlocator=E2=80=9D). =  Java-based UDF (see my previous message with the 
TweetLib example). You can then connect this  as a = secondary feed, connected to 
=E2=80=9CCoordTwitterFeed=E2=80=9D.

E.g.:
create secondary feed ProcessedTwitterFeed from feed = CoordTwitterFeed
apply function = =E2=80=9Ctweelib#tweetlocator";

Thereafter you can connect the feeds to = appropriate datasets.

(Since the find-intersection is an AQL function, you can also = call =E2=80=9Ctweelib#tweetlocator=E2=80=9D
inside = this function).

Best,
-heri

On Oct 30, 2015, at 12:18 AM, Young-Seok Kim <kisskys@gmail.com> = wrote:

I'm not sure whether the following UDF is possible or not, = but hopefully it
is.
What we're trying to do = is to have the following UDF.

The UDF
1) accepts an incoming tweet record through a feed job as an = input and then
2) takes a field, more specifically, = coordinate field value from the tweet
record and sends a = spatial-intersect query using the coordinate in order to
find out the corresponding county of the coordinates (we have = created
AsterixDB instance which stores US county shapes = records into a dataset and
created R-tree index on the = polygon field of it, so the query will return
the county = effectively using the R-tree index)
3) creates a new tweet = record consisting of the original record's fields +
the = returned county value
4) ingests to a tweet dataset in the = AsterixDB instance.

Can we have such an = UDF?

Best,
Young-Seok


On Thu, Oct 29, 2015 at 3:53 = PM, Heri Ramampiaro <heriram@gmail.com> wrote:


2. Can we use AQL function in those kind of = feed UDFs?

Can you give an = example of what you are trying to do?
I.e do you want to = run an AQL inside a UDF or use an AQL function as a
UDF = connected to a running feed?

-heri

= --Apple-Mail=_889D6A8F-D637-469D-8143-AAB6C01472E4--