Return-Path: X-Original-To: apmail-asterixdb-users-archive@minotaur.apache.org Delivered-To: apmail-asterixdb-users-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1045E1798E for ; Tue, 3 Nov 2015 06:27:00 +0000 (UTC) Received: (qmail 9840 invoked by uid 500); 3 Nov 2015 06:27:00 -0000 Delivered-To: apmail-asterixdb-users-archive@asterixdb.apache.org Received: (qmail 9807 invoked by uid 500); 3 Nov 2015 06:26:59 -0000 Mailing-List: contact users-help@asterixdb.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@asterixdb.incubator.apache.org Delivered-To: mailing list users@asterixdb.incubator.apache.org Received: (qmail 9797 invoked by uid 99); 3 Nov 2015 06:26:59 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2015 06:26:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 14B3FC2AB0 for ; Tue, 3 Nov 2015 06:26:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.001 X-Spam-Level: **** X-Spam-Status: No, score=4.001 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id tEePGd62WifD for ; Tue, 3 Nov 2015 06:26:48 +0000 (UTC) Received: from sink.edelkey.net (sink.edelkey.net [213.138.147.137]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 1E19D23057 for ; Tue, 3 Nov 2015 06:26:47 +0000 (UTC) Received: from asgw2.edelkey.net (asgw2.edelkey.net [213.138.142.179]) by sink.edelkey.net (8.14.7/8.14.7) with ESMTP id tA36QeLS004073 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Tue, 3 Nov 2015 08:26:40 +0200 Received: from VTTCASHUB4.ad.vtt.fi (213214159131.edelkey.net [213.214.159.131]) by asgw2.edelkey.net (8.15.0.59/8.15.0.59) with ESMTPS id tA36QeXu005668 (version=TLSv1 cipher=AES128-SHA bits=128 verify=NOT) for ; Tue, 3 Nov 2015 08:26:40 +0200 Received: from VTTMAIL3.ad.vtt.fi ([fe80::1403:8e16:3f7a:b6d1]) by VTTCASHUB4.ad.vtt.fi ([fe80::2d27:c616:29b6:e6e5%13]) with mapi id 14.03.0224.002; Tue, 3 Nov 2015 08:26:40 +0200 From: =?iso-8859-1?Q?P=E4=E4kk=F6nen_Pekka?= To: "users@asterixdb.incubator.apache.org" Subject: Re: AsterixDB - UDF Thread-Topic: Re: AsterixDB - UDF Thread-Index: AdEV/lMywfh31hSxQ5erl3hcQL6bLQ== Date: Tue, 3 Nov 2015 06:26:39 +0000 Message-ID: <792997416F795F469B4A87A6FEC9F9766F550AA1@VTTMAIL3.ad.vtt.fi> Accept-Language: fi-FI, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [130.188.126.117] Content-Type: multipart/alternative; boundary="_000_792997416F795F469B4A87A6FEC9F9766F550AA1VTTMAIL3advttfi_" MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2015-11-03_06:,, signatures=0 --_000_792997416F795F469B4A87A6FEC9F9766F550AA1VTTMAIL3advttfi_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, Yes, I have been testing UDF in AQL. It seems that I get a throughput of ~200 queries/s, when using multithreadi= ng in the client. By batching 100 UDF operations/query, throughput increases to +10000 UDF op= erations/s. I am wondering, if UDF is connected to a feed, how can I check that the UDF= is processed fast enough, if there would be real time processing requirements? Regards, -Pekka You can test UDFs (and should!!) by using them in AQL queries against the kinds of data that will be coming in on the feed. Once they are tested and working 100%, you can use them on a feed and be confident that they'll be doing the right thing. Does that help? Cheers, Mike On 10/28/15 3:53 AM, P=E4=E4kk=F6nen Pekka wrote: > > Hi, > > Thanks. I am now able to create a simple UDF. > > If I connect the UDF to a feed, is it possible to know from outside of > AsterixDB, when execution of UDF calls have been completed? > > Or should I use AQL via REST API (to call the UDF) for such purpose? > > -Pekka > > Hi Pekka, > > For making UDFs, you should not necessarily have to import AsterixDB > > into Eclipse/Intellij, as long as you are comfortable with maven and > > your text editor of choice. It may be easiest however. > > As for the content of the asterix-external-data zip, yes, other UDFs > > should follow the general example of that archive. That XML file ( > > https://github.com/apache/incubator-asterixdb/blob/master/asterix-externa= l-data/src/test/resources/library_descriptor.xml) > > corresponds to these set of functions: > > (https://github.com/apache/incubator-asterixdb/tree/master/asterix-extern= al-data/src/test/java/org/apache/asterix/external/library) > > Hope that helps! Sorry there isn't a more end-to-end documentation > > regarding implementing a UDF, but hopefully the source examples are > > illustrative enough. > > Thanks, > > -Ian > > On Mon, Oct 26, 2015 at 6:02 AM, P=E4=E4kk=F6nen Pekka > wrote: > > > Hi, > > > > > > > > > > > > In order to program UDFs, do I need to install AsterixDB development > > > environment to Eclipse? > > > > > > > > > > > > I am also wondering about the content of the UDF ZIP bundle. > > > > > > Should the content be similar to > > > asterix-external-data-0.8.7-SNAPSHOT-binary-assembly.zip, which contain= s > > > > > > a JAR and library_descriptor.xml? > > > > > > > > > > > > Regards, > > > > > > -Pekka > > > > > > > > > > > > > > > > > > > --_000_792997416F795F469B4A87A6FEC9F9766F550AA1VTTMAIL3advttfi_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Hi,

 

Yes, I have been te= sting UDF in AQL.

It seems that I get= a throughput of ~200 queries/s, when using multithreading in the client.

By batching 100 UDF= operations/query, throughput increases to +10000 UDF operations/s. &nb= sp;

 

I am wondering, if = UDF is connected to a feed, how can I check that the UDF is processed fast = enough, if

there would be real= time processing requirements?

 

Regards,=

-Pekka

 

 

 

 

 

 

 

You can test UDFs (= and should!!) by using them in AQL queries against

the kinds of data t= hat will be coming in on the feed.  Once they are

tested and working = 100%, you can use them on a feed and be confident

that they'll be doi= ng the right thing. Does that help?

 

Cheers,<= /span>

Mike

 

 

On 10/28/15 3:53 AM= , P=E4=E4kk=F6nen Pekka wrote:

> 

> Hi,=

> 

> Thanks. I am n= ow able to create a simple UDF.

> 

> If I connect t= he UDF to a feed, is it possible to know from outside of

> AsterixDB, whe= n execution of UDF calls have been completed?

> 

> Or should I us= e AQL via REST API (to call the UDF) for such purpose?

> 

> -Pekka

> 

> Hi Pekka,=

> 

> For making UDF= s, you should not necessarily have to import AsterixDB

> 

> into Eclipse/I= ntellij, as long as you are comfortable with maven and

> 

> your text edit= or of choice. It may be easiest however.

> 

> As for the con= tent of the asterix-external-data zip, yes, other UDFs

> 

> should follow = the general example of that archive. That XML file (

> 

> https://github= .com/apache/incubator-asterixdb/blob/master/asterix-external-data/src/test/= resources/library_descriptor.xml)

> 

> corresponds to= these set of functions:

> 

> (https://githu= b.com/apache/incubator-asterixdb/tree/master/asterix-external-data/src/test= /java/org/apache/asterix/external/library)

> 

> Hope that help= s! Sorry there isn't a more end-to-end documentation

> 

> regarding impl= ementing a UDF, but hopefully the source examples are

> 

> illustrative e= nough.

> 

> Thanks,

> 

> -Ian

> 

> On Mon, Oct 26= , 2015 at 6:02 AM, P=E4=E4kk=F6nen Pekka

> <Pekka.Paak= konen@vtt.fi> wrote:

> 

> > Hi,<= /o:p>

> 

> >

> 

> >

> 

> >

> 

> > In order = to program UDFs, do I need to install AsterixDB development

> 

> > environme= nt to Eclipse?

> 

> >

> 

> >

> 

> >

> 

> > I am also= wondering about the content of the UDF ZIP bundle.

> 

> >

> 

> > Should th= e content be similar to

> 

> > asterix-e= xternal-data-0.8.7-SNAPSHOT-binary-assembly.zip, which contains<= /span>

> 

> >

> 

> > a JAR and= library_descriptor.xml?

> 

> >

> 

> >

> 

> >

> 

> > Regards,

> 

> >

> 

> > -Pekka=

> 

> >

> 

> >

> 

> >

> 

> >

> 

> >

> 

> >

> 

 

--_000_792997416F795F469B4A87A6FEC9F9766F550AA1VTTMAIL3advttfi_--