Return-Path: X-Original-To: apmail-asterixdb-users-archive@minotaur.apache.org Delivered-To: apmail-asterixdb-users-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0180617210 for ; Wed, 23 Sep 2015 03:19:45 +0000 (UTC) Received: (qmail 38799 invoked by uid 500); 23 Sep 2015 03:19:42 -0000 Delivered-To: apmail-asterixdb-users-archive@asterixdb.apache.org Received: (qmail 38722 invoked by uid 500); 23 Sep 2015 03:19:42 -0000 Mailing-List: contact users-help@asterixdb.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@asterixdb.incubator.apache.org Delivered-To: mailing list users@asterixdb.incubator.apache.org Received: (qmail 38676 invoked by uid 99); 23 Sep 2015 03:19:42 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Sep 2015 03:19:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 32DEE1809BF for ; Wed, 23 Sep 2015 03:19:42 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.88 X-Spam-Level: ** X-Spam-Status: No, score=2.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id hfdyiMvJ4pUE for ; Wed, 23 Sep 2015 03:19:36 +0000 (UTC) Received: from mail-wi0-f175.google.com (mail-wi0-f175.google.com [209.85.212.175]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id D1D7821141 for ; Wed, 23 Sep 2015 03:19:35 +0000 (UTC) Received: by wicgb1 with SMTP id gb1so186315567wic.1 for ; Tue, 22 Sep 2015 20:19:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=g4xgHO5h6fO26J3Gc5Wi1vWtCedKRxCyPi1B2+IKEXA=; b=ZefqxVqWov3TPRg4Aq+hXuuYUpcQZBYW5vtx7QG63nBxb2dLip689TEqKTT1+lkCTL doWQwfjIt5MNx8iXbvVnSgJo5Pxa6voCPChAXu4E2tmQ/WTGsKP2WCDOgWThfo4JPZMU XjVlQ4XngYsBI/K6is9/KuWROrkwSpH6aUbUA0eVDIYdfVBv5mx0vNBSjOtdd814HlBy EiH/KxW9nqRjvBlcrrmCjZHA3hJxgsf1cYZZJcJDhlbgn/rz2Wr8GT7glcq5ruDBpsP8 03H0Rhp02EhtOFV0O3al9QXl5VLiTi4qkjrIRmHDJ7jYKqJGVTNPPWDLXmbtK2uvp0TT G8fA== MIME-Version: 1.0 X-Received: by 10.194.22.135 with SMTP id d7mr32206758wjf.78.1442978369420; Tue, 22 Sep 2015 20:19:29 -0700 (PDT) Received: by 10.194.28.101 with HTTP; Tue, 22 Sep 2015 20:19:29 -0700 (PDT) In-Reply-To: References: Date: Wed, 23 Sep 2015 08:49:29 +0530 Message-ID: Subject: Re: Twitter feed ingestion issue From: Ashok Raman To: users@asterixdb.incubator.apache.org Content-Type: multipart/alternative; boundary=047d7b5d5c2220c40c05206195e1 --047d7b5d5c2220c40c05206195e1 Content-Type: text/plain; charset=UTF-8 Thanks, Ian, for your quick response and support, looking forward. Best, Ashok On Wed, Sep 23, 2015 at 1:20 AM, Ian Maxon wrote: > Hi Ashok, > What you're seeing is definitely a bug, some of us have become aware > of it but only relatively recently (earlier this month). Basically > what appears to have happened is that this tutorial has existed for > quite a long time, but there is a discrepancy between how the code > changed in the last merge of feeds, and how the tutorial was updated. > The source of the stack you see is because the output of the adapter > is not matching the format of the type internally. > We plan to fix this as soon as possible, so stay tuned... > > Thanks, > -Ian > > > On Mon, Sep 21, 2015 at 11:24 PM, Ashok Raman > wrote: > > Hi, > > > > This is the sequence I followed with 0.8.7-Snapshot release: > > > ------------------------------------------------------------------------------- > > create dataverse feeds; > > use dataverse feeds; > > > > create type TwitterUser if not exists as open{ > > screen_name: string, > > language: string, > > friends_count: int32, > > status_count: int32, > > name: string, > > followers_count: string > > }; > > create type Tweet if not exists as open{ > > id: string, > > user: TwitterUser, > > latitude:double, > > longitude:double, > > created_at:string, > > message_text:string, > > topics: [string] > > }; > > > > create dataset Tweets (Tweet) > > primary key id; > > create feed TwitterFeed if not exists using "push_twitter" > > (("type-name"="Tweet"), > > ("consumer.key"=""), > > ("consumer.secret"=""), > > ("access.token"=""), > > ("access.token.secret"=""), > > ("topics"="allergy")); > > > > connect feed TwitterFeed to dataset Tweets; > > ------------------------------------------------------------------- > > While the cc.log has - > > "INFO: Received message > > > {"nodeId":"my_asterix_node1","partition":0,"runtime-type":"STORE","dataverse":"feeds","dataset":"Tweets","value":-1,"feed":"TwitterFeed","value-type":"OUTFLOW_RATE","message-type":"FEED_REPORT"} > > " > > > > my_asterix_node1.log has > > > > org.apache.asterix.metadata.feeds.AdapterExecutor run > > SEVERE: Exception during feed ingestion java.lang.NullPointerException > > java.lang.Exception: java.lang.NullPointerException > > at > > org.apache.asterix.external.dataset.adapter.ClientBasedFeedAdapter.st > > art(ClientBasedFeedAdapter.java:126) > > at > > org.apache.asterix.metadata.feeds.AdapterExecutor.run(AdapterExecutor > > .java:52) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. > > java:1145) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor > > .java:615) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.lang.NullPointerException > > at > > org.apache.asterix.external.dataset.adapter.ClientBasedFeedAdapter.st > > art(ClientBasedFeedAdapter.java:118) > > ... 4 more > > org.apache.asterix.common.exceptions.AsterixException: > > java.lang.ArrayIndexOutOf > > BoundsException: 6 > > at > > org.apache.asterix.external.dataset.adapter.FeedClient.nextTuple(Feed > > Client.java:106) > > at > > org.apache.asterix.external.dataset.adapter.ClientBasedFeedAdapter.st > > art(ClientBasedFeedAdapter.java:92) at > > org.apache.asterix.metadata.feeds.AdapterExecutor.run(AdapterExecutor > > .java:52) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. > > java:1145) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor > > .java:615) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.lang.ArrayIndexOutOfBoundsException: 6 > > at > org.apache.asterix.om.base.ARecord.getValueByPos(ARecord.java:51) > > at > > org.apache.asterix.external.dataset.adapter.FeedClient.writeRecord(Fe > > edClient.java:117) > > at > > org.apache.asterix.external.dataset.adapter.FeedClient.nextTuple(Feed > > Client.java:87) > > ... 5 more > > " > > > > > > > > Thanks for any help, > > Ashok > > > --047d7b5d5c2220c40c05206195e1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks, Ian, for your quick response a= nd support, looking forward.
Best,
Ashok
<= /div>

On Wed, Sep = 23, 2015 at 1:20 AM, Ian Maxon <imaxon@uci.edu> wrote:
Hi Ashok,
What you're seeing is definitely a bug, some of us have become aware of it but only relatively recently (earlier this month). Basically
what appears to have happened is that this tutorial has existed for
quite a long time, but there is a discrepancy between how the code
changed in the last merge of feeds, and how the tutorial was updated.
The source of the stack you see is because the output of the adapter
is not matching the format of the type internally.
We plan to fix this as soon as possible, so stay tuned...

Thanks,
-Ian


On Mon, Sep 21, 2015 at 11:24 PM, Ashok Raman <ashoknraman@gmail.com> wrote:
> Hi,
>
> This is the sequence I followed with 0.8.7-Snapshot release:
> ----------------------------------------------------------------------= ---------
>=C2=A0 create dataverse feeds;
>=C2=A0 =C2=A0 =C2=A0use dataverse feeds;
>
>=C2=A0 =C2=A0 =C2=A0create type TwitterUser if not exists as open{
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0screen_name: string,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0language: string,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0friends_count: int32,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0status_count: int32,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0name: string,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0followers_count: string
>=C2=A0 =C2=A0 =C2=A0};
>=C2=A0 =C2=A0 =C2=A0create type Tweet if not exists as open{
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0id: string,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0user: TwitterUser,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0latitude:double,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0longitude:double,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0created_at:string,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0message_text:string,
> topics: [string]
>=C2=A0 =C2=A0 =C2=A0};
>
>=C2=A0 =C2=A0 =C2=A0create dataset Tweets (Tweet)
>=C2=A0 =C2=A0 =C2=A0primary key id;
>=C2=A0 =C2=A0 =C2=A0create feed TwitterFeed if not exists using "p= ush_twitter"
>=C2=A0 =C2=A0 =C2=A0(("type-name"=3D"Tweet"),
>=C2=A0 =C2=A0 =C2=A0 ("consumer.key"=3D""),
>=C2=A0 =C2=A0 =C2=A0 ("consumer.secret"=3D""),
>=C2=A0 =C2=A0 =C2=A0 ("access.token"=3D""),
>=C2=A0 =C2=A0 =C2=A0 ("access.token.secret"=3D""),<= br> > ("topics"=3D"allergy"));
>
>=C2=A0 =C2=A0 =C2=A0connect feed TwitterFeed to dataset Tweets;
> ------------------------------------------------------------------- > While the cc.log has -
> "INFO: Received message
> {"nodeId":"my_asterix_node1","partition"= :0,"runtime-type":"STORE","dataverse":"f= eeds","dataset":"Tweets","value":-1,&quo= t;feed":"TwitterFeed","value-type":"OUTFLOW_R= ATE","message-type":"FEED_REPORT"}
> "
>
> my_asterix_node1.log has
>
> org.apache.asterix.metadata.feeds.AdapterExecutor run
> SEVERE: Exception during feed ingestion java.lang.NullPointerException=
> java.lang.Exception: java.lang.NullPointerException
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.asterix.= external.dataset.adapter.ClientBasedFeedAdapter.st
> art(ClientBasedFeedAdapter.java:126)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.asterix.metadata.feeds.AdapterExecutor.run(AdapterExecutor<= br> > .java:52)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.<= br> > java:1145)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor<= br> > .java:615)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thread.java:7= 45)
> Caused by: java.lang.NullPointerException
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.asterix.= external.dataset.adapter.ClientBasedFeedAdapter.st
> art(ClientBasedFeedAdapter.java:118)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0... 4 more
> org.apache.asterix.common.exceptions.AsterixException:
> java.lang.ArrayIndexOutOf
> BoundsException: 6
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.asterix.external.dataset.adapter.FeedClient.nextTuple(Feed<= br> > Client.java:106)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.asterix.= external.dataset.adapter.ClientBasedFeedAdapter.st
> art(ClientBasedFeedAdapter.java:92)=C2=A0 =C2=A0 =C2=A0 =C2=A0 at
> org.apache.asterix.metadata.feeds.AdapterExecutor.run(AdapterExecutor<= br> > .java:52)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.<= br> > java:1145)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor<= br> > .java:615)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thread.java:7= 45)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 6
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.asterix.om.base.ARecord= .getValueByPos(ARecord.java:51)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.asterix.external.dataset.adapter.FeedClient.writeRecord(Fe<= br> > edClient.java:117)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.asterix.external.dataset.adapter.FeedClient.nextTuple(Feed<= br> > Client.java:87)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0... 5 more
> "
>
>
>
> Thanks for any help,
> Ashok
>

--047d7b5d5c2220c40c05206195e1--