asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raman Grover <ramangrove...@gmail.com>
Subject Re: Unable to connect secondary feed to primary feed
Date Thu, 11 Jun 2015 22:55:43 GMT
Q: When we delete the dataverse, how do we actually clean up the library?
Is there a weird state between dataverse deleted but library still
installed via managix?

When deleting a dataverse, we search for the contained components that
includes datasets, types, indexes and *libraries*. For each library, we
further drill down and enumerate the contained components which may include
UDFs and adaptors. We invoke drop statements on each library component
following by a drop library statement.
Note that managix does not maintain any state information about the
installed libraries for an AsterixDB instance. This information is
maintained only as part of AsterixDB metadata.

Regards,
Raman

On Thu, Jun 11, 2015 at 3:52 PM, Ian Maxon <imaxon@uci.edu> wrote:

> The issue is that the 'connect feed' statement for the secondary feed
> silently fails, because the library used to process that feed isn't found.
> The primary is fine, it doesn't need the library.
>
> Q: When we delete the dataverse, how do we actually clean up the library?
> Is there a weird state between dataverse deleted but library still
> installed via managix?
>
> On Thu, Jun 11, 2015 at 3:43 PM, Eugenia Gabrielova <
> genia.likes.science@gmail.com> wrote:
>
> > Thank you for the correction about the query - I had no idea it would
> have
> > that effect. This worked!!!
> >
> > Would it be accurate to say that the UI should not have reported the
> > primary feed successfully created, and the issue with the Secondary Feed
> > was because the Primary Feed was not properly created?
> >
> > On Thu, Jun 11, 2015 at 3:31 PM, Eugenia Gabrielova <
> > genia.likes.science@gmail.com> wrote:
> >
> > > Thank you for the explanation, I'll try this now!
> > >
> > > On Thu, Jun 11, 2015 at 3:29 PM, Raman Grover <ramangrover29@gmail.com
> >
> > > wrote:
> > >
> > >> As per the steps mentioned for reproducing the issue,
> > >> step 0 involves installing the external library testlib in a dataverse
> > >> (assumed to be "feeds" dataverse). On system startup (following
> managix
> > >> install command), the library components (UDFs) get installed in the
> > stated
> > >> dataverse.
> > >>
> > >> However, subsequently as part of step 1,  the dataverse is dropped
> (drop
> > >> dataverse feeds if exists;) which results in un-installing the
> contents
> > of
> > >> the dataverse including the library and thus the installed UDFs
> > thereof. At
> > >> runtime (connect feed statement), the required library is not found.
> > There
> > >> is an error-propagation issue here as this can be caught earlier and
> > thrown
> > >> at the UI.
> > >>
> > >> Before going further, it would be great if you can confirm the above
> > >> steps and that the dataverse was actually dropped post installation of
> > >> library.
> > >> Further, I tried reproducing with the above statements (but removing
> the
> > >> drop and create dataverse statements from step 2) and got the feeds
> > >> connected and running fine.
> > >> Attached is a screen shot of the recent run showing the primary and
> > >> secondary feeds in connected state together with the data rates for
> > each.
> > >>
> > >> Regards,
> > >> Raman
> > >>
> > >> On Thu, Jun 11, 2015 at 2:41 PM, Eugenia Gabrielova <
> > >> genia.likes.science@gmail.com> wrote:
> > >>
> > >>> *Summary*: I'm able to create and connect a feed from Twitter, but
I
> am
> > >>> unable to connect a secondary feed. The web UI hangs, and the feeds
> UI
> > >>> does
> > >>> not detect the secondary feed. Nothing amiss in the cc.log or
> > [node].log.
> > >>>
> > >>> *To Replicate:*
> > >>> *Step 0*
> > >>> Check out this change on Asterix:
> > >>> https://asterix-gerrit.ics.uci.edu/#/c/248/
> > >>> Check out this change on Hyracks:
> > >>> https://asterix-gerrit.ics.uci.edu/#/c/249/
> > >>>
> > >>> Install as you normally would for feeds, including installing testlib
> > >>> with
> > >>> managix. I can document my command line steps on request :).
> > >>>
> > >>> Feeds console here: http://localhost:19003/
> > >>>
> > >>> *Step 1 - Twitter Feed in Web UI - Success!*
> > >>> drop dataverse feeds if exists;
> > >>> create dataverse feeds;
> > >>> use dataverse feeds;
> > >>>
> > >>> create type TwitterUser  if not exists
> > >>> as open{
> > >>> screen_name: string,
> > >>> language: string,
> > >>> friends_count: int32,
> > >>> status_count: int32,
> > >>> name: string,
> > >>> followers_count: string
> > >>> };
> > >>>
> > >>> create type Tweet if not exists as open{
> > >>> id: string,
> > >>> user:TwitterUser,
> > >>> latitude:double,
> > >>> longitude:double,
> > >>> created_at:string,
> > >>> message_text:string
> > >>> };
> > >>>
> > >>> create dataset Tweets(Tweet) if not exists
> > >>> primary key id;
> > >>>
> > >>> create feed TwitterFeed if not exists
> > >>> using "push_twitter"
> > >>> (("type-name"="Tweet"),("location"="US"));
> > >>>
> > >>> connect feed TwitterFeed to dataset Tweets;
> > >>>
> > >>> *Step 2 - Create (but don't connect Processed Tweets) in Web UI -
> > >>> success!*
> > >>> use dataverse feeds;
> > >>>
> > >>> create type ProcessedTweet if not exists as open {
> > >>> id: string,
> > >>> user_name:string,
> > >>> location:point,
> > >>> created_at:string,
> > >>> message_text:string,
> > >>> country: string,
> > >>> topics: [string]
> > >>> };
> > >>>
> > >>> create dataset ProcessedTweets(ProcessedTweet) if not exists
> > >>> primary key id;
> > >>>
> > >>> create index location_index if not exists on
> ProcessedTweets(location)
> > >>> type
> > >>> rtree;
> > >>>
> > >>> create feed ProcessedTwitterFeed if not exists
> > >>> from TwitterFeed
> > >>> apply function testlib#processRawTweet;
> > >>>
> > >>> *Step 3 - Connect ProcessedTwitterFeed :( - ???*
> > >>> use dataverse feeds;
> > >>>
> > >>> connect feed ProcessedTwitterFeed to dataset ProcessedTweets;
> > >>>
> > >>> *Observation: *Secondary feed does not appear at localhost:19003, and
> > web
> > >>> UI hangs (no return success).
> > >>>
> > >>> Thanks for any help and insight. Please let me know if I should
> provide
> > >>> further detail.
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Raman
> > >>
> > >
> > >
> >
>



-- 
Raman

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message