asterixdb-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Carey <dtab...@gmail.com>
Subject Re: Socket feed questions
Date Thu, 22 Oct 2015 16:09:04 GMT
I think we need to have tuning parameters - like batch size and maximum 
tolerable latency (in case there's a lull and you still want to push 
stuff with some worst-case delay). @Raman Grover - remind me (us) what's 
available in this regard?

On 10/22/15 4:29 AM, Pääkkönen Pekka wrote:
>
> Hi,
>
> Yes, you are right. I tried sending a larger amount of data, and data 
> is now stored to the database.
>
> Does it make sense to configure a smaller batch size in order to get 
> more frequent writes?
>
> Or would it significantly impact performance?
>
> -Pekka
>
> Data moves through the pipeline in frame-sized batches, so one
>
> (uniformed :-)) guess is that you aren't running very long, and you're
>
> only seeing the data flow when you close because only then do you have a
>
> batch's worth.  Is that possible?  You can test this by running longer
>
> (more data) and seeing if you start to see the expected incremental
>
> flow/inserts. (And we need tunability in this area, e.g., parameters on
>
> how much batching and/or low much latency to tolerate on each feed.)
>
> On 10/21/15 4:45 AM, Pääkkönen Pekka wrote:
>
> >
>
> > Hi,
>
> >
>
> > Thanks, now I am able to create a socket feed, and save items to the
>
> > dataset from the feed.
>
> >
>
> > It seems that data items are written to the dataset after I close the
>
> > socket at the client.
>
> >
>
> > Is there some way to indicate to AsterixDB feed (with a newline or
>
> > other indicator) that data can be written to the database, when the
>
> > connection is open?
>
> >
>
> > After I close the socket at the client, the feed seems to close down.
>
> > Or is it only paused, until it is resumed?
>
> >
>
> > -Pekka
>
> >
>
> > Hi Pekka,
>
> >
>
> > That's interesting, I'm not sure why the CC would appear as being down
>
> >
>
> > to Managix. However if you can access the web console, it that
>
> >
>
> > evidently isn't the case.
>
> >
>
> > As for data ingestion via sockets, yes it is possible, but it kind of
>
> >
>
> > depends on what's meant by sockets. There's no tutorial for it, but
>
> >
>
> > take a look at SocketBasedFeedAdapter in the source, as well as
>
> >
>
> > https://github.com/kisskys/incubator-asterixdb/blob/kisskys/indexonlyhilbertbtree/asterix-experiments/src/main/java/org/apache/asterix/experiment/client/SocketTweetGenerator.java
>
> >
>
> > for some examples of how it works.
>
> >
>
> > Hope that helps!
>
> >
>
> > Thanks,
>
> >
>
> > -Ian
>
> >
>
> > On Mon, Oct 19, 2015 at 10:15 PM, Pääkkönen Pekka
>
> > <Pekka.Paakkonen@vtt.fi> wrote:
>
> > > Hi Ian,
>
> > >
>
> > >
>
> > >
>
> > > Thanks for the reply.
>
> > >
>
> > > I compiled AsterixDB v0.87 and started it.
>
> > >
>
> > >
>
> > >
>
> > > However, I get the following warnings:
>
> > >
>
> > > INFO: Name:my_asterix
>
> > >
>
> > > Created:Mon Oct 19 08:37:16 UTC 2015
>
> > >
>
> > > Web-Url:http://192.168.101.144:19001
>
> > >
>
> > > State:UNUSABLE
>
> > >
>
> > >
>
> > >
>
> > > WARNING!:Cluster Controller not running at master
>
> > >
>
> > >
>
> > >
>
> > > Also, I see the following warnings in my_asterixdb1.log. there are no
>
> > > warnings or errors in cc.log
>
> > >
>
> > > “
>
> > >
>
> > > Oct 19, 2015 8:37:39 AM
>
> > > org.apache.hyracks.api.lifecycle.LifeCycleComponentManager configure
>
> > >
>
> > > SEVERE: LifecycleComponentManager configured
>
> > > org.apache.hyracks.api.lifecycle.LifeCycleComponentManager@7559ec47
>
> > >
>
> > > ..
>
> > >
>
> > > INFO: Completed sharp checkpoint.
>
> > >
>
> > > Oct 19, 2015 8:37:40 AM 
> org.apache.asterix.om.util.AsterixClusterProperties
>
> > > getIODevices
>
> > >
>
> > > WARNING: Configuration parameters for nodeId my_asterix_node1 not 
> found. The
>
> > > node has not joined yet or has left.
>
> > >
>
> > > Oct 19, 2015 8:37:40 AM 
> org.apache.asterix.om.util.AsterixClusterProperties
>
> > > getIODevices
>
> > >
>
> > > WARNING: Configuration parameters for nodeId my_asterix_node1 not 
> found. The
>
> > > node has not joined yet or has left.
>
> > >
>
> > > Oct 19, 2015 8:38:38 AM
>
> > > org.apache.hyracks.control.common.dataset.ResultStateSweeper sweep
>
> > >
>
> > > INFO: Result state cleanup instance successfully completed.”
>
> > >
>
> > >
>
> > >
>
> > > I seems that AsterixDB is running, and I can access it at port 19001.
>
> > >
>
> > >
>
> > >
>
> > > The documentation shows ingestion of tweets, but I would be interested in
>
> > > using sockets.
>
> > >
>
> > > Is it possible to ingest data from sockets?
>
> > >
>
> > >
>
> > >
>
> > > Regards,
>
> > >
>
> > > -Pekka
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > > Hey there Pekka,
>
> > >
>
> > > Your intuition is correct, most of the newer feeds features are in the
>
> > >
>
> > > current master branch and not in the (very) old 0.8.6 release. If you'd
>
> > >
>
> > > like to experiment with them you'll have to build from source. The 
> details
>
> > >
>
> > > about that are here:
>
> > >
>
> > > https://asterixdb.incubator.apache.org/dev-setup.html#setting-up-an-asterix-development-environment-in-eclipse
>
> > >
>
> > > , but they're probably a bit overkill for just trying to get the compiled
>
> > >
>
> > > binaries. For that all you really need to do is :
>
> > >
>
> > > - Clone Hyracks from git
>
> > >
>
> > > - 'mvn clean install -DskipTests'
>
> > >
>
> > > - Clone AsterixDB
>
> > >
>
> > > - 'mvn clean package -DskipTests'
>
> > >
>
> > > Then, the binaries will sit in asterix-installer/target
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > > For an example, the documentation shows how to set up a feed that's
>
> > >
>
> > > ingesting Tweets:
>
> > >
>
> > > https://asterix-jenkins.ics.uci.edu/job/asterix-test-full/site/asterix-doc/feeds/tutorial.html
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > > Thanks,
>
> > >
>
> > > -Ian
>
> > >
>
> > >
>
> > >
>
> > >
>
> > >
>
> > > On Wed, Oct 7, 2015 at 9:48 PM, Pääkkönen Pekka <Pekka.Paakkonen@vtt.fi>
>
> > >
>
> > > wrote:
>
> > >
>
> > >
>
> > >
>
> > >> Hi,
>
> > >
>
> > >>
>
> > >
>
> > >>
>
> > >
>
> > >>
>
> > >
>
> > >> I would like to experiment with a socket-based feed.
>
> > >
>
> > >>
>
> > >
>
> > >> Can you point me to an example on how to utilize them?
>
> > >
>
> > >>
>
> > >
>
> > >> Do I need to install 0.8.7-snapshot version of AsterixDB in order to
>
> > >
>
> > >> experiment with feeds?
>
> > >
>
> > >>
>
> > >
>
> > >>
>
> > >
>
> > >>
>
> > >
>
> > >> Regards,
>
> > >
>
> > >>
>
> > >
>
> > >> -Pekka Pääkkönen
>
> > >
>
> > >>
>
> > >
>
> > >
>
> >
>


Mime
View raw message