asterixdb-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pääkkönen Pekka <Pekka.Paakko...@vtt.fi>
Subject Re: Socket feed questions
Date Thu, 22 Oct 2015 11:29:04 GMT
Hi,

Yes, you are right. I tried sending a larger amount of data, and data is now stored to the
database.

Does it make sense to configure a smaller batch size in order to get more frequent writes?
Or would it significantly impact performance?

-Pekka


Data moves through the pipeline in frame-sized batches, so one
(uniformed :-)) guess is that you aren't running very long, and you're
only seeing the data flow when you close because only then do you have a
batch's worth.  Is that possible?  You can test this by running longer
(more data) and seeing if you start to see the expected incremental
flow/inserts.  (And we need tunability in this area, e.g., parameters on
how much batching and/or low much latency to tolerate on each feed.)

On 10/21/15 4:45 AM, Pääkkönen Pekka wrote:
>
> Hi,
>
> Thanks, now I am able to create a socket feed, and save items to the
> dataset from the feed.
>
> It seems that data items are written to the dataset after I close the
> socket at the client.
>
> Is there some way to indicate to AsterixDB feed (with a newline or
> other indicator) that data can be written to the database, when the
> connection is open?
>
> After I close the socket at the client, the feed seems to close down.
> Or is it only paused, until it is resumed?
>
> -Pekka
>
> Hi Pekka,
>
> That's interesting, I'm not sure why the CC would appear as being down
>
> to Managix. However if you can access the web console, it that
>
> evidently isn't the case.
>
> As for data ingestion via sockets, yes it is possible, but it kind of
>
> depends on what's meant by sockets. There's no tutorial for it, but
>
> take a look at SocketBasedFeedAdapter in the source, as well as
>
> https://github.com/kisskys/incubator-asterixdb/blob/kisskys/indexonlyhilbertbtree/asterix-experiments/src/main/java/org/apache/asterix/experiment/client/SocketTweetGenerator.java
>
> for some examples of how it works.
>
> Hope that helps!
>
> Thanks,
>
> -Ian
>
> On Mon, Oct 19, 2015 at 10:15 PM, Pääkkönen Pekka
> <Pekka.Paakkonen@vtt.fi> wrote:
> > Hi Ian,
> >
> >
> >
> > Thanks for the reply.
> >
> > I compiled AsterixDB v0.87 and started it.
> >
> >
> >
> > However, I get the following warnings:
> >
> > INFO: Name:my_asterix
> >
> > Created:Mon Oct 19 08:37:16 UTC 2015
> >
> > Web-Url:http://192.168.101.144:19001
> >
> > State:UNUSABLE
> >
> >
> >
> > WARNING!:Cluster Controller not running at master
> >
> >
> >
> > Also, I see the following warnings in my_asterixdb1.log. there are no
> > warnings or errors in cc.log
> >
> > "
> >
> > Oct 19, 2015 8:37:39 AM
> > org.apache.hyracks.api.lifecycle.LifeCycleComponentManager configure
> >
> > SEVERE: LifecycleComponentManager configured
> > org.apache.hyracks.api.lifecycle.LifeCycleComponentManager@7559ec47
> >
> > ..
> >
> > INFO: Completed sharp checkpoint.
> >
> > Oct 19, 2015 8:37:40 AM org.apache.asterix.om.util.AsterixClusterProperties
> > getIODevices
> >
> > WARNING: Configuration parameters for nodeId my_asterix_node1 not found. The
> > node has not joined yet or has left.
> >
> > Oct 19, 2015 8:37:40 AM org.apache.asterix.om.util.AsterixClusterProperties
> > getIODevices
> >
> > WARNING: Configuration parameters for nodeId my_asterix_node1 not found. The
> > node has not joined yet or has left.
> >
> > Oct 19, 2015 8:38:38 AM
> > org.apache.hyracks.control.common.dataset.ResultStateSweeper sweep
> >
> > INFO: Result state cleanup instance successfully completed."
> >
> >
> >
> > I seems that AsterixDB is running, and I can access it at port 19001.
> >
> >
> >
> > The documentation shows ingestion of tweets, but I would be interested in
> > using sockets.
> >
> > Is it possible to ingest data from sockets?
> >
> >
> >
> > Regards,
> >
> > -Pekka
> >
> >
> >
> >
> >
> >
> >
> > Hey there Pekka,
> >
> > Your intuition is correct, most of the newer feeds features are in the
> >
> > current master branch and not in the (very) old 0.8.6 release. If you'd
> >
> > like to experiment with them you'll have to build from source. The details
> >
> > about that are here:
> >
> > https://asterixdb.incubator.apache.org/dev-setup.html#setting-up-an-asterix-development-environment-in-eclipse
> >
> > , but they're probably a bit overkill for just trying to get the compiled
> >
> > binaries. For that all you really need to do is :
> >
> > - Clone Hyracks from git
> >
> > - 'mvn clean install -DskipTests'
> >
> > - Clone AsterixDB
> >
> > - 'mvn clean package -DskipTests'
> >
> > Then, the binaries will sit in asterix-installer/target
> >
> >
> >
> >
> >
> > For an example, the documentation shows how to set up a feed that's
> >
> > ingesting Tweets:
> >
> > https://asterix-jenkins.ics.uci.edu/job/asterix-test-full/site/asterix-doc/feeds/tutorial.html
> >
> >
> >
> >
> >
> > Thanks,
> >
> > -Ian
> >
> >
> >
> >
> >
> > On Wed, Oct 7, 2015 at 9:48 PM, Pääkkönen Pekka <Pekka.Paakkonen@vtt.fi>
> >
> > wrote:
> >
> >
> >
> >> Hi,
> >
> >>
> >
> >>
> >
> >>
> >
> >> I would like to experiment with a socket-based feed.
> >
> >>
> >
> >> Can you point me to an example on how to utilize them?
> >
> >>
> >
> >> Do I need to install 0.8.7-snapshot version of AsterixDB in order to
> >
> >> experiment with feeds?
> >
> >>
> >
> >>
> >
> >>
> >
> >> Regards,
> >
> >>
> >
> >> -Pekka Pääkkönen
> >
> >>
> >
> >
>


Mime
View raw message