Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@streams.incubator.apache.org
Received-SPF: pass (athena.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
MIME-Version: 1.0
In-Reply-To: 
 <CAL8syOZqLnqx-Nvnu1QNe9KUbUzifdxdC2NneRVfnumTWLcDXw@mail.gmail.com>
References: <SNT149-W95AE9C5776E27CB53C1A068A170@phx.gbl>
	<CAL8syOYwARg8-nrmKafN5h+0E3vov08efETtsd9-2E0Nfe4j3g@mail.gmail.com>
	<SNT149-W7766DF44F2D60541FE584C8A100@phx.gbl>
	<SNT149-W16111ADD95BF49A081F4ED8A100@phx.gbl>
	<SNT149-W3044C0A75355E3684986DF8A1F0@phx.gbl>
	<CAL8syOaPD3JHnpzpgqkJN0MCUyBE3mE6hwvxb1td1baYv6TbGg@mail.gmail.com>
	<CAFNO4Hh+hpWR1suVEY31jE852mNkygTnEf=KfYRp+N97E=4LyA@mail.gmail.com>
	<SNT149-W18BC9E448B5178B4645E78A040@phx.gbl>
	<CAFNO4HgJ-+ZUhUuTYoWvcyj_unqhNdgW=y=Mms6k-XM7G-0kHw@mail.gmail.com>
	<SNT149-W802CB5D944878E5F3225668A040@phx.gbl>
	<CAFNO4HgrWgVihmVS09jEM1XGRmEoReY_eaWBa=Yup+1LN+3vuA@mail.gmail.com>
	<CAL8syOZaVGyF+nkvGp1USgsAr3rhTqmiabjdyTPkpmZ=HtpNgw@mail.gmail.com>
	<SNT149-W33C8B1B7EF3B56FBBF19508A040@phx.gbl>
	<CAL8syObuAFPMYqn68OoyXMB5bKeW5TQEiupiPGyq_ykcAtiepg@mail.gmail.com>
	<SNT149-W1299A32D45439CFB17E2178A030@phx.gbl>
	<CAL8syOZqLnqx-Nvnu1QNe9KUbUzifdxdC2NneRVfnumTWLcDXw@mail.gmail.com>
Date: Thu, 24 Oct 2013 09:55:10 -0700
Message-ID: 
 <CAFNO4Hhd9btPbxGfyHC5gSBOaMwUJ4a40Tca-URenR6td071+Q@mail.gmail.com>
Subject: Re: Using Camel for Routing
From: Chris Geer <chris@cxtsoftware.com>
To: "dev@streams.incubator.apache.org" <dev@streams.incubator.apache.org>
Content-Type: multipart/alternative; boundary=047d7b2e073b32149a04e97f7f98

--047d7b2e073b32149a04e97f7f98
Content-Type: text/plain; charset=ISO-8859-1

I agree with Jason personally, flexibility and scalability trumps simple.

That being said, I honestly don't have enough time right now to spend
developing on my vision so it's unfair of me to push too hard in that
direction. If the active developers want to go down the single service path
maybe that's the approach for now and we can potentially look at
alternatives down the road.

Chris


On Thu, Oct 24, 2013 at 8:08 AM, Jason Letourneau
<jletourneau80@gmail.com>wrote:

> I think this comes down to a fundamental question - how flexible vs
> simple do people want the deployment and configuration of Streams to
> be?
>
> Personally, as I've said, I prefer flexible over simple, but I think
> the current architecture reaches a happy medium with the WAR
> deployment option.
>
> If the majority wants simple over flexible, then perhaps the web
> service architecture you've proposed is the way to go.  I think its
> worth putting to vote.
>
> Jason
>
> On Wed, Oct 23, 2013 at 11:29 AM, Danny Sullivan <dsullivan7@hotmail.com>
> wrote:
> > Hey Guys,
> >
> > I'm trying to do a little more research in the efficiency between both
> the
> > web-service and the camel implementations of streams. I set up Apache
> JMeter
> > (http://jmeter.apache.org/) to do some load testing on both setups. I
> set up
> > JMeter to make 1000 POST requests sending valid activitystrea.msformatted
> > json to a single publisher url over the course of 5 seconds. I've
> attached
> > .csv files that give the average response the results, but I found that
> the
> > web service was .4 seconds faster on average. My concern is that these
> > results may not be consistent for the bigger use cases, and Camel may be
> > faster when deployed to multiple servers. It seems though that for early
> use
> > cases, the web service would be faster.
> >
> > I do want to make Streams as fast and as scalable as possible and I
> > appreciate the patience you guys are giving me through through this big
> > design change proposal. I would be happy to try a different load test
> setup
> > and, as always, would be happy to hear thoughts on the two
> implementations.
> >
> > Thanks
> >
> > -Danny
> >
> >> Date: Wed, 16 Oct 2013 18:49:02 -0400
> >
> >> Subject: Re: Using Camel for Routing
> >> From: jletourneau80@gmail.com
> >> To: dev@streams.incubator.apache.org
> >>
> >> I think it depends on how you define 'no problem' - deployment
> >> infrastructure is a big part of that - messaging provides more
> flexibility
> >> in deployment and capability additions at different layers of the
> >> application vs a single web app and a shotgun approach to evolving it
> >>
> >> I agree it should still be on the table as we define what is most
> >> important
> >> to this community
> >>
> >> On Wednesday, October 16, 2013, Danny Sullivan wrote:
> >>
> >> > I think if we sacrifice scalability in the long run by switching from
> >> > Camel to a web service then we should absolutely continue to use
> >> > messaging
> >> > and Camel. I'm confident, however, that applications exist that have a
> >> > web
> >> > service based model that service millions of requests and have no
> >> > problem
> >> > scaling without the help of Camel or any other messaging framework.
> >> > I'm going to keep the web-service branch separate for now, but I'd
> still
> >> > like to keep the option on the table until we explore building
> scalable
> >> > applications without the use of Camel. Though I can see the benefits
> of
> >> > using Camel, I think the complexity it adds makes it worth taking a
> look
> >> > at
> >> > other options.
> >> > -Danny
> >> >
> >> > > Date: Wed, 16 Oct 2013 16:26:43 -0400
> >> > > Subject: Re: Using Camel for Routing
> >> > > From: jletourneau80@gmail.com
> >> > > To: dev@streams.incubator.apache.org
> >> > >
> >> > > Yeah the more I think through this in terms of implications of a
> sole
> >> > > web app - I am skepticle that it scales to interesting use cases. I
> >> > > am not interested in streams to monitor "a few" facebook friends, I
> am
> >> > > interested in streams to monitor millions of events and activities
> >> > > generated from web site activity (clicks or tweets whatever) or
> >> > > enterprise assets across the globe.
> >> > >
> >> > > On Wed, Oct 16, 2013 at 4:15 PM, Chris Geer <chris@cxtsoftware.com>
> >> > wrote:
> >> > > > On Wed, Oct 16, 2013 at 11:58 AM, Danny Sullivan <
> >> > dsullivan7@hotmail.com>wrote:
> >> > > >
> >> > > >> I see the value in Streams as the ability of subscribers to get
> >> > activity
> >> > > >> based on filters.
> >> > > >
> >> > > >
> >> > > > I agree
> >> > > >
> >> > > >
> >> > > >> To be clear, I do not want to eliminate processing, but do want
> to
> >> > make
> >> > > >> the focus of processing tailored to the subscribers' filters. My
> >> > > >> more
> >> > grand
> >> > > >> vision is that only the most relevant activity will be returned
> to
> >> > > >> the
> >> > > >> Subscriber and similar activity will be aggregated into single
> >> > activities
> >> > > >> (this would be a little down the road). This processing would
> >> > > >> happen
> >> > after
> >> > > >> the activity is stored in the DB.
> >> > > >>
> >> > > >
> >> > > > This brings me back to the concern about having it all within one
> >> > > > web
> >> > app.
> >> > > > If the webapp accepts inputs, aggregates/filters it and manages
> it's
> >> > own
> >> > > > subscribers, how you do scale it, or provide redundancy? The
> beauty
> >> > > > of
> >> > > > using a message based system (i.e. Camel...along with Storm
> >> > > > possibly)
> >> > is
> >> > > > that it allows you to partition the application however you want.
> So
> >> > if you
> >> > > > want to have your ingest web service running on one machine and
> your
> >> > > > subscriber service running somewhere else and a third machine for
> >> > > > your
> >> > > > really resource intensive aggregation component, you can. With the
> >> > > > web
> >> > app
> >> > > > you could do vertical scaling on one machine, or load balancing of
> >> > > > all
> >> > > > components across multiple machines but it doesn't give you the
> save
> >> > > > flexibility.
> >> > > >
> >> > > >
> >> > > >> Do you imagine that Streams will come packaged with some
> processing
> >> > > >> components that will format other non-activitystrea.ms formatted
> >> > json? I
> >> > > >> do think it'd be pretty cool if you could fire up Streams, input
> a
> >> > couple
> >> > > >> Facebook Friends and people you follow on Twitter, and have all
> of
> >> > that
> >> > > >> activity posted in one place. That might be better suited for a
> new
> >> > > >> separate module (something like streams-format?) if we decide on
> >> > > >> the
> >> > > >> web-application route.
> >> > > >>
> >> > > >
> >> > > > I think it has to come with something to entice more users. If
> >> > everyone has
> >> > > > to build their own translation to even run "Hello World" it will
> be
> >> > > > a
> >> > > > larger barrier to entry.
> >> > > >
> >> > > >
> >> > > >> -Danny
> >> > > >>
> >> > > >> > Date: Wed, 16 Oct 2013 11:22:51 -0700
> >> > > >> > Subject: Re: Using Camel for Routing
> >> > > >> > From: chris@cxtsoftware.com
> >> > > >> > To: dev@streams.incubator.apache.org
> >> > > >> >
> >> > > >> > On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan <
> >> > dsullivan7@hotmail.com
> >> > > >> >wrote:
> >> > > >> >
> >> > > >> > > I think what is happening here is that we have different
> ideas
> >> > about
> >> > > >> the
> >> > > >> > > architecture of the application.
> >> > > >> > > Here's what I'm thinking:
> >> > > >> > > *All processing of activity would happen BEFORE it's posted
> to
> >> > > >> Streams*All
> >> > > >> > > activity posted to Streams is in activitystrea.ms format
> >> > > >> > > The Camel/EIP/OSGi way:
> >> > > >>
>

--047d7b2e073b32149a04e97f7f98--