streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Blackmon [W2O Digital]" <sblack...@w2odigital.com>
Subject Re: Using Camel for Routing
Date Wed, 16 Oct 2013 20:12:29 GMT
I think a community driven approach to normalizing APIs into activities
would be hugely valuable.

Capture and normalization is the sole focus of some of the modules we¹ve
been working on.  Currently they can only run within storm but there¹s no
reason they shouldn¹t be able to deploy into any container that can spawn
threads which can open socket connections.

We intend to push some of this soon - with initial out-of-the-box support
for twitter and moreover metabase.  My hope is that we can convince
developers within data providers to join the project to perfect and
maintain those modules.

Steve Blackmon
Director, Data Sciences
W2O Group
3000 E Cesar Chavez St
Austin, Texas 78701
cell 512.965.0451 | work 512.402.6366
twitter @steveblackmon







On 10/16/13, 1:58 PM, "Danny Sullivan" <dsullivan7@hotmail.com> wrote:

>I see the value in Streams as the ability of subscribers to get activity
>based on filters. To be clear, I do not want to eliminate processing, but
>do want to make the focus of processing tailored to the subscribers'
>filters. My more grand vision is that only the most relevant activity
>will be returned to the Subscriber and similar activity will be
>aggregated into single activities (this would be a little down the road).
>This processing would happen after the activity is stored in the DB.
>Do you imagine that Streams will come packaged with some processing
>components that will format other non-activitystrea.ms formatted json? I
>do think it'd be pretty cool if you could fire up Streams, input a couple
>Facebook Friends and people you follow on Twitter, and have all of that
>activity posted in one place. That might be better suited for a new
>separate module (something like streams-format?) if we decide on the
>web-application route.
>-Danny
>
>> Date: Wed, 16 Oct 2013 11:22:51 -0700
>> Subject: Re: Using Camel for Routing
>> From: chris@cxtsoftware.com
>> To: dev@streams.incubator.apache.org
>>
>> On Wed, Oct 16, 2013 at 11:12 AM, Danny Sullivan
>><dsullivan7@hotmail.com>wrote:
>>
>> > I think what is happening here is that we have different ideas about
>>the
>> > architecture of the application.
>> > Here's what I'm thinking:
>> > *All processing of activity would happen BEFORE it's posted to
>>Streams*All
>> > activity posted to Streams is in activitystrea.ms format
>> > The Camel/EIP/OSGi way:
>> > *The user is able to post anything to Streams, doesn't matter what
>>format
>> > it's in*The user can add processing components to Streams that format
>> > everything that is input to activitystrea.ms format, these can be
>>easily
>> > plugged in, moved around, or taken out
>> > If we decide to go the second route, I think Camel would be the best
>>way
>> > to go. However, I'd like to make the case for the first approach. The
>> > application can be quickly started and tested by a developer and user
>>with
>> > the the first approach. If the user decides they want to experiment
>>with
>> > twitter/facebook data (like Chris suggested) they can write a custom
>> > application (which can be a camel/eip application or even a python,
>>ruby,
>> > etc application) that takes in the twitter/facebook data, formats it
>>to
>> > activitystrea.ms format and then POSTs it to Streams. Also note: the
>> > webapp would NOT need to be restarted to add these. I don't think
>>we'd lose
>> > a lot by going this route: in order to add the custom processing
>>components
>> > in Camel, a Java component would still need to be written before it's
>> > plugged in to the camel.xml (and I think you would be tied to Java
>>with
>> > this approach).
>> > Quite simply: We should keep formatting activity separate processing
>> > activity
>> > Perhaps I am missing some processing that could happen in between an
>> > activity's entrance to Streams and it's storage? If this is the case,
>>I
>> > would make the argument that this processing should happen on an
>>activities
>> > EXIT from Streams (there should be a way for a Subscriber to specify,
>>in
>> > its filters or by some other means, the activity that it wants to see)
>> >
>>
>> Danny, what do you see as the scope for Streams? If you eliminate data
>> input in non-activitystrea.ms format and eliminate processing data what
>> does it do, just store each message (unmodified) and output it to
>> subscribers?
>>
>>
>> > -Danny
>> >
>> > > Date: Wed, 16 Oct 2013 10:11:23 -0700
>> > > Subject: Re: Using Camel for Routing
>> > > From: chris@cxtsoftware.com
>> > > To: dev@streams.incubator.apache.org
>> > >
>> > > Jason,
>> > >
>> > > I agree with you. I guess my vision of streams was a little grander
>>in
>> > the
>> > > long run where you could configure the system to accept input from
>> > various
>> > > sources (facebook, twitter...) and process them in certain ways. So
>>it
>> > > would have components that would know how to talk with Twitter and
>> > convert
>> > > tweets to activities then send it to the right process. So
>>depending on
>> > how
>> > > you configure your system, it would create the proper routes from
>> > component
>> > > to component to allow processing dynamically which would require a
>> > > framework like Camel (don't think Storm helps us here since it's a
>> > compile
>> > > time configuration to my understanding). The nice thing about
>>something
>> > > like camel is it allows you to reconfigure all the routes at
>>runtime.
>> > This
>> > > is another reason I liked OSGI support as well as it allows
>>deploying new
>> > > components without restarting the server (webapp). Of course to
>>scale it
>> > > would require being able to deploy the configuration across multiple
>> > > machines which gets fun.
>> > >
>> > > Maybe my vision is too big for this project but it's where my head
>>has
>> > been
>> > > at.
>> > >
>> > > Chris
>> > >
>> > >
>> > > On Wed, Oct 16, 2013 at 7:54 AM, Jason Letourneau
>> > > <jletourneau80@gmail.com>wrote:
>> > >
>> > > > My only real complaint with this approach is that interjecting new
>> > > > components (processing components) becomes less plug and play
>>without
>> > > > writing Java code in streams - with the Camel routing approach,
>> > > > streams Java is a black box and you just change the camel xml.
>> > > >
>> > > > The primary rationale in using Camel was gaining all of the
>> > > > architectural benefits one gains from  enterprise integration
>>patterns
>> > > > (truth: simplicity is not one and is a tradeoff for flexibility,
>>but
>> > > > generally is considered a universal truth not one relegated to
>>Camel
>> > > > and EIP).
>> > > >
>> > > > Jason
>> > > >
>> > > > On Fri, Oct 11, 2013 at 10:16 AM, Danny Sullivan <
>> > dsullivan7@hotmail.com>
>> > > > wrote:
>> > > > > Hey All,
>> > > > > I would like to merge the webservice branch with trunk. I think
>> > making
>> > > > streams web service based allows for a better focus on the goal
>>of the
>> > > > application: process activitystrea.ms formatted activity json.
>>While
>> > > > moving to a webservice model would lose us support for many of
>>the use
>> > > > cases Chris mentioned (i.e twitter and facebook), I think the
>>downsides
>> > > > would be mitigated by allowing and encouraging users to write
>>custom
>> > > > appenders that take in twitter/facebook data, format it to
>> > > > activitystrea.ms standards, and POST to the Streams Server. These
>> > > > appenders could even be written in any language as long as the
>>output
>> > is
>> > > > activitystrea.ms json. Compared to the trunk branch, I think the
>>web
>> > > > service design employs the "black box" idiom in a much simpler
>> > fashion. I
>> > > > would be happy to hear contradicting thoughts and would encourage
>>a
>> > > > discussion on this design switch.
>> > > > > Thanks!
>> > > > > Danny
>> > > > >> From: dsullivan7@hotmail.com
>> > > > >> To: dev@streams.incubator.apache.org
>> > > > >> Subject: RE: Using Camel for Routing
>> > > > >> Date: Fri, 4 Oct 2013 16:07:06 -0400
>> > > > >>
>> > > > >> I wrote up something quickly so I could demo the functionality
>>as a
>> > web
>> > > > service as opposed to Camel. The new branch is available here:
>> > > > >>
>> >
>>http://cp.mcafee.com/d/FZsScz8A71NJ5BV4QsI6zAQsIK6XCQQmhPMWWrMUSCyMy-yCOy
>>edFEIzDxRQTDzqqb339EVuujjdQ87XoGHm0ac9RgAhBfj-ndC4WEi8ODF_bCXKE_XWr_nV55B
>>ZVZBXHTbEKe9YyVtxNUsqeumKzp5dmVEVvVkffGhBrwqrhdI6XYCej79zANOoUTsSjDdqympg
>>P3JmSNf-00V7RGQPqeVa5CO6PH7ajz-GQMx7om9_j01k-9ClLMSxVAL7VJNwnojKnEgfSNlmI
>>3z2tk94pjQ_BMlisvRmDixpIxIWNXlJIj_w09J5wQsIFCMmd96y09QB0yq81blvfd43JoCy05
>>RrDa14TsT_Py-uR5BLSC
>> > > > >> note that the new publishing and subscribing urls are:
>> > > > >>
>> > > >
>> >
>>http://cp.mcafee.com/d/1jWVIg43qbbO9EVod79EVpsdTdFEIzDxRQTxNJd5x5Z5dB4srj
>>hp7f3HFLf6QQm66jhOYYCCrEgfSNlmI0kojGx8zauDYKrc9RgAhBfj-ndTth_TQT-LOabbXPX
>>bTnKnhssjV5OX3zMUQsYJt6OaqJPhO_OEuvkzaT0QSCrodTVcsCej79zANNKVI06vmSDAWQCA
>>ZtW-kD7ZlEn05CHboISgfBgDYgnV6l9_406vmSDAWQCAZtW-kD7ZlEn05CHbphIxiQp4fy2_8
>>OFfUCQfcBU_dKc2X2tOZ21-SaGRwsojGx8zauDYK2Gjz-GQWkbdAdDmfqJJyvY01dEI6zBBcS
>>2NF8Qg1eAE4jh09qHVVEwtH4Qg0KHsVg8CXC_C3cpxp4WcOz
>> > > > >> I used a Spring Web Service only because I was more familiar
>>with
>> > it, I
>> > > > would like to research other alternatives (CXF) if it's clear
>>that a
>> > web
>> > > > service is a realistic alternative. The main changes to the
>> > application are
>> > > > the web controller located here:
>> > > > >>
>> > > >
>> >
>>http://cp.mcafee.com/d/2DRPoO83gAd6QmnAjhOMqejhOOUrKrjhp7f3HFL3zqqb2bWara
>>8USCyOeu7njuudFEIccCzBVVdcTgwvJyGJo0EMDl2h6kZfVsSojGx8zauDYKrKWz_LFLZvAkm
>>nTDSnKLsKyUUDObBS77xNEVVqWdAkRrCzB_BgY-F6lK1FJASMrLOoVcsCej79zztPpesRG9pB
>>3ceRrr4_U03AvmHjdEXAEmr8reIsFefWHj24txoDZc05jUCpm_ajz-GQbw2OAroHIEo8WpT00
>>eqJJyvVisvRmCaCrpqI9NGuvYs69N_lq0bV0iH2sqDD_7x7jeCQfcBU_dKc2X2tOZ21-SaGRw
>>sojGx8zauDYK2Gjz-GQWkbdAdDmfqJJyvY01dEI6zBBcS2NF8Qg1eAE4jh09qHVVEwtH4Qg0K
>>HsVg8CXCTVkAs4DJ
>> > > > >> Which handles the input and output of the application, which
>>were
>> > > > handled by the camel context before. The other main change is the
>>use
>> > of
>> > > > the streams-components module located here:
>> > > > >>
>> > > >
>> >
>>http://cp.mcafee.com/d/2DRPoArhpuhd7b1EVd7bbxKVJd5AsYeKCYedFEI8LEFIEzzqqb
>>8VUttdVUSCyMMOqenDAQPt21-SaGRw2z2tk94pjQ_BPpxeG4ycFWvOVKXGf--C_R-hhpvuvpu
>>WZOWbzyv8Knosu76zDBHEShjlKqen-l3PWApmU6CS3r1K_9zANOoVcsCedTdAVPmEBCkcMXlJ
>>Ij_w0ehZqJcSzKixpIxIWNOAU_GJc8hS5yvQM0lfypBrYFefWHgKKJm7MDj3q7CiYvCT61txe
>>Vux0_r5lqMec9RgAhBfj-n1l9N_lqta5CO6PH7JmSNf-00CQm3hOOCr1oQAq80Dik29Ew4JlY
>>YQgeRyq80nlKsE4jtPqkR9rHK5
>> > > > >> This module contains service classes that return the correct
>>output
>> > on
>> > > > each request sent to the controller. I reused the Publisher and
>> > Subscriber
>> > > > objects as well as the activityConsumerWarehouse and
>> > > > activitySubscriberWarehouse, I think they would be moved to the
>> > > > streams-components package if we want to go this route.
>> > > > >> Overall, I think the flow through the application is much
more
>> > > > understandable. Previously, a developer would have to follow an
>> > activity
>> > > > through the camelContext.xml as well as through dynamically
>>created
>> > routes
>> > > > verses the web service which provides clear entry and exit points
>>to
>> > the
>> > > > application. I'd really like to hear feedback on this one, I
>>really
>> > don't
>> > > > want to lose any efficiencies that we could be from
>> > > > messaging/osgi/camel/activemq.
>> > > > >> Thanks,
>> > > > >> Danny
>> > > > >>
>> > > > >> > From: dsullivan7@hotmail.com
>> > > > >> > To: dev@streams.incubator.apache.org
>> > > > >> > Subject: RE: Using Camel for Routing
>> > > > >> > Date: Thu, 3 Oct 2013 20:17:13 -0400
>> > > > >> >
>> > > > >> > Thanks for the update Jason. I'll keep the Spring MVC
app in
>>a
>> > > > separate branch (hopefully it won't take too long) and then
>>perhaps we
>> > can
>> > > > reassess having both component and web container deployment? I
>>think
>> > it's a
>> > > > good idea to keep both options in mind especially because this
>> > technology
>> > > > is relatively young.
>> > > > >> > Thanks!
>> > > > >> > Danny
>> > > > >> >
>> > > > >> > > Date: Thu, 3 Oct 2013 19:50:59 -0400
>> > > > >> > > Subject: Re: Using Camel for Routing
>> > > > >> > > From: jletourneau80@gmail.com
>> > > > >> > > To: dev@streams.incubator.apache.org
>> > > > >> > >
>> > > > >> > > Because of the osgi container support in addition
to
>>deployment
>> > as
>> > > > a web
>> > > > >> > > app - when last we visited this topic - I think
the
>>majority
>> > was in
>> > > > favor
>> > > > >> > > of keeping the component style deployment in addition
to
>>web
>> > > > container
>> > > > >> > > deployment
>> > > > >> > >
>> > > > >> > > On Thursday, October 3, 2013, Danny Sullivan wrote:
>> > > > >> > >
>> > > > >> > > > Hey all,
>> > > > >> > > > I've been considering taking a stab at moving
streams
>>from
>> > it's
>> > > > current
>> > > > >> > > > configuration with Camel to a Spring Web Service.
>>Concerns
>> > over
>> > > > failover
>> > > > >> > > > and load balancing would be mitigated by the
Cassandra DB
>> > > > (running on 9160)
>> > > > >> > > > and Storm (which depends on an instance of
Zookeeper on
>>2181).
>> > > > Before I
>> > > > >> > > > start lancing any windmills I thought it'd
be nice to get
>> > input
>> > > > on why the
>> > > > >> > > > design choice was made to base the application
around
>>Camel?
>> > > > >> > > > I think that tracking the flow of an activity
through the
>> > > > application will
>> > > > >> > > > be much easier if we use Spring MVC. I'll
set up a new
>>branch
>> > > > using the MVC
>> > > > >> > > > architecture and I'll keep the community updated
as I go
>> > along.
>> > > > >> > > > Danny
>> > > > >> >
>> > > > >>
>> > > > >
>> > > >
>> >
>> >
>




CONFIDENTIALITY NOTICE: This e-mail, along with any documents, files, or attachments, may
contain information that is confidential, privileged, or otherwise exempt from disclosure.
If you are not the intended recipient or person responsible for delivering it to the intended
recipient, you are hereby notified that any disclosure, copying, printing, distribution or
use of any information contained in or attached to this e-mail is strictly prohibited. If
you have received this e-mail in error, please immediately notify the sender and delete the
original e-mail and its attachments without reading, printing, or saving in any manner. This
e-mail message should not be interpreted to include a digital or electronic signature that
can be used to authenticate an agreement, contract or other legal document, nor to reflect
an intention to be bound to any legally-binding agreement or contract. Your cooperation is
appreciated. Thank you.

Mime
View raw message