streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Franklin <m.ben.frank...@gmail.com>
Subject Re: Starting the project
Date Thu, 22 May 2014 20:46:11 GMT
On Thu, May 22, 2014 at 11:46 AM, Steve Blackmon <sblackmon@apache.org>wrote:

> Renato,
>
> Yes there's no question we need to update the website - not least to
> help newcomers locate and get started with the examples.
>
> I would like to see us creating more integration tests like
> Top100FeedsTest which source and pass current-time real-world data
> through provider modules to confirm they work.  Social APIs are
> continuously evolving and this could help us catch any changes that we
> weren't aware of and prepared for prior to the change.
>
> Persistance module testing should also be done and would probably look
> a lot like gora's testing.  Let's follow up on ideas for aligning
> efforts along those lines.  A 10-20 minute build when integration
> testing is enabled isn't unreasonable.
>
> I'm not sure you meant to suggest it explicitly, but this note
> triggered a thought that the contrib modules perhaps could be split
> out into another repo as well.  Like examples, a failing test within a
> specific datasource or database integration shouldn't necessarily
> break the whole build.  APIs aren't perfect and may have transient
> issues, and given that our poms lean toward the bleeding edge
> occasionally database or database driver bugs are going to cause test
> failures.
>
> I like the way storm has handled this - they've use git submodules to
> associate useful integrations that make storm easier to build on with
> the top-level project, while essentially delegating maintenance of
> those modules to other projects/companies.  A similar model could work
> well for streams.
>

IMO, we should avoid git submodules for this.  I also don't think it needs
to be another entire repo, it could just be moved to a separate top-level
directory


>
> Thanks for the suggestions!
>
> Steve Blackmon
> sblackmon@apache.org
>
> On Thu, May 22, 2014 at 7:22 AM, Renato Marroquín Mogrovejo
> <renatoj.marroquin@gmail.com> wrote:
> > Maybe even something like this should be done but for the contrib
> modules.
> >
> >
> https://github.com/apache/incubator-streams/blob/master/streams-contrib/streams-provider-rss/src/test/java/org/apache/streams/rss/test/Top100FeedsTest.java
> >
> > That would be the something like examples. Wdyt?
> >
> >
> > Renato M.
> >
> >
> > 2014-05-22 12:05 GMT+02:00 Renato Marroquín Mogrovejo <
> > renatoj.marroquin@gmail.com>:
> >
> >> Hi Steve,
> >>
> >> Comments inline.
> >>
> >> 2014-05-21 20:59 GMT+02:00 Steve Blackmon <sblackmon@apache.org>:
> >>
> >> I will happily accept pull requests to streams-examples, although I
> >>> don't think it would need to change for an update to the provider to
> >>> be applied (except perhaps to add that verb filtering step)
> >>>
> >>
> >> This is what I meant.
> >>
> >>
> >>> I am willing to donate all of the examples, but I don't think they
> >>> belong in the main repo.  What if there were 50 examples many
> >>> containing integration tests?  Build would take forever - plus I don't
> >>> think that a single example failing a test should break the build.
> >>> Mahout suffered from really long-builds and handled it by separating
> >>> the examples out from core.
> >>>
> >>
> >> Well, over in Apache Giraph they have over 20 graph example algorithms
> and
> >> the way they test them is just by doing some simple computations, I
> don't
> >> know how long it takes, but I think it makes sense to at least try it
> and
> >> see how long it takes.
> >> Over in Gora we have a test suite with the different operations
> supported
> >> that is run over the different data stores, and depending on your
> machine,
> >> it might take from 10 - 20 mins. There was a time where the HBase tests
> >> were taking around 30 mins by themselves but we fixed them and are now
> run
> >> in less than 5 mins.
> >> Maybe your examples could be converted into a test suite? I mean reading
> >> from twitter(or a file) and persisting into different places. But
> anyhow,
> >> your examples should be referenced somewhere as it is what it makes
> sense
> >> for a newcomer :D
> >>
> >> When the project was hosting in SVN, we had the ability to create and
> >>> checkout different roots (the master pom was in a different root for
> >>> example).  If we can do the same using Git I propose we put the
> >>> examples in as a flat multi-module project that way.
> >>>
> >>
> >> +1
> >>
> >>
> >> Renato M.
> >>
> >> Any thoughts?
> >>>
> >>> Steve Blackmon
> >>> sblackmon@apache.org
> >>>
> >>> On Wed, May 21, 2014 at 11:59 AM, Renato Marroquín Mogrovejo
> >>> <renatoj.marroquin@gmail.com> wrote:
> >>> > I tried updating the site through ApacheCMS but I guess I don't have
> the
> >>> > rights to do it ;)
> >>> > This is what I got:
> >>> >
> >>> > The operation is forbidden by the server: Commit failed (details
> >>> follow)::
> >>> > Changing file
> >>> >
> >>>
> '/usr/local/cms/wc/streams/rmarroquin-NlOO6g/trunk/content/source-repository.mdtext'
> >>> > is forbidden by the server: Access to
> >>> >
> >>>
> '/repos/asf/!svn/txr/1596615-yz8u/incubator/streams/site/trunk/content/source-repository.mdtext'
> >>> > forbidden at /usr/local/cms/webgui/lib/ASF/CMS/WC/Commit.pm line 64
> >>> >
> >>> > So what is the best/easy way to update the site?
> >>> >
> >>> > @Steve
> >>> > Yeah, I finally got the time to try this out :) and looks very
> promising
> >>> > for sure!
> >>> > Is anybody working on STREAMS-57? Could I pick it up? I think this
> >>> should
> >>> > be pretty straight forward.
> >>> > And are there any plans to put your examples into the code base? It's
> >>> quite
> >>> > puzzling going from repo to another (plus the one from the site) to
> >>> > accomplish this. And then if I could work on STREAMS-57, would I have
> >>> to do
> >>> > the pull request into W2OGroup/streams-examples repo? I don't think
> so
> >>> > right? how would this work?
> >>> > Thanks!
> >>> >
> >>> >
> >>> > Renato M.
> >>> >
> >>> >
> >>> > 2014-05-21 17:42 GMT+02:00 Steve Blackmon <steve@blackmon.org>:
> >>> >
> >>> >> Cool I'm glad to hear you got it running. That exception occurs
> because
> >>> >> twitter sends a preamble listing all of your friends when you open
a
> >>> >> stream. The provider doesn't have logic to handle it yet, so it
> tries
> >>> to
> >>> >> build an activity and throws that exception.
> >>> >>
> >>> >> STREAMS-57 would suppress this warning - although once non-content
> >>> >> activities with new verbs like follow are generated we will want
a
> >>> >> filtering mechanism in place so it remains easy to build indexes
of
> >>> just
> >>> >> shares and posts.
> >>> >>
> >>> >> On Wed, May 21, 2014 at 9:23 AM, Renato Marroquín Mogrovejo <
> >>> >> renatoj.marroquin@gmail.com> wrote:
> >>> >> > Hey Matt,
> >>> >> >
> >>> >> > So the examples worked :D at least the one retrieving the
twitter
> >>> user's
> >>> >> > history. The one trying to track users activity keeps on getting
> this
> >>> >> > exception:
> >>> >> >
> >>> >> > 16:09:11.151 [pool-2-thread-2] DEBUG
> o.a.s.t.p.TwitterTypeConverter -
> >>> >> > ACTIVITY
> >>> >> > org.apache.streams.exceptions.ActivitySerializerException:
> >>> unrecognized
> >>> >> type
> >>> >> > at
> >>> >> >
> >>> >>
> >>> >>
> >>>
> org.apache.streams.twitter.serializer.TwitterJsonActivitySerializer.deserialize(TwitterJsonActivitySerializer.java:74)
> >>> >> > at
> >>> >> >
> >>> >>
> >>> >>
> >>>
> org.apache.streams.twitter.processor.TwitterTypeConverter.convert(TwitterTypeConverter.java:65)
> >>> >> > at
> >>> >> >
> >>> >>
> >>> >>
> >>>
> org.apache.streams.twitter.processor.TwitterTypeConverter.process(TwitterTypeConverter.java:146)
> >>> >> > at
> >>> >> >
> >>> >>
> >>> >>
> >>>
> org.apache.streams.local.tasks.StreamsProcessorTask.run(StreamsProcessorTask.java:72)
> >>> >> > at
> >>> >>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>> >> > at
> >>> >> >
> >>> >>
> >>> >>
> >>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>> >> > at
> >>> >> >
> >>> >>
> >>> >>
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>> >> > at java.lang.Thread.run(Thread.java:744)
> >>> >> >
> >>> >> > But everything seems to work in spite of that exception :)
Cool
> >>> stuff! So
> >>> >> > can I open a JIRA issue for updating the site and putting
the
> patch
> >>> there
> >>> >> > then?
> >>> >> > I will write another email with some other issues.
> >>> >> > Thanks Matt!
> >>> >> >
> >>> >> >
> >>> >> > Renato M.
> >>> >> >
> >>> >> >
> >>> >> > 2014-05-21 15:59 GMT+02:00 Matt Franklin <
> m.ben.franklin@gmail.com>:
> >>> >> >
> >>> >> >> On Wed, May 21, 2014 at 9:00 AM, Renato Marroquín Mogrovejo
<
> >>> >> >> renatoj.marroquin@gmail.com> wrote:
> >>> >> >>
> >>> >> >> > Hi Matt,
> >>> >> >> >
> >>> >> >> > Thank for pointing me to these examples. I will try
them out
> and
> >>> >> report
> >>> >> >> > back.
> >>> >> >> > But I have more questions now. I have found:
> >>> >> >> >
> >>> >> >> > https://github.com/apache/incubator-streams -->
referenced by
> the
> >>> >> >> examples
> >>> >> >> > you sent me and more active
> >>> >> >> > https://github.com/apache/streams --> referenced
by the
> website
> >>> >> >> >
> >>> >> >>
> >>> >> >> Oops. Yes it is the first one. The second one was a mirror
of
> SVN.
> >>> >> >>
> >>> >> >>
> >>> >> >> >
> >>> >> >> > Which one is the official one? I guess the first
one right.
> >>> Shouldn't
> >>> >> we
> >>> >> >> > change this on the website? does that mean that Apache
Streams
> >>> have
> >>> >> two
> >>> >> >> git
> >>> >> >> > repos? I can fix that on the website if you point
me to the
> right
> >>> >> place.
> >>> >> >> >
> >>> >> >>
> >>> >> >> Website is in SVN using Apache CMS.
> >>> >> >>
> >>> >> >>
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > Renato M.
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > 2014-05-21 2:14 GMT+02:00 Matt Franklin <
> m.ben.franklin@gmail.com
> >>> >:
> >>> >> >> >
> >>> >> >> > > On Tue, May 20, 2014 at 4:28 PM, Renato Marroquín
Mogrovejo <
> >>> >> >> > > renatoj.marroquin@gmail.com> wrote:
> >>> >> >> > >
> >>> >> >> > > > Sorry I pressed the sent button before
I finished writing
> the
> >>> >> email.
> >>> >> >> > > > Anyways . . . Tomcat 7.0.39, ServiceMix
5.0.1 and Cassandra
> >>> 2.0.4.
> >>> >> >> > > > But I am getting this exception:
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> > > We are unfortunately in the middle of converging
the new
> model
> >>> and
> >>> >> the
> >>> >> >> > old
> >>> >> >> > > model. This runtime should be working, but it
needs some
> work to
> >>> >> >> ensure
> >>> >> >> > > that the new components in streams-contrib work.
If you are
> >>> looking
> >>> >> to
> >>> >> >> > get
> >>> >> >> > > started with a newer runtime, streams-runtime-local
is a good
> >>> place.
> >>> >> >> > >
> >>> >> >> > > Steve posted a link to some examples he wrote
for how to get
> >>> >> started:
> >>> >> >> > >
> >>> >> >> > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>> >>
> >>>
> https://github.com/w2ogroup/streams-examples/tree/master/twitter-history-elasticsearch
> >>> >> >> > >
> >>> >> >> > > We need to create some of our own examples (or
ask him
> kindly to
> >>> >> donate
> >>> >> >> > > some to us), but until the site is fixed, this
is a good
> start.
> >>> I
> >>> >> will
> >>> >> >> > > create an issue for the Caml runtime issue you
found here.
> >>> >> >> > >
> >>> >> >> > >
> >>> >> >> > > >
> >>> >> >> > > > May 20, 2014 10:15:29 PM
> >>> org.apache.catalina.startup.ContextConfig
> >>> >> >> > > > getServletContainerInitializer
> >>> >> >> > > > SEVERE: The ServletContentInitializer
> >>> >> >> > > > [org.springframework.web.SpringServletContainerInitializer]
> >>> could
> >>> >> not
> >>> >> >> > be
> >>> >> >> > > > created
> >>> >> >> > > > java.lang.ClassNotFoundException:
> >>> >> >> > > > org.springframework.web.SpringServletContainerInitializer
> >>> >> >> > > > at
> >>> >> >> > > >
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>> >>
> >>>
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1713)
> >>> >> >> > > > at
> >>> >> >> > > >
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>> >>
> >>>
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1558)
> >>> >> >> > > > at java.lang.Class.forName0(Native Method)
> >>> >> >> > > > at java.lang.Class.forName(Class.java:270)
> >>> >> >> > > > at
> >>> >> >> > > >
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>> >>
> >>>
> org.apache.catalina.startup.ContextConfig.getServletContainerInitializer(ContextConfig.java:1659)
> >>> >> >> > > > at
> >>> >> >> > > >
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>> >>
> >>>
> org.apache.catalina.startup.ContextConfig.processServletContainerInitializers(ContextConfig.java:1569)
> >>> >> >> > > > at
> >>> >> >> > > >
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>> >>
> >>>
> org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1277)
> >>> >> >> > > > at
> >>> >> >> > > >
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>> >>
> >>>
> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878)
> >>> >> >> > > > at
> >>> >> >> > > >
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>> >>
> >>>
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:369)
> >>> >> >> > > > at
> >>> >> >> > > >
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>> >>
> >>>
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
> >>> >> >> > > > at
> >>> >> >> > > >
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>> >>
> >>>
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
> >>> >> >> > > > at
> >>> >> >> > > >
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>> >>
> >>>
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5269)
> >>> >> >> > > > at
> >>> >> >> > > >
> >>> >> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> >>> >> >> > > >
> >>> >> >> > > > That let me to [2]. Is this related to
this? or is it just
> me
> >>> >> doing
> >>> >> >> > > things
> >>> >> >> > > > wrong here?
> >>> >> >> > > > Thanks for any pointers!
> >>> >> >> > > >
> >>> >> >> > > >
> >>> >> >> > > > Renato M.
> >>> >> >> > > >
> >>> >> >> > > > 2014-05-20 22:25 GMT+02:00 Renato Marroquín
Mogrovejo <
> >>> >> >> > > > renatoj.marroquin@gmail.com>:
> >>> >> >> > > >
> >>> >> >> > > > > Hey guys,
> >>> >> >> > > > >
> >>> >> >> > > > > This is a long overdue email but with
little steps I will
> >>> start
> >>> >> >> going
> >>> >> >> > > :)
> >>> >> >> > > > > So I have been trying to run the whole
project based
> on[1].
> >>> As
> >>> >> >> there
> >>> >> >> > is
> >>> >> >> > > > no
> >>> >> >> > > > > version for any of the pieces involved
in running this
> (if
> >>> there
> >>> >> is
> >>> >> >> > > > please
> >>> >> >> > > > > point me to them, if not, I am happy
to put them on the
> site
> >>> >> >> > > somewhere).
> >>> >> >> > > > I
> >>> >> >> > > > > chose
> >>> >> >> > > > >
> >>> >> >> > > > >
> >>> >> >> > > > >
> >>> >> >> > > > > [1]
> http://streams.incubator.apache.org/streams-usage.html
> >>> >> >> > > > > [2]
> >>> >> >> > > > >
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>> >>
> >>>
> http://stackoverflow.com/questions/7692497/tomcat-wont-load-my-meta-inf-services-javax-servlet-servletcontainerinitializ
> >>> >> >> > > > >
> >>> >> >> > > >
> >>> >> >> > >
> >>> >> >> >
> >>> >> >>
> >>> >>
> >>>
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message