streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Blackmon <sblack...@apache.org>
Subject Re: Starting the project
Date Thu, 22 May 2014 15:46:00 GMT
Renato,

Yes there's no question we need to update the website - not least to
help newcomers locate and get started with the examples.

I would like to see us creating more integration tests like
Top100FeedsTest which source and pass current-time real-world data
through provider modules to confirm they work.  Social APIs are
continuously evolving and this could help us catch any changes that we
weren't aware of and prepared for prior to the change.

Persistance module testing should also be done and would probably look
a lot like gora's testing.  Let's follow up on ideas for aligning
efforts along those lines.  A 10-20 minute build when integration
testing is enabled isn't unreasonable.

I'm not sure you meant to suggest it explicitly, but this note
triggered a thought that the contrib modules perhaps could be split
out into another repo as well.  Like examples, a failing test within a
specific datasource or database integration shouldn't necessarily
break the whole build.  APIs aren't perfect and may have transient
issues, and given that our poms lean toward the bleeding edge
occasionally database or database driver bugs are going to cause test
failures.

I like the way storm has handled this - they've use git submodules to
associate useful integrations that make storm easier to build on with
the top-level project, while essentially delegating maintenance of
those modules to other projects/companies.  A similar model could work
well for streams.

Thanks for the suggestions!

Steve Blackmon
sblackmon@apache.org

On Thu, May 22, 2014 at 7:22 AM, Renato Marroquín Mogrovejo
<renatoj.marroquin@gmail.com> wrote:
> Maybe even something like this should be done but for the contrib modules.
>
> https://github.com/apache/incubator-streams/blob/master/streams-contrib/streams-provider-rss/src/test/java/org/apache/streams/rss/test/Top100FeedsTest.java
>
> That would be the something like examples. Wdyt?
>
>
> Renato M.
>
>
> 2014-05-22 12:05 GMT+02:00 Renato Marroquín Mogrovejo <
> renatoj.marroquin@gmail.com>:
>
>> Hi Steve,
>>
>> Comments inline.
>>
>> 2014-05-21 20:59 GMT+02:00 Steve Blackmon <sblackmon@apache.org>:
>>
>> I will happily accept pull requests to streams-examples, although I
>>> don't think it would need to change for an update to the provider to
>>> be applied (except perhaps to add that verb filtering step)
>>>
>>
>> This is what I meant.
>>
>>
>>> I am willing to donate all of the examples, but I don't think they
>>> belong in the main repo.  What if there were 50 examples many
>>> containing integration tests?  Build would take forever - plus I don't
>>> think that a single example failing a test should break the build.
>>> Mahout suffered from really long-builds and handled it by separating
>>> the examples out from core.
>>>
>>
>> Well, over in Apache Giraph they have over 20 graph example algorithms and
>> the way they test them is just by doing some simple computations, I don't
>> know how long it takes, but I think it makes sense to at least try it and
>> see how long it takes.
>> Over in Gora we have a test suite with the different operations supported
>> that is run over the different data stores, and depending on your machine,
>> it might take from 10 - 20 mins. There was a time where the HBase tests
>> were taking around 30 mins by themselves but we fixed them and are now run
>> in less than 5 mins.
>> Maybe your examples could be converted into a test suite? I mean reading
>> from twitter(or a file) and persisting into different places. But anyhow,
>> your examples should be referenced somewhere as it is what it makes sense
>> for a newcomer :D
>>
>> When the project was hosting in SVN, we had the ability to create and
>>> checkout different roots (the master pom was in a different root for
>>> example).  If we can do the same using Git I propose we put the
>>> examples in as a flat multi-module project that way.
>>>
>>
>> +1
>>
>>
>> Renato M.
>>
>> Any thoughts?
>>>
>>> Steve Blackmon
>>> sblackmon@apache.org
>>>
>>> On Wed, May 21, 2014 at 11:59 AM, Renato Marroquín Mogrovejo
>>> <renatoj.marroquin@gmail.com> wrote:
>>> > I tried updating the site through ApacheCMS but I guess I don't have the
>>> > rights to do it ;)
>>> > This is what I got:
>>> >
>>> > The operation is forbidden by the server: Commit failed (details
>>> follow)::
>>> > Changing file
>>> >
>>> '/usr/local/cms/wc/streams/rmarroquin-NlOO6g/trunk/content/source-repository.mdtext'
>>> > is forbidden by the server: Access to
>>> >
>>> '/repos/asf/!svn/txr/1596615-yz8u/incubator/streams/site/trunk/content/source-repository.mdtext'
>>> > forbidden at /usr/local/cms/webgui/lib/ASF/CMS/WC/Commit.pm line 64
>>> >
>>> > So what is the best/easy way to update the site?
>>> >
>>> > @Steve
>>> > Yeah, I finally got the time to try this out :) and looks very promising
>>> > for sure!
>>> > Is anybody working on STREAMS-57? Could I pick it up? I think this
>>> should
>>> > be pretty straight forward.
>>> > And are there any plans to put your examples into the code base? It's
>>> quite
>>> > puzzling going from repo to another (plus the one from the site) to
>>> > accomplish this. And then if I could work on STREAMS-57, would I have
>>> to do
>>> > the pull request into W2OGroup/streams-examples repo? I don't think so
>>> > right? how would this work?
>>> > Thanks!
>>> >
>>> >
>>> > Renato M.
>>> >
>>> >
>>> > 2014-05-21 17:42 GMT+02:00 Steve Blackmon <steve@blackmon.org>:
>>> >
>>> >> Cool I'm glad to hear you got it running. That exception occurs because
>>> >> twitter sends a preamble listing all of your friends when you open a
>>> >> stream. The provider doesn't have logic to handle it yet, so it tries
>>> to
>>> >> build an activity and throws that exception.
>>> >>
>>> >> STREAMS-57 would suppress this warning - although once non-content
>>> >> activities with new verbs like follow are generated we will want a
>>> >> filtering mechanism in place so it remains easy to build indexes of
>>> just
>>> >> shares and posts.
>>> >>
>>> >> On Wed, May 21, 2014 at 9:23 AM, Renato Marroquín Mogrovejo <
>>> >> renatoj.marroquin@gmail.com> wrote:
>>> >> > Hey Matt,
>>> >> >
>>> >> > So the examples worked :D at least the one retrieving the twitter
>>> user's
>>> >> > history. The one trying to track users activity keeps on getting
this
>>> >> > exception:
>>> >> >
>>> >> > 16:09:11.151 [pool-2-thread-2] DEBUG o.a.s.t.p.TwitterTypeConverter
-
>>> >> > ACTIVITY
>>> >> > org.apache.streams.exceptions.ActivitySerializerException:
>>> unrecognized
>>> >> type
>>> >> > at
>>> >> >
>>> >>
>>> >>
>>> org.apache.streams.twitter.serializer.TwitterJsonActivitySerializer.deserialize(TwitterJsonActivitySerializer.java:74)
>>> >> > at
>>> >> >
>>> >>
>>> >>
>>> org.apache.streams.twitter.processor.TwitterTypeConverter.convert(TwitterTypeConverter.java:65)
>>> >> > at
>>> >> >
>>> >>
>>> >>
>>> org.apache.streams.twitter.processor.TwitterTypeConverter.process(TwitterTypeConverter.java:146)
>>> >> > at
>>> >> >
>>> >>
>>> >>
>>> org.apache.streams.local.tasks.StreamsProcessorTask.run(StreamsProcessorTask.java:72)
>>> >> > at
>>> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> >> > at
>>> >> >
>>> >>
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> >> > at
>>> >> >
>>> >>
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> >> > at java.lang.Thread.run(Thread.java:744)
>>> >> >
>>> >> > But everything seems to work in spite of that exception :) Cool
>>> stuff! So
>>> >> > can I open a JIRA issue for updating the site and putting the patch
>>> there
>>> >> > then?
>>> >> > I will write another email with some other issues.
>>> >> > Thanks Matt!
>>> >> >
>>> >> >
>>> >> > Renato M.
>>> >> >
>>> >> >
>>> >> > 2014-05-21 15:59 GMT+02:00 Matt Franklin <m.ben.franklin@gmail.com>:
>>> >> >
>>> >> >> On Wed, May 21, 2014 at 9:00 AM, Renato Marroquín Mogrovejo
<
>>> >> >> renatoj.marroquin@gmail.com> wrote:
>>> >> >>
>>> >> >> > Hi Matt,
>>> >> >> >
>>> >> >> > Thank for pointing me to these examples. I will try them
out and
>>> >> report
>>> >> >> > back.
>>> >> >> > But I have more questions now. I have found:
>>> >> >> >
>>> >> >> > https://github.com/apache/incubator-streams --> referenced
by the
>>> >> >> examples
>>> >> >> > you sent me and more active
>>> >> >> > https://github.com/apache/streams --> referenced by
the website
>>> >> >> >
>>> >> >>
>>> >> >> Oops. Yes it is the first one. The second one was a mirror
of SVN.
>>> >> >>
>>> >> >>
>>> >> >> >
>>> >> >> > Which one is the official one? I guess the first one right.
>>> Shouldn't
>>> >> we
>>> >> >> > change this on the website? does that mean that Apache
Streams
>>> have
>>> >> two
>>> >> >> git
>>> >> >> > repos? I can fix that on the website if you point me to
the right
>>> >> place.
>>> >> >> >
>>> >> >>
>>> >> >> Website is in SVN using Apache CMS.
>>> >> >>
>>> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> > Renato M.
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > 2014-05-21 2:14 GMT+02:00 Matt Franklin <m.ben.franklin@gmail.com
>>> >:
>>> >> >> >
>>> >> >> > > On Tue, May 20, 2014 at 4:28 PM, Renato Marroquín
Mogrovejo <
>>> >> >> > > renatoj.marroquin@gmail.com> wrote:
>>> >> >> > >
>>> >> >> > > > Sorry I pressed the sent button before I finished
writing the
>>> >> email.
>>> >> >> > > > Anyways . . . Tomcat 7.0.39, ServiceMix 5.0.1
and Cassandra
>>> 2.0.4.
>>> >> >> > > > But I am getting this exception:
>>> >> >> > > >
>>> >> >> > >
>>> >> >> > > We are unfortunately in the middle of converging
the new model
>>> and
>>> >> the
>>> >> >> > old
>>> >> >> > > model. This runtime should be working, but it needs
some work to
>>> >> >> ensure
>>> >> >> > > that the new components in streams-contrib work.
If you are
>>> looking
>>> >> to
>>> >> >> > get
>>> >> >> > > started with a newer runtime, streams-runtime-local
is a good
>>> place.
>>> >> >> > >
>>> >> >> > > Steve posted a link to some examples he wrote for
how to get
>>> >> started:
>>> >> >> > >
>>> >> >> > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>> >>
>>> https://github.com/w2ogroup/streams-examples/tree/master/twitter-history-elasticsearch
>>> >> >> > >
>>> >> >> > > We need to create some of our own examples (or ask
him kindly to
>>> >> donate
>>> >> >> > > some to us), but until the site is fixed, this is
a good start.
>>> I
>>> >> will
>>> >> >> > > create an issue for the Caml runtime issue you found
here.
>>> >> >> > >
>>> >> >> > >
>>> >> >> > > >
>>> >> >> > > > May 20, 2014 10:15:29 PM
>>> org.apache.catalina.startup.ContextConfig
>>> >> >> > > > getServletContainerInitializer
>>> >> >> > > > SEVERE: The ServletContentInitializer
>>> >> >> > > > [org.springframework.web.SpringServletContainerInitializer]
>>> could
>>> >> not
>>> >> >> > be
>>> >> >> > > > created
>>> >> >> > > > java.lang.ClassNotFoundException:
>>> >> >> > > > org.springframework.web.SpringServletContainerInitializer
>>> >> >> > > > at
>>> >> >> > > >
>>> >> >> > > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>> >>
>>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1713)
>>> >> >> > > > at
>>> >> >> > > >
>>> >> >> > > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>> >>
>>> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1558)
>>> >> >> > > > at java.lang.Class.forName0(Native Method)
>>> >> >> > > > at java.lang.Class.forName(Class.java:270)
>>> >> >> > > > at
>>> >> >> > > >
>>> >> >> > > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>> >>
>>> org.apache.catalina.startup.ContextConfig.getServletContainerInitializer(ContextConfig.java:1659)
>>> >> >> > > > at
>>> >> >> > > >
>>> >> >> > > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>> >>
>>> org.apache.catalina.startup.ContextConfig.processServletContainerInitializers(ContextConfig.java:1569)
>>> >> >> > > > at
>>> >> >> > > >
>>> >> >> > > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>> >>
>>> org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1277)
>>> >> >> > > > at
>>> >> >> > > >
>>> >> >> > > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>> >>
>>> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878)
>>> >> >> > > > at
>>> >> >> > > >
>>> >> >> > > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>> >>
>>> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:369)
>>> >> >> > > > at
>>> >> >> > > >
>>> >> >> > > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>> >>
>>> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
>>> >> >> > > > at
>>> >> >> > > >
>>> >> >> > > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>> >>
>>> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
>>> >> >> > > > at
>>> >> >> > > >
>>> >> >> > > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>> >>
>>> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5269)
>>> >> >> > > > at
>>> >> >> > > >
>>> >> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
>>> >> >> > > >
>>> >> >> > > > That let me to [2]. Is this related to this?
or is it just me
>>> >> doing
>>> >> >> > > things
>>> >> >> > > > wrong here?
>>> >> >> > > > Thanks for any pointers!
>>> >> >> > > >
>>> >> >> > > >
>>> >> >> > > > Renato M.
>>> >> >> > > >
>>> >> >> > > > 2014-05-20 22:25 GMT+02:00 Renato Marroquín
Mogrovejo <
>>> >> >> > > > renatoj.marroquin@gmail.com>:
>>> >> >> > > >
>>> >> >> > > > > Hey guys,
>>> >> >> > > > >
>>> >> >> > > > > This is a long overdue email but with little
steps I will
>>> start
>>> >> >> going
>>> >> >> > > :)
>>> >> >> > > > > So I have been trying to run the whole
project based on[1].
>>> As
>>> >> >> there
>>> >> >> > is
>>> >> >> > > > no
>>> >> >> > > > > version for any of the pieces involved
in running this (if
>>> there
>>> >> is
>>> >> >> > > > please
>>> >> >> > > > > point me to them, if not, I am happy to
put them on the site
>>> >> >> > > somewhere).
>>> >> >> > > > I
>>> >> >> > > > > chose
>>> >> >> > > > >
>>> >> >> > > > >
>>> >> >> > > > >
>>> >> >> > > > > [1] http://streams.incubator.apache.org/streams-usage.html
>>> >> >> > > > > [2]
>>> >> >> > > > >
>>> >> >> > > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>> >>
>>> http://stackoverflow.com/questions/7692497/tomcat-wont-load-my-meta-inf-services-javax-servlet-servletcontainerinitializ
>>> >> >> > > > >
>>> >> >> > > >
>>> >> >> > >
>>> >> >> >
>>> >> >>
>>> >>
>>>
>>
>>

Mime
View raw message