streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sblackmon <sblack...@apache.org>
Subject Re: Getting started / tutorial help
Date Sat, 15 Oct 2016 20:13:10 GMT
All,

Please take a look at this improvement to the examples portion of the website and share any
thoughts.
http://streams.staging.apache.org/site/0.4-incubating-SNAPSHOT/streams-examples/index.html

In contrast to with the current page:
http://streams.incubator.apache.org/site/0.4-incubating-SNAPSHOT/streams-examples/

Major differences are:
- More examples: now 4 flink and 6 local
- Each example module now has one integration test for each pipelines, covering all steps
from providers to persisters
- Added a new left sidebar, with a contextual collection of instructions that are relevant
to running each example
- HOW-TO’s for launching supporting services for testing with docker maven plugin
- A separate page for each pipeline, even within modules that only have one stream
- A HOCON paradigm using config snippets and includes for supplying configuration required
by each pipeline
- Snippets for running pipelines using SBT and Docker (neither of which require building the
code first)

This section still needs some clean-up and styling, and could still be way better, but maybe
this takes us from 2/10 to 5/10 ?

Are the rest of you able to follow page flow? Is the information supplied adequate and adequately
organized for you to successfully run the pipeline with mvn verify, and without?

Steve

On October 5, 2016 at 5:17:32 PM, sblackmon (sblackmon@apache.org) wrote:

Trevor, others,

I’ve opened a few pull requests containing a Flink example module with integration tests
demonstrating five pipelines working end-to-end in local mode.

https://github.com/apache/incubator-streams/pull/289

https://github.com/apache/incubator-streams-examples/pull/12

If you have a chance to code review and replicate this, that would be terrific.  I believe
this demonstrates clearly that Streams is suitable for embedding within real-time parallel
processing frameworks.

As we discussed in Berlin there are a lot of cool possible demos based on feeding social streams
into Zeppelin workbooks and/or Flink complex event processing libraries.

Steve

On September 30, 2016 at 9:20:09 AM, Steve Blackmon (steve@blackmon.org) wrote:

Trevor,

I don’t see sample.json in the code tree:

https://github.com/apache/incubator-streams-examples/tree/flink/flink/flink-twitter-collection/src/main/jsonschema

Did you happen create that file while trying to run the job or copy it there inadvertently?

The jsonschema2pojo plugin will parse any .json files in that directory and it will fail if
they aren’t proper json schemas.

Please keep me posted on your progress.

Cheers,
Steve
On September 30, 2016 at 9:00:53 AM, Trevor Grant (trevor.d.grant@gmail.com) wrote:

Hey all,

just trying to get started on Apache Streams.

I was trying to build the examples (on the Flink branch), when I
-DskipTests I can get a build but when I do mvn clean package I get the
following errors:

skip non existing resourceDirectory
/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/main/xmlschema
[INFO] skip non existing resourceDirectory
/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/main/resources
[INFO] skip non existing resourceDirectory
/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/site/diagrams
[INFO]
[INFO] --- jsonschema2pojo-maven-plugin:0.4.1:generate (default) @
flink-twitter-collection ---
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] streams-examples-flink ............................ SUCCESS [1.752s]
[INFO] flink-twitter-collection .......................... FAILURE [4.383s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 7.455s
[INFO] Finished at: Fri Sep 30 07:58:14 GMT-06:00 2016
[INFO] Final Memory: 30M/430M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.jsonschema2pojo:jsonschema2pojo-maven-plugin:0.4.1:generate (default)
on project flink-twitter-collection: Execution default of goal
org.jsonschema2pojo:jsonschema2pojo-maven-plugin:0.4.1:generate failed:
Error parsing document:
file:/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/main/jsonschema/sample.json:
Unexpected end-of-input within/between OBJECT entries
[ERROR] at [Source:
file:/home/rawkintrevo/gits/incubator-streams-examples/flink/flink-twitter-collection/src/main/jsonschema/sample.json;
line: 69, column: 2279]


Should I be worried?

tg

Trevor Grant
Data Scientist
https://github.com/rawkintrevo
http://stackexchange.com/users/3002022/rawkintrevo
http://trevorgrant.org

*"Fortunate is he, who is able to know the causes of things." -Virgil*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message