heron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Kanjilal <sxk1...@hotmail.com>
Subject Re: Hello heron community
Date Mon, 30 Oct 2017 21:26:50 GMT
I'm conflicted a bit on 1, here's why I feel like most of our users would want to use the spouts/bolts
API who are coming over from Storm or using heron for the first time, however the Streamlets
interface seems like it has the right level of abstraction, I think I will start with the
Streamlet interface and add in the spout/bolt API as needed.   One other thought I had was
to figure out an integration plan with spark, to this end the fit between heron and spark
would be in using spark streaming where the stream endpoint would be coming from heron, not
sure if that should be part of this effort or not, what do you guys think?


________________________________
From: Sanjeev Kulkarni <sanjeevrk@gmail.com>
Sent: Monday, October 30, 2017 2:17 PM
To: dev@heron.incubator.apache.org
Subject: Re: Hello heron community

I have a couple of questions
1. Do you plan on exposing both the low level(spouts/bolts) and the
streamlet api? Or are you preferring one over another?
2. My suggestion would be to start with the Streamlet interface. Primarily
because a) Most of scala ml libraries that I've seen tend to operate on the
streamlet kind of interface and b) It has a far smaller surface area(i.e.
number of interfaces) so might be easy to get something up quickly for
testing.
Thanks!

On Mon, Oct 30, 2017 at 2:13 PM, Saikat Kanjilal <sxk1969@hotmail.com>
wrote:

> Thanks Sanjeev,  I had an initial idea that I wanted to float on the list,
> I was thinking that as part of the initial scala port of the API I'd like
> to propose that we use this tool: http://javatoscala.com/
Java to Scala converter<http://javatoscala.com/>
javatoscala.com
How does it work? Java to Scala converter is created with Play framework and Scalagen library.
I don't want you to see my code. No problem. Java to Scala converter ...



>
> Java to Scala converter<http://javatoscala.com/>
Java to Scala converter<http://javatoscala.com/>
javatoscala.com
How does it work? Java to Scala converter is created with Play framework and Scalagen library.
I don't want you to see my code. No problem. Java to Scala converter ...



> javatoscala.com
> How does it work? Java to Scala converter is created with Play framework
> and Scalagen library. I don't want you to see my code. No problem. Java to
> Scala converter ...
>
>
> I propose that we use the above tool to do an initial conversion of all
> the Java functional and oo interfaces and then plug that into the initial
> design doc as well as the codebase to get things off the ground.   I have
> already forked the heron repo and will be doing this off of a child branch
> based on my fork.
>
>
> How does that sound to folks, anyone has strong objections, of course I
> fully realize that I will still need to do a lot of work around refactoring
> the code even when using the converter so I'm prepared for that, the goal
> in using the converters is to save a bit of time so that manual
> intervention isnt necessarily needed in designing every interface.
>
> Thoughts.
>
> ________________________________
> From: Sanjeev Kulkarni <sanjeevrk@gmail.com>
> Sent: Sunday, October 29, 2017 10:18 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> Hi Saikat,
> Welcome to the Heron users group. Your plan of action sounds good to me. A
> native scala interface would be great for all those scala fans. Eagerly
> awaiting design doc.
> Thanks!
>
> On Sun, Oct 29, 2017 at 6:17 PM, Saikat Kanjilal <sxk1969@hotmail.com>
> wrote:
>
> > Hello Folks,
> >
> > I'm interested in taking and driving the following github component (this
> > means design/development/architecture and more).
> >
> >
> > https://github.com/twitter/heron/issues/668
> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>

Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.



> github.com
> Support for Scala Instance and APIs.
>
>
>
> >
> >
> > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> > github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies · Issue #668 ...<
> > https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>

Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.



> [https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://
> github.com/twitter/heron/issues/668>
>
> Ability to write native scala topologies · Issue #668 · twitter/heron<
> https://github.com/twitter/heron/issues/668>
[https://avatars1.githubusercontent.com/u/367684?s=400&v=4]<https://github.com/twitter/heron/issues/668>

Ability to write native scala topologies · Issue #668 · twitter/heron<https://github.com/twitter/heron/issues/668>
github.com
Support for Scala Instance and APIs.



> github.com
> Support for Scala Instance and APIs.
>
>
>
> > github.com
> > heron - Heron is a realtime, distributed, fault-tolerant stream
> processing
> > engine from Twitter
> >
> >
> >
> >
> > I've joined the heron slack channel as recommended and am ready to start
> > by creating a master JIRA issue to track this if not already done, I
> wanted
> > some guidance from a committer or two to help guide me a bit take care of
> > at he administrative details and how to get this kicked off the ground.
> My
> > background includes working on apache reef codebase as well as dabbling
> in
> > apache mahout and apache flume as well as storm and various nosql/graph
> > databases.  Looking forward to being part of the community.
> >
> >
> >
> > The rough plan of action in my mind so you know:
> >
> > 1) Review heron architecture and codebase in detail
> >
> > 2) Create an umbrella JIRA with the first series of tasks
> >
> > 3) Come up with a design doc
> >
> > 4) Build the interfaces
> >
> > 5) Add more dev tasks to JIRA
> >
> > 6) Drive the implementation
> >
> >
> > Thanks in advance for your help.
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message