heron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Kanjilal <sxk1...@hotmail.com>
Subject Re: Hello heron community
Date Tue, 31 Oct 2017 21:54:14 GMT
Here's an initial PR showing the scala traits that I got as a result of running the javatoscala
converter on the streamlets API, please take a look and let me know if you have any feedback,
next steps include putting these on the design doc and then starting work on the implementations.
 To reiterate the work was done against a child branch in my fork of heron.


https://github.com/skanjila/heron/pull/1


Thanks


________________________________
From: Saikat Kanjilal <sxk1969@hotmail.com>
Sent: Monday, October 30, 2017 9:01 PM
To: dev@heron.incubator.apache.org
Subject: Re: Hello heron community


Thanks Ashwin and yes I am aware of this from reading the documentation.  I have actually
written a little component in reef that runs a simple reef job using spark running on yarn,
will be curious to look at this code when I have some more time.


________________________________
From: aas.ojj@gmail.com <aas.ojj@gmail.com> on behalf of Ashvin A <aasojj@gmail.com>
Sent: Monday, October 30, 2017 11:36 AM
To: dev@heron.incubator.apache.org
Subject: Re: Hello heron community

Hi Saikat,

Welcome !

Since you are familiar with REEF, you might want to know that the YARN port
of Heron is based on REEF. You can find some details here:
https://twitter.github.io/heron/docs/operators/deployment/schedulers/yarn/.
Heron Documentation - Apache Hadoop YARN Cluster ...<https://twitter.github.io/heron/docs/operators/deployment/schedulers/yarn/>
twitter.github.io
In addition to out-of-the-box schedulers for Aurora, Heron can also be deployed on a YARN
cluster with the YARN scheduler. The YARN scheduler is implemented using the ...




-ashvin

On Sun, Oct 29, 2017 at 10:19 PM, Karthik Ramasamy <kramasamy@gmail.com>
wrote:

> Saikat -
>
> Welcome to the Heron Community. It will be great to see support for Scale
> API. Current API are of two forms -
>
> - procedural/object oriented api (this is very low level where you have to
> write the DAG operators and assemble them) - somewhat similar to Storm
> - functional (this is more higher level that use functions like map,
> flatMap, etc)
>
> Here are few suggestions and recommendations.
>
> - Join the slack for any quick questions and answers (which you have
> already done)
>
> - We use Github to track issues. In your case, it might be a easier to
> track using projects - for your convenience
> I have created a project called Scala API for Heron -
> https://github.com/twitter/heron/projects/3 <https://github.com/twitter/
[https://avatars1.githubusercontent.com/u/50278?s=400&v=4]<https://github.com/twitter/heron/projects/3>

twitter/heron<https://github.com/twitter/heron/projects/3>
github.com
heron - Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter



> heron/projects/3>
>
> - It will be great to have a design doc, adding this as a first task into
> backlog. Once the design doc is complete, you can post it for review and
> comments
> for the community. Google doc is an easy way to share and get feedback
> unless there is other preference.
>
> - Once review is complete, you can create additional tasks and issues and
> link them to the project.
>
> Hope this helps to jump start.
>
> cheers
> /karthik
>
> > On Oct 29, 2017, at 6:17 PM, Saikat Kanjilal <sxk1969@hotmail.com>
> wrote:
> >
> > Hello Folks,
> >
> > I'm interested in taking and driving the following github component
> (this means design/development/architecture and more).
> >
> >
> > https://github.com/twitter/heron/issues/668
> >
> >
> > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> github.com/twitter/heron/issues/668>
> >
> > Ability to write native scala topologies ยท Issue #668 ...<
> https://github.com/twitter/heron/issues/668>
> > github.com
> > heron - Heron is a realtime, distributed, fault-tolerant stream
> processing engine from Twitter
> >
> >
> >
> >
> > I've joined the heron slack channel as recommended and am ready to start
> by creating a master JIRA issue to track this if not already done, I wanted
> some guidance from a committer or two to help guide me a bit take care of
> at he administrative details and how to get this kicked off the ground.  My
> background includes working on apache reef codebase as well as dabbling in
> apache mahout and apache flume as well as storm and various nosql/graph
> databases.  Looking forward to being part of the community.
> >
> >
> >
> > The rough plan of action in my mind so you know:
> >
> > 1) Review heron architecture and codebase in detail
> >
> > 2) Create an umbrella JIRA with the first series of tasks
> >
> > 3) Come up with a design doc
> >
> > 4) Build the interfaces
> >
> > 5) Add more dev tasks to JIRA
> >
> > 6) Drive the implementation
> >
> >
> > Thanks in advance for your help.
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message