heron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sijie Guo <guosi...@gmail.com>
Subject Re: Hello heron community
Date Wed, 01 Nov 2017 00:27:59 GMT
I have a few thoughts about the scala API and made a comment in your pull
request, purely from a scala user perspective. It would be good if the
scala api can have those scala syntactic sugar.

The comments are:

- function: it would be good to support scala function, so a scala user can
use lambda expression. also it would be good to support java function in
scala api as well. because if there are functions rewritten in java
function, they can be reused in a scala heron app.
- key/value: it would be good to support tuple *(K, V)*. it makes
transformation code much clean in a scala app when transforming between a
streamlet and a kv streamlet.
- package name: I saw the package name is same as java api. it would be
good to distinguish them.

- Sijie



On Tue, Oct 31, 2017 at 2:54 PM, Saikat Kanjilal <sxk1969@hotmail.com>
wrote:

> Here's an initial PR showing the scala traits that I got as a result of
> running the javatoscala converter on the streamlets API, please take a look
> and let me know if you have any feedback, next steps include putting these
> on the design doc and then starting work on the implementations.  To
> reiterate the work was done against a child branch in my fork of heron.
>
>
> https://github.com/skanjila/heron/pull/1
>
>
> Thanks
>
>
> ________________________________
> From: Saikat Kanjilal <sxk1969@hotmail.com>
> Sent: Monday, October 30, 2017 9:01 PM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
>
> Thanks Ashwin and yes I am aware of this from reading the documentation.
> I have actually written a little component in reef that runs a simple reef
> job using spark running on yarn, will be curious to look at this code when
> I have some more time.
>
>
> ________________________________
> From: aas.ojj@gmail.com <aas.ojj@gmail.com> on behalf of Ashvin A <
> aasojj@gmail.com>
> Sent: Monday, October 30, 2017 11:36 AM
> To: dev@heron.incubator.apache.org
> Subject: Re: Hello heron community
>
> Hi Saikat,
>
> Welcome !
>
> Since you are familiar with REEF, you might want to know that the YARN port
> of Heron is based on REEF. You can find some details here:
> https://twitter.github.io/heron/docs/operators/deployment/schedulers/yarn/
> .
> Heron Documentation - Apache Hadoop YARN Cluster ...<
> https://twitter.github.io/heron/docs/operators/deployment/schedulers/yarn/
> >
> twitter.github.io
> In addition to out-of-the-box schedulers for Aurora, Heron can also be
> deployed on a YARN cluster with the YARN scheduler. The YARN scheduler is
> implemented using the ...
>
>
>
>
> -ashvin
>
> On Sun, Oct 29, 2017 at 10:19 PM, Karthik Ramasamy <kramasamy@gmail.com>
> wrote:
>
> > Saikat -
> >
> > Welcome to the Heron Community. It will be great to see support for Scale
> > API. Current API are of two forms -
> >
> > - procedural/object oriented api (this is very low level where you have
> to
> > write the DAG operators and assemble them) - somewhat similar to Storm
> > - functional (this is more higher level that use functions like map,
> > flatMap, etc)
> >
> > Here are few suggestions and recommendations.
> >
> > - Join the slack for any quick questions and answers (which you have
> > already done)
> >
> > - We use Github to track issues. In your case, it might be a easier to
> > track using projects - for your convenience
> > I have created a project called Scala API for Heron -
> > https://github.com/twitter/heron/projects/3 <https://github.com/twitter/
> [https://avatars1.githubusercontent.com/u/50278?s=400&v=4]<
> https://github.com/twitter/heron/projects/3>
>
> twitter/heron<https://github.com/twitter/heron/projects/3>
> github.com
> heron - Heron is a realtime, distributed, fault-tolerant stream processing
> engine from Twitter
>
>
>
> > heron/projects/3>
> >
> > - It will be great to have a design doc, adding this as a first task into
> > backlog. Once the design doc is complete, you can post it for review and
> > comments
> > for the community. Google doc is an easy way to share and get feedback
> > unless there is other preference.
> >
> > - Once review is complete, you can create additional tasks and issues and
> > link them to the project.
> >
> > Hope this helps to jump start.
> >
> > cheers
> > /karthik
> >
> > > On Oct 29, 2017, at 6:17 PM, Saikat Kanjilal <sxk1969@hotmail.com>
> > wrote:
> > >
> > > Hello Folks,
> > >
> > > I'm interested in taking and driving the following github component
> > (this means design/development/architecture and more).
> > >
> > >
> > > https://github.com/twitter/heron/issues/668
> > >
> > >
> > > [https://avatars1.githubusercontent.com/u/367684?v=4&s=400]<https://
> > github.com/twitter/heron/issues/668>
> > >
> > > Ability to write native scala topologies ยท Issue #668 ...<
> > https://github.com/twitter/heron/issues/668>
> > > github.com
> > > heron - Heron is a realtime, distributed, fault-tolerant stream
> > processing engine from Twitter
> > >
> > >
> > >
> > >
> > > I've joined the heron slack channel as recommended and am ready to
> start
> > by creating a master JIRA issue to track this if not already done, I
> wanted
> > some guidance from a committer or two to help guide me a bit take care of
> > at he administrative details and how to get this kicked off the ground.
> My
> > background includes working on apache reef codebase as well as dabbling
> in
> > apache mahout and apache flume as well as storm and various nosql/graph
> > databases.  Looking forward to being part of the community.
> > >
> > >
> > >
> > > The rough plan of action in my mind so you know:
> > >
> > > 1) Review heron architecture and codebase in detail
> > >
> > > 2) Create an umbrella JIRA with the first series of tasks
> > >
> > > 3) Come up with a design doc
> > >
> > > 4) Build the interfaces
> > >
> > > 5) Add more dev tasks to JIRA
> > >
> > > 6) Drive the implementation
> > >
> > >
> > > Thanks in advance for your help.
> > >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message