apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siyuan Hua <siy...@datatorrent.com>
Subject Re: Writing Streaming application in Scala.
Date Wed, 30 Sep 2015 04:11:33 GMT
Hi Tushar,

This is the jira https://malhar.atlassian.net/browse/APEX-175

Siyuan

On Tue, Sep 29, 2015 at 9:02 PM, Tushar Gosavi <tushar@datatorrent.com>
wrote:

> I can take up this work. Siyun let me know if you have already opened a
> Jira for this, else I will open one today.
>
> - Tushar.
>
> On Wed, Sep 30, 2015 at 2:08 AM, Chetan Narsude <chetan@datatorrent.com>
> wrote:
>
> > +1
> >
> > The current tutorial just tells the Scala Apex developer to walk. Scala
> > tuned API can make them run.
> >
> > Siyuan, why don't you open a JIRA issue to track it so someone interested
> > can take it?
> >
> > --
> > Chetan
> >
> > On Tue, Sep 29, 2015 at 1:27 PM, Siyuan Hua <siyuan@datatorrent.com>
> > wrote:
> >
> > > This is a good example. But the code here is still in Java style.
> > > I wish we could have another abstraction of scala api in the roadmap
> that
> > > could leverage the functional paradigm to write code in a concise way:
> > > for example:
> > > A simple statement like this
> > > stream.map(filter(tuple) -> {}, 5).reduce(aggregate(tuple, tuple)->{},
> 1)
> > > could spin up 5 filter operators and 1 aggregation operator
> > >
> > > Siyuan
> > >
> > >
> > > On Tue, Sep 29, 2015 at 7:40 AM, Munagala Ramanath <
> ram@datatorrent.com>
> > > wrote:
> > >
> > > > Languages like Scala, Clojure and dialects like Jython and JRuby are
> > > easier
> > > > to
> > > > support since they are JVM based and can interoperate with Java.
> > > >
> > > > Non JVM lanugages are a lot more work to support and also present
> > > > significant
> > > > performance issues because converting objects across the JVM boundary
> > is
> > > > often a heavy duty process.
> > > >
> > > > Ram
> > > >
> > > > On Mon, Sep 28, 2015 at 11:36 PM, Chinmay Kolhatkar <
> > > > chinmay@datatorrent.com
> > > > > wrote:
> > > >
> > > > > This is great. Would it also make sense to have similar using
> native
> > > > > language like C++ using JNI bindings.
> > > > >
> > > > > To my knowledge, complex enterprise based applications prefer
> > language
> > > > > closer to native for high performance and better memory
> utilization.
> > > > > If we could get writing a streaming application in C++ along with
> > power
> > > > of
> > > > > Apex, that would add a lot of value in my opinion.
> > > > >
> > > > > Any thoughts?
> > > > >
> > > > >
> > > > > ~ Chinmay.
> > > > >
> > > > > On Mon, Sep 28, 2015 at 9:06 PM, Munagala Ramanath <
> > > ram@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Tushar,
> > > > > >
> > > > > > This is a great project to add to dtBackyard at:
> > > > > >
> > > >
> > https://sites.google.com/a/datatorrent.com/dtbackyard/project-ownership
> > > > > >
> > > > > > Ram
> > > > > >
> > > > > > On Mon, Sep 28, 2015 at 7:29 AM, Tushar Gosavi <
> > > tushar@datatorrent.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I am trying to see how we can write a Streaming Application
> using
> > > > Scala
> > > > > > > language,
> > > > > > > as Scala targets JVM and has good interoperability with
Java.
> > > > > > >
> > > > > > > I am able to port simple pi demo in Scala along with the
> > operators.
> > > > > > Please
> > > > > > > take
> > > > > > > a look at PR#40 (
> > > > > https://github.com/apache/incubator-apex-malhar/pull/40
> > > > > > )
> > > > > > >  for this work, and provide feedback.
> > > > > > >
> > > > > > > Created following operators in Scala
> > > > > > > - RandomItemGenerator : Generate random points in 2d space.
> > > > > > > - PiCalculator                  : Calculates value of 
pi
> > > > > > > - ConsoleOutOperator    : Write value of tuple on stdout,
I
> could
> > > not
> > > > > use
> > > > > > > the Java operator available in Malhar, because scala does
more
> > > strong
> > > > > > type
> > > > > > > checking. (I need to look into this more)
> > > > > > >
> > > > > > > And use these operators to write a simple streaming
> application.
> > > > > > >
> > > > > > > The following functionality is working
> > > > > > > - Scala Operators recovers correctly from the crash.
> > > > > > > - Scala supports generating setters/getters using @BeanProperty
> > > > > > annotation,
> > > > > > > and these
> > > > > > >   properties become accessible through config file and
we can
> set
> > > > them
> > > > > > > dynamically through dtcli.
> > > > > > > - get-operator-properties is working.
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > -Tushar.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message