apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhupesh Chawda <bhup...@datatorrent.com>
Subject Re: Integration with Apache Samoa
Date Wed, 02 Mar 2016 04:44:59 GMT
Hi David,

Here is the working branch you can look at:
https://github.com/bhupeshchawda/incubator-samoa/tree/samoa-apex

As I mentioned, the launch part needs to be worked on. Currently I have a
few hacks in my local environment.
You can use the test cases though to get an idea.

-Bhupesh

On Wed, Mar 2, 2016 at 1:01 AM, David Yan <david@datatorrent.com> wrote:

> Hi Bhupesh,
>
> That's good progress.  Can you send us a link to the code you did for
> this?  Or maybe a review-only PR?
>
> David
>
> On Mon, Feb 29, 2016 at 10:15 PM, Bhupesh Chawda <bhupesh@datatorrent.com>
> wrote:
>
> > Hi All,
> >
> > Here is the status of integration of Apache Apex into Apache Samoa.
> >
> >    - Samoa API implemented and able to convert Samoa topology into Apex
> > Dag.
> >    - Implemented partitioning support using parallelism hints from Samoa
> >    API.
> >    - Implemented stream multiplexing:
> >       - Added All-based partitioner. Upstream tuples go to all downstream
> >       partitions
> >       - Stream codec for Key based partitioning
> >       - Stream codec for Random partitioning
> >    - Able to launch a Samoa task on the cluster. This has to be worked
> on.
> >    Currently some hacks are used by calling DTCli explicitly from the
> main
> >    entry point in Samoa code. Also jars are needed to be manually
> bundled.
> >    This will be worked on in this sprint.
> >    - Tested the following algorithms on local cluster:
> >       - Prequential Evaluation using Vertical Hoeffding Tree classifier.
> >       This is a decision tree based classifier.
> >       - Clustering using CluStream algorithm.
> >    - I have asked clarifications on some more details of these algorithms
> >    as well as serialization issues with Samoa classes. I am waiting for
> > some
> >    response from the Samoa community.
> >    - Although Samoa does not have many algorithms currently (it is a
> >    framework for developing algorithms), more algorithms are expected as
> a
> >    part of their roadmap:
> >    https://cwiki.apache.org/confluence/display/SAMOA/Roadmap
> >
> > Thanks,
> >
> > Bhupesh
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message