apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Yan <da...@datatorrent.com>
Subject Re: Integration with Apache Samoa
Date Mon, 07 Nov 2016 19:09:34 GMT
It took perseverance to get this merged, Good work Bhupesh!

On Mon, Nov 7, 2016 at 1:25 AM, Bhupesh Chawda <bhupesh@datatorrent.com>
wrote:

> Hi All,
>
> The PR for making Apex a runner for SAMOA has been merged.
>
> Apache SAMOA now has an additional runner with Apache Apex -
> https://github.com/apache/incubator-samoa/tree/master/samoa-apex
>
> Thanks.
>
> ~ Bhupesh
>
> On Mon, Mar 28, 2016 at 11:05 AM, Bhupesh Chawda <bhupesh@datatorrent.com>
> wrote:
>
> > Hi All,
> >
> > Here is the status of the integration, since the last update:
> >
> >    - Launch process cleaned up. Launch still happening through DTCli
> >    calls. Once APEXCORE-405
> >    <https://issues.apache.org/jira/browse/APEXCORE-405> is implemented,
> >    this will become a lot better.
> >    - Some issues still causing problems in running Samoa apps on Apex.
> >    Containers getting killed. This may be due to low memory. This is
> work in
> >    progress.
> >
> > ~Bhupesh
> >
> > On Wed, Mar 2, 2016 at 10:14 AM, Bhupesh Chawda <bhupesh@datatorrent.com
> >
> > wrote:
> >
> >> Hi David,
> >>
> >> Here is the working branch you can look at:
> >> https://github.com/bhupeshchawda/incubator-samoa/tree/samoa-apex
> >>
> >> As I mentioned, the launch part needs to be worked on. Currently I have
> a
> >> few hacks in my local environment.
> >> You can use the test cases though to get an idea.
> >>
> >> -Bhupesh
> >>
> >> On Wed, Mar 2, 2016 at 1:01 AM, David Yan <david@datatorrent.com>
> wrote:
> >>
> >>> Hi Bhupesh,
> >>>
> >>> That's good progress.  Can you send us a link to the code you did for
> >>> this?  Or maybe a review-only PR?
> >>>
> >>> David
> >>>
> >>> On Mon, Feb 29, 2016 at 10:15 PM, Bhupesh Chawda <
> >>> bhupesh@datatorrent.com>
> >>> wrote:
> >>>
> >>> > Hi All,
> >>> >
> >>> > Here is the status of integration of Apache Apex into Apache Samoa.
> >>> >
> >>> >    - Samoa API implemented and able to convert Samoa topology into
> Apex
> >>> > Dag.
> >>> >    - Implemented partitioning support using parallelism hints from
> >>> Samoa
> >>> >    API.
> >>> >    - Implemented stream multiplexing:
> >>> >       - Added All-based partitioner. Upstream tuples go to all
> >>> downstream
> >>> >       partitions
> >>> >       - Stream codec for Key based partitioning
> >>> >       - Stream codec for Random partitioning
> >>> >    - Able to launch a Samoa task on the cluster. This has to be
> worked
> >>> on.
> >>> >    Currently some hacks are used by calling DTCli explicitly from the
> >>> main
> >>> >    entry point in Samoa code. Also jars are needed to be manually
> >>> bundled.
> >>> >    This will be worked on in this sprint.
> >>> >    - Tested the following algorithms on local cluster:
> >>> >       - Prequential Evaluation using Vertical Hoeffding Tree
> >>> classifier.
> >>> >       This is a decision tree based classifier.
> >>> >       - Clustering using CluStream algorithm.
> >>> >    - I have asked clarifications on some more details of these
> >>> algorithms
> >>> >    as well as serialization issues with Samoa classes. I am waiting
> for
> >>> > some
> >>> >    response from the Samoa community.
> >>> >    - Although Samoa does not have many algorithms currently (it is
a
> >>> >    framework for developing algorithms), more algorithms are expected
> >>> as a
> >>> >    part of their roadmap:
> >>> >    https://cwiki.apache.org/confluence/display/SAMOA/Roadmap
> >>> >
> >>> > Thanks,
> >>> >
> >>> > Bhupesh
> >>> >
> >>>
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message