apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Yan <da...@datatorrent.com>
Subject Re: Integration with Apache Samoa
Date Tue, 01 Mar 2016 19:31:50 GMT
Hi Bhupesh,

That's good progress.  Can you send us a link to the code you did for
this?  Or maybe a review-only PR?


On Mon, Feb 29, 2016 at 10:15 PM, Bhupesh Chawda <bhupesh@datatorrent.com>

> Hi All,
> Here is the status of integration of Apache Apex into Apache Samoa.
>    - Samoa API implemented and able to convert Samoa topology into Apex
> Dag.
>    - Implemented partitioning support using parallelism hints from Samoa
>    API.
>    - Implemented stream multiplexing:
>       - Added All-based partitioner. Upstream tuples go to all downstream
>       partitions
>       - Stream codec for Key based partitioning
>       - Stream codec for Random partitioning
>    - Able to launch a Samoa task on the cluster. This has to be worked on.
>    Currently some hacks are used by calling DTCli explicitly from the main
>    entry point in Samoa code. Also jars are needed to be manually bundled.
>    This will be worked on in this sprint.
>    - Tested the following algorithms on local cluster:
>       - Prequential Evaluation using Vertical Hoeffding Tree classifier.
>       This is a decision tree based classifier.
>       - Clustering using CluStream algorithm.
>    - I have asked clarifications on some more details of these algorithms
>    as well as serialization issues with Samoa classes. I am waiting for
> some
>    response from the Samoa community.
>    - Although Samoa does not have many algorithms currently (it is a
>    framework for developing algorithms), more algorithms are expected as a
>    part of their roadmap:
>    https://cwiki.apache.org/confluence/display/SAMOA/Roadmap
> Thanks,
> Bhupesh

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message