Mailing-List: contact dev-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mahout.apache.org
Received-SPF: pass (nike.apache.org: domain of ssc.open@googlemail.com
 designates 74.125.83.41 as permitted sender)
Message-ID: <53600050.5090502@apache.org>
Date: Tue, 29 Apr 2014 21:41:04 +0200
From: Sebastian Schelter <ssc@apache.org>
Reply-To: ssc@apache.org, ssc@apache.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:24.0) Gecko/20100101 Thunderbird/24.4.0
MIME-Version: 1.0
To: dev@mahout.apache.org
Subject: Re: Straw poll re: H2O ?
References: 
 <CAEccTyy7QtXU6ryod+2H5j6wFhVibXTsssPbjjoxy9r_a6PU2g@mail.gmail.com>
	<CAFboF2xZQ6wKM3ttR=FH1dBLq-h-=fkbOTRaULM7v=56RcbsGw@mail.gmail.com>
	<535E9970.7010907@apache.org>
	<CAPud8TrPMrFp9humPkGcU-tzG0rx+ZTO8NzqAcjziLhpz=sr_g@mail.gmail.com>
	<A93B3758-6D0C-4E0A-B092-903123C26EF6@occamsmachete.com>
	<BAY179-W561EE10461A2CF682FCC9DA8470@phx.gbl>
 <CAFboF2yb52s314nJ0BM+XswyDrbRXU42ZCaK1tzbrgV+hShbXQ@mail.gmail.com>
 <535FFC34.6050702@apache.org>
In-Reply-To: <535FFC34.6050702@apache.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

For reasons of transparency in this discussion, I should add that I am a 
committer on the upcoming Stratosphere ASF podling, co-worker of the 
main developers and have contributed to it as part of my PhD.

On 04/29/2014 09:23 PM, Sebastian Schelter wrote:
> Anand,
>
> I'm trying to answer some of your questions, and my answers highlight
> the points that I would like to see clarified about h20.
>
> On 04/28/2014 11:13 PM, Anand Avati wrote:
>
>> 1. Why is the DSL claiming to have (in its vision) logical vs physical
>> separation if not for providing multiple compute backends?
>
> This is not a claim or a vision, the DSL already has this separation.
> Take for example o.a.m.sparkbindings.drm.plan.OpAtA, thats the logical
> operator for executing a Transpose-Times-Self matrix multiplication. In
> o.a.m.sparkbindings.blas.AtA you will find two physical operator
> implementations for that. The choice which one to use depends on whether
> there is enough memory to hold certain intermediary results in memory.
>
> The primary intention of a separation into logical and physical
> operators is to allow for a declarative programming style on the users
> side and for an optimizer on the system side which automatically chooses
> the optimal physical operator for the execution of a specific program.
>
> This choice of the physical operator might depend on the shape and
> amount of the data processed as well on the underlying available
> resources. *The separation into logical and physical operators clearly
> doesn't imply to have multiple backends*. It only makes it very easy to
> support them.
>
>>
>> 2. Does the proposal of having a new DSL backend in the future (for e.g
>> stratosphere as suggested elsewhere) make you:
>
>> -- worry that stratosphere would be a dependency to Mahout?
>
> Stratosphere has been accepted as a incubator project in the ASF
> recently, so the worry about such a dependency is naturally less than
> about an externally managed project like h20.
>
>> -- worry that as a user/commiter/contributor you have to worry about a
>> new
>> framework?
>
> In my eyes, there is a big difference between Spark/Stratosphere and
> h20. Spark and Stratosphere have a clearly defined programming and
> execution model. They execute programs that are composed of a DAG of
> operators. The set of operators has clearly defined semantics and
> parallelization strategies. If you compare their operators, you will
> find that they offer pretty much the same in lightly different flavors.
> For both, there are scientific papers that in detail explain all these
> things.
>
> I have asked about a detailed description of h20's programming model and
> execution model and I searched the documentation, but I haven't been
> able to find something that clearly describes how things are done. I
> would love to read up on this, but until I'm presented with this, I have
> to assume that such a principled foundation is missing.
>
>
> --sebastian
>