Mailing-List: contact dev-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mahout.apache.org
Received-SPF: pass (nike.apache.org: domain of pat.ferrel@gmail.com designates
 209.85.220.41 as permitted sender)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\))
Subject: Re: Straw poll re: H2O ?
From: Pat Ferrel <pat.ferrel@gmail.com>
In-Reply-To: <53626AD3.50804@gmail.com>
Date: Thu, 1 May 2014 09:01:43 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <C0509755-47B3-4A79-B39E-716127AE8C0D@gmail.com>
References: 
 <CAEccTyy7QtXU6ryod+2H5j6wFhVibXTsssPbjjoxy9r_a6PU2g@mail.gmail.com>
	<CAFboF2xZQ6wKM3ttR=FH1dBLq-h-=fkbOTRaULM7v=56RcbsGw@mail.gmail.com>
	<535E9970.7010907@apache.org>
	<CAPud8TrPMrFp9humPkGcU-tzG0rx+ZTO8NzqAcjziLhpz=sr_g@mail.gmail.com>
	<A93B3758-6D0C-4E0A-B092-903123C26EF6@occamsmachete.com>
	<BAY179-W561EE10461A2CF682FCC9DA8470@phx.gbl>
	<CAFboF2yb52s314nJ0BM+XswyDrbRXU42ZCaK1tzbrgV+hShbXQ@mail.gmail.com>
	<535FFC34.6050702@apache.org>	<53600050.5090502@apache.org>
	<CAO-+YE+W5shAMsty5LShqyP-Xc8_Qecbhg29D8BX4LaoFXQ30A@mail.gmail.com>
	<37069B8F-9388-4048-85E3-BA456EEB2DBF@gmail.com>
	<CAJwFCa3DBZ6xXUk4OOA=wdWFOa0WawnHn3_QjdbLqj7BCArRzQ@mail.gmail.com>
	<CAPud8Tq2mUnbYvLUzauftfo7pJajTKHFvb8rdrzScuRox69SWg@mail.gmail.com>
	<CAJwFCa3_jO1cpifmu2A_rcCZLSLcxYhwphNcd_EbYBkJHu9VTQ@mail.gmail.com>
 <CAPud8TpMh6gXd970HsF+z57S_hBRHgKBAOb1AgUrzkfRj=K3Rw@mail.gmail.com>
 <53626AD3.50804@gmail.com>
To: dev@mahout.apache.org

Odd that the Kmeans implementation isn=92t a way to demonstrate =
performance. Seems like anyone could grab that and try it with the same =
data on MLlib and perform a principled analysis. Or just run the same =
data through h2o and MLlib. This seems like a good way to look at the =
forrest instead of the trees.

BTW any generalization effort to support two execution engines will have =
to abstract away the SparkContext. This is where IO, job control, and =
engine tuning happens. Abstracting the DSL is not sufficient. Any =
hypothetical MahoutContext (a good idea for sure) if it deviated =
significantly from a SparkContext will have broad impact.

=
http://spark.apache.org/docs/0.9.1/api/core/index.html#org.apache.spark.Sp=
arkContext


On May 1, 2014, at 8:40 AM, Cliff Click <cclick0@gmail.com> wrote:

H2O will launch an internal Task in the single-digit microsecond range.  =
Because of this, we can launch 100,000's (millions?) a second... leading =
to fine-grained data parallelism, and high CPU utilization.  This is a =
big piece of our single-node speed.  Some other distributed =
Task-launching solutions I've seen tend to require a network-hop =
per-task... leading to your 10ms to launch as task requirement, leading =
to a limit of a few 1000 Tasks/sec requiring tasks that are much larger =
and coarser than H2O's... leading to much lower CPU utilization.

Also, I'm getting 200micro-second ping's between my datacenter =
machines.... down from 10msec.  It's decent commodity hardware, nothing =
special.  Meaning: H2O can launch task on an entire 32-node cluster in =
about 1msec, starting from a single driving node (log-tree fanout, depth =
5, 200micro-second single UDP packet launch, 1micro-second internal task =
launch).

And this latency matters when the work itself is lots and lots "small" =
jobs, as is common when a DSL such as Mahout or Spark/Scala or R is =
driving simple operators over bulk data.

Cliff


On 4/30/2014 3:35 PM, Dmitriy Lyubimov wrote:
> This is kind of an old news. They all do, for years now. I've been =
building a system that does real time distributed pipelines (~30 ms to =
start all steps in pipeline + in-core complexity) for years. Note that =
node-to-node hop in clouds are usually mean at about 10ms so =
microseconds are kind of out of question for network performance reasons =
in real life except for private racks. The only thing that doesn't do =
this is the MR variety of Hadoop.=20