mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy Lyubimov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAHOUT-1489) Interactive Scala & Spark Bindings Shell & Script processor
Date Fri, 11 Apr 2014 01:03:55 GMT

    [ https://issues.apache.org/jira/browse/MAHOUT-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966097#comment-13966097
] 

Dmitriy Lyubimov commented on MAHOUT-1489:
------------------------------------------

Ok. this is way too complicated for me. I did a quick hack \[1] which seems to work at least
in local mode . Works like a charm. Here is the session dump (need just to start o.a.m.sparkbindings.shell.Main
class from idea). I also filtered out most of spark debug messages that are enabled by default:

{code:title="Mahout Spark Shell session"}
14/04/10 17:52:27 INFO spark.HttpServer: Starting HTTP Server
14/04/10 17:52:27 INFO server.Server: jetty-7.6.8.v20121106
14/04/10 17:52:27 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:60204
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 0.9.0
      /_/

Using Scala version 2.10.3 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51)
Type in expressions to have them evaluated.
Type :help for more information.
Created spark context..
Spark context available as sc.

scala> val a = dense((1,2,3),(3,4,5))
a: org.apache.mahout.math.DenseMatrix = 
{
  0  =>	{0:1.0,1:2.0,2:3.0}
  1  =>	{0:3.0,1:4.0,2:5.0}
}

scala> val drmA = drmParallelize(a)
drmA: org.apache.mahout.sparkbindings.drm.CheckpointedDrm[Int] = org.apache.mahout.sparkbindings.drm.CheckpointedDrmBase@791b95a1

scala> val drmAtA = drmA.t %*% drmA
drmAtA: org.apache.mahout.sparkbindings.drm.DrmLike[Int] = OpAB(OpAt(org.apache.mahout.sparkbindings.drm.CheckpointedDrmBase@791b95a1),org.apache.mahout.sparkbindings.drm.CheckpointedDrmBase@791b95a1)

scala> drmAtA.collect
res0: org.apache.mahout.math.Matrix = 
{
  0  =>	{0:10.0,1:14.0,2:18.0}
  1  =>	{0:14.0,1:20.0,2:26.0}
  2  =>	{0:18.0,1:26.0,2:34.0}
}

scala> 
{code}

I suggest you to fork my branch, take it as a basis. It basically works, now the stuff that
needs to happen is to verify it in distributed mode and modify mahout shell script to set
up proper paths etc. to launch it w.r.t "mahout shell <master>" command. Should be simple
an uneventful enough now. 

     \[1\]: https://github.com/dlyubimov/mahout-commits/tree/shell



> Interactive Scala & Spark Bindings Shell & Script processor
> -----------------------------------------------------------
>
>                 Key: MAHOUT-1489
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-1489
>             Project: Mahout
>          Issue Type: New Feature
>    Affects Versions: 1.0
>            Reporter: Saikat Kanjilal
>            Assignee: Dmitriy Lyubimov
>             Fix For: 1.0
>
>
> Build an interactive shell /scripting (just like spark shell). Something very similar
in R interactive/script runner mode.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message