mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Kanjilal <sxk1...@hotmail.com>
Subject RE: [jira] [Comment Edited] (MAHOUT-1489) Interactive Scala & Spark Bindings Shell & Script processor
Date Mon, 31 Mar 2014 15:04:04 GMT
I put a question/discussion item for Dmitry and I to resolve on this, I would like to push
for having a Mahout interpreter, the shell should be used for executing spark related commands
in the context of Mahout.  However to get there we first need to bring in the necessary spark
and scala dependencies and get the code compiling as a first step before making the leap to
a mahout interpreter.

> Date: Mon, 31 Mar 2014 11:00:11 -0400
> Subject: Re: [jira] [Comment Edited] (MAHOUT-1489) Interactive Scala & Spark Bindings
Shell & Script processor
> From: smarty.juice@gmail.com
> To: dev@mahout.apache.org
> 
> Hi Saikat,
> 
> main purpose of HTTPSever.scala is to execute code from interactive shell
> that also means and send JARs to worker node and added to SparkContext
> 
> I was thinking if we get Mahout running on Spark, should we redirect shell
> commands to spark interpreter or have our own Mahout interpreter, do we
> want interactive shell to for mahout-only?
> 
> Thanks,
> Hardik
> 
> 
> On Sun, Mar 30, 2014 at 6:06 PM, Saikat Kanjilal (JIRA) <jira@apache.org>wrote:
> 
> >
> >     [
> > https://issues.apache.org/jira/browse/MAHOUT-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954848#comment-13954848]
> >
> > Saikat Kanjilal edited comment on MAHOUT-1489 at 3/30/14 10:04 PM:
> > -------------------------------------------------------------------
> >
> > Ok more progress, so here's what I've done so far and need a bit of
> > brainstorming/guidanc, have checked in all changes to git hub repo
> > specified above, although code still not building yet, regardless I've:
> > 1) merged code from spark onto the shell package under
> > org.apache.mahout.sparkbindings
> > 2) replaced all the classes under number 1 to have the correct package
> > structure
> > 3) added two new maven dependencies around the scala compiler and jlang
> > both using 2.10.0 to resolve some errors associated with scala.tools and
> > scala.***.jline etc
> >
> > Now I noticed that the spark code brings in an HTTPServer which is
> > associated with running jetty locally, is this something we want inside our
> > spark shell, if not I'll need to refactor a bunch of code to remove these
> > dependencies and understand deeper what or how to replace this, the errors
> > we are getting are associated with this
> >
> > Dmitry would love some guiodance/discussion on how to proceed around the
> > HTTPServer
> >
> > Thanks in advance.
> >
> >
> > was (Author: kanjilal):
> > Ok more progress, so here's what I've done so far and need a bit of
> > brainstorming/guidanc, have checked in all changes to git hub repo,
> > although code still not building yet, regardless I've:
> > 1) merged code from spark onto the shell package under
> > org.apache.mahout.sparkbindings
> > 2) replaced all the classes under number 1 to have the correct package
> > structure
> > 3) added two new maven dependencies around the scala compiler and jlang
> > both using 2.10.0 to resolve some errors associated with scala.tools and
> > scala.***.jline etc
> >
> > Now I noticed that the spark code brings in an HTTPServer which is
> > associated with running jetty locally, is this something we want inside our
> > spark shell, if not I'll need to refactor a bunch of code to remove these
> > dependencies and understand deeper what or how to replace this, the errors
> > we are getting are associated with this
> >
> > Dmitry would love some guiodance/discussion on how to proceed around the
> > HTTPServer
> >
> > Thanks in advance.
> >
> > > Interactive Scala & Spark Bindings Shell & Script processor
> > > -----------------------------------------------------------
> > >
> > >                 Key: MAHOUT-1489
> > >                 URL: https://issues.apache.org/jira/browse/MAHOUT-1489
> > >             Project: Mahout
> > >          Issue Type: New Feature
> > >    Affects Versions: 1.0
> > >            Reporter: Saikat Kanjilal
> > >            Assignee: Dmitriy Lyubimov
> > >             Fix For: 1.0
> > >
> > >
> > > Build an interactive shell /scripting (just like spark shell). Something
> > very similar in R interactive/script runner mode.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.2#6252)
> >
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message