mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Conwell <j...@iamjohn.me>
Subject Re: Cancel running distributed RecommenderJob
Date Wed, 04 Apr 2012 13:00:17 GMT
This type of job control needs to be tightly integrated into mahouts core,
where all long running mahout jobs have a way to id and kill the hadoop
jobs it controls.   Without this it'll be hard to seamlessly integrate
mahout into production systems

Sent via the iTuringTest
On Apr 4, 2012 5:13 AM, "Sean Owen" <srowen@gmail.com> wrote:

> Yes something like that would work.
>
> On Wed, Apr 4, 2012 at 2:00 PM, Sören Brunk <soren.brunk@deri.org> wrote:
> > But i'd have to find out that id manually by looking in the web interface
> > right?
> > I think it will be difficult to find the right hadoop job, because
> multiple
> > RecommenderJobs can run at the same time in my system.
> >
> > So, just in theory, a way to make this work from java would mean to add a
> > method to RecommenderJob that calls killJob() on the currently running
> > hadoop job. For mahout jobs used in RecommenderJob (e.g.
> RowSimilarityJob)
> > the same had to be done so RecommenderJob could also cancel mahout sub
> > tasks.
> >
> > Do you think it could work that way?
> >
> >
> >
> > On 02/04/12 19:05, Sean Owen wrote:
> >>
> >> You can use the Hadoop interface itself (like, the command-line hadoop
> >> tool) to kill a job by its ID. If you kill one MapReduce job the
> >> entire process should halt after that.
> >>
> >> On Mon, Apr 2, 2012 at 6:44 PM, Sören Brunk<soren.brunk@deri.org>
>  wrote:
> >>>
> >>> Hi,
> >>>
> >>> I'm using the distributed RecommenderJob from within a Java program.
> >>> For that, in a separate thread, I'm creating a RecommenderJob object,
> >>> call
> >>> setConf() for the hadoop configuration and then run() with the job
> >>> parameters.
> >>> This is working fine for me but now I would like to be able to stop a
> >>> running job.
> >>> Not sure if that's possible at all since RecommenderJob encapsulates
> >>> several
> >>> Hadoop jobs (or even other Mahout jobs that call Hadoop in turn) and
> runs
> >>> them in a blocking way.
> >>>
> >>> Would be interesting for other Mahout jobs as well.
> >>> Any ideas?
> >>>
> >>> Thanks,
> >>>
> >>> --
> >>> Sören Brunk
> >>> Research Assistant
> >>> Data Intensive Infrastructures Unit (DI2)
> >>> Digital Enterprise Research Institute
> >>> National University of Ireland Galway
> >>>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message