hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Veentjer <alarmnum...@gmail.com>
Subject Re: Scheduling map/reduce jobs
Date Wed, 05 Jan 2011 19:55:14 GMT
Hi Tylen,

I'm not worried about the scheduling since there are tons of mechanisms
available for that. But what I want to prevent is that every node in the
cluster starts to schedule the same map/reduce jobs. Only one machine is
sufficient and when that machine fails, another one should take over.

On Wed, Jan 5, 2011 at 7:31 PM, Tyler Coffin <tcoffin@rim.com> wrote:

> Cron works great and is probably already on your systems.
> -----Original Message-----
> From: Peter Veentjer [mailto:alarmnummer@gmail.com]
> Sent: January 5, 2011 12:35
> To: user@hbase.apache.org
> Subject: Scheduling map/reduce jobs
> He Guys,
> although it isn't completely related to HBase. Is there support for
> scheduling map reduce jobs?
> E.g. I want to do a map reduce job that automatically removes certain
> elements from hbase and perhaps some additional cleanup (I know that there
> is support for lease times).
> I could have every node in the cluster schedule this job every hour, but if
> there are n nodes I don't want this job to be running n times. Only 1 time
> would be sufficient. I can create some checking logic for scheduling a map
> reduce job where a job that should not be run is rejected/ignored, but if
> there is something out of the box.
> Another question:
> We are building an environment where client provided plugins can be
> deployed
> and obey some kind of SLA (e.g. we want it to be running on 2 nodes at
> least). So the old plugin needs to be undeployed, the jars of the new
> plugin
> need to be copied on demand to the nodes and the new plugin needs to be
> deployed. Is there functionality for this behavior out of the box or does
> it
> need to be implemented by ourselves?
> ---------------------------------------------------------------------
> This transmission (including any attachments) may contain confidential
> information, privileged material (including material protected by the
> solicitor-client or other applicable privileges), or constitute non-public
> information. Any use of this information by anyone other than the intended
> recipient is prohibited. If you have received this transmission in error,
> please immediately reply to the sender and delete this information from your
> system. Use, dissemination, distribution, or reproduction of this
> transmission by unintended recipients is not authorized and may be unlawful.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message