hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hungsheng Tsao, Ph. D phone" <Hungsheng.t...@oracle.com>
Subject Re: Hadoop and SGE
Date Wed, 30 Jun 2010 12:05:10 GMT
Plwasw chwck thw docs die  oge6.2u 5
  or 6
There is integration of oge and hadoop

------- Original message -------
> From: Dmitry Pushkarev <umka@stanford.edu>
> To: common-user@hadoop.apache.org
> Sent: 30.6.'10,  6:59
> Dear Hadoop users,
> I'm in the process of building a new cluster for our lab and I'm trying 
> to
> run SGE simultaneously with hadoop. Idea is that each node would function 
> as
> datanode at all times, but depending on situation and a fraction of nodes
> will run SGE instead of plain. SGE jobs will not have access to HDFS or
> local filesystem (except for /tmp) and will run out of external NAS, they
> aren't supposed to be IO bound.
> I'm trying to figure out of what's the best way to setup this resource
> sharing. One way would be to shutdown tasktrackers on reserved nodes and 
> add
> them to SGE pool. Another way is run tasktrackers as SGE jobs and each
> tasktracker would shut down after some idle time.
> Has anyone tried something like this? I'd appreciate any advice.
> Thanks.

View raw message