hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajiv Chittajallu <raj...@yahoo-inc.com>
Subject Re: HOD and Condor
Date Tue, 29 Apr 2008 22:59:39 GMT
Hi David,

HOD is released as a contrrib in hadoop 0.16. The latest release is here 
http://www.apache.org/dist/hadoop/core/hadoop-0.16.3/hadoop-0.16.3.tar.gz

Currently HOD has supports only torque as the resource manager.

-Rajiv

On 04/30/08 03:32, David Riddle wrote:
> Hi,
> 
> I am trying to setup Hadoop on Demand on a cluster running Condor. I
> downloaded  hod-open-4.tar.gz from here:
> 
> http://issues.apache.org/jira/browse/HADOOP-1301
> 
> First of all I am not sure if this is the most recent version to use
> or if this the appropriate place to retrieve it. The website for
> Hadoop does not seem to have a download link for hod even though it
> contains documentation for hod.
> 
> The version I download seems to have some support for condor but it is
> not documented. I tryed to setup the hodrc as best I could but with
> not much sucesses. I have included a copy of my config file below if
> someone could help me to get hod working I would be grateful. When I
> run the following command:
> 
> ./hod -c ../conf/hodrc -m 3 -a "run dfs -ls /"
> 
> I recieve the following errors:
> 
> Failed to start hod shell: <type 'exceptions.AttributeError'>
> NodePoolDesc instance has no attribute 'getPkgDirs'
> 
> 
> Traceback (most recent call last):
>   File "hod", line 463, in <module>
>     exitStatus = main(hodConfig)
>   File "/home/ugrad/ddriddle/src/hod/hodlib/Hod/hod.py", line 120, in main
>     if np: np.finalize()
>   File "/home/ugrad/ddriddle/src/hod/hodlib/NodePools/condor.py", line
> 73, in finalize
>     self.freeNodeSet(ns)
>   File "/home/ugrad/ddriddle/src/hod/hodlib/NodePools/condor.py", line
> 64, in freeNodeSet
>     self._delNodeSet(nodeSet)
>   File "/home/ugrad/ddriddle/src/hod/hodlib/NodePools/condor.py", line
> 219, in _delNodeSet
>     pkgdirs = npd.getPkgDirs()
> AttributeError: NodePoolDesc instance has no attribute 'getPkgDirs'
> 
> 
> Thanks,
> 
> David Riddle
> 
> # Template hodrc file
> #
> 
> # -- hod shell specific configuration
> [hod]
> temp-dir             = /tmp/hod-ddriddle
> #java-home            = ${JAVA_HOME}
> java-home            = /home/ugrad/ddriddle/lib/java
> debug                = 3
> stream               = True
> non-interactive      = false
> allocate-timeout     = 3600
> #cluster              = ${CLUSTER_NAME}
> cluster              = hal9000
> #hadoop-install       = ${HADOOP_HOME}
> hadoop-install       = /home/ugrad/ddriddle/src/hadoop-0.15.3
> #log-dir              = /var/hod/logs-${RM_USER}
> #log-collection-dir   = /var/hod/logs-${RM_USER}
> #syslog-address       = localhost:8814
> #cluster-factor       = 1.8
> 
> # -- resource manager configuration
> [resource_manager]
> #id                  = ${RM_ID}
> id                  = condor
> #pbs-user            = ${RM_USER}
> pbs-user            = ddriddle
> pbs-server          = hal9000-server.cs.uiuc.edu
> queue               = parallel
> batch-home          = /home/hal/condor/condor
> env-vars            = HOD_PYTHON_HOME=/home/ugrad/ddriddle/bin
> 
> # -- serive_registry configuration
> [service_registry]
> debug                = 2
> http-address         = :8002
> xrs-address          = :8879
> #stream               = False
> #log-dir              = /var/hod/logs-${RM_USER}
> #xrs-port-range     = 32768-65536
> 
> # -- ringmaster configuration
> [ringmaster]
> temp-dir             = /tmp/hod-ddriddle
> work-dirs            =
> /tmp/hod-ddriddle/1,/tmp/hod-ddriddle/2,/tmp/hod-ddriddle/3,/tmp/hod-ddriddle/4
> register             = true
> xrs-port-range       = 32768-65536
> http-port-range      = 8000-9000
> debug                = 2
> #stream               = False
> #syslog-address       = localhost:8814
> #log-dir              = /var/hod/logs-${RM_USER}
> #svcrgy-addr          = :8879
> 
> # -- hod ring configuration
> [hodring]
> temp-dir             = /tmp/hod-ddriddle
> register             = true
> #java-home            = ${JAVA_HOME}
> java-home            = /home/ugrad/ddriddle/lib/java
> debug                = 2
> xrs-port-range       = 32768-65536
> http-port-range      = 8000-9000
> #log-dir              = /var/hod/logs-${RM_USER}
> #log-destination-uri  = local://var/log-${RM_USER}
> #stream               = false
> #syslog-address       = localhost:8814
> #svcrgy-addr          = :8879
> 
> [gridservice-mapred]
> external             = false
> tracker_port         = 8030
> info_port            = 50080
> #pkgs                 = ${HADOOP_HOME}
> pkgs                 = /home/ugrad/ddriddle/src/hadoop-0.15.3/
> #host                 = localhost
> 
> [gridservice-hdfs]
> external            = false
> fs_port             = 8020
> info_port           = 50070
> pkgs                 = /home/ugrad/ddriddle/src/hadoop-0.15.3/
> #pkgs                = ${HADOOP_HOME}
> #host                 = localhost
> 


Mime
View raw message