hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Venner <ja...@attributor.com>
Subject Re: More HOD: is there anyway to get HOD to copy back all of the log files to the submit node?
Date Thu, 28 Feb 2008 05:21:08 GMT
You are my hero of the day Hemanth, these 3 things will make our life so 
much simpler.

Not knowing torque at all and trying to deal with two systems that are 
essentially unknown has been making the learning curve quite steep.

Hemanth Yamijala wrote:
> Jason Venner wrote:
>> I have found that HOD writes a series of log files to directories on 
>> the virtual cluster master, if you specify log directories.
>> The interesting part is figuring out which machine was the virtual 
>> cluster master, if you have a decent sized pool of machines.
> Can you explain what you mean by 'virtual cluster master' ? I guess 
> you could mean the node which ran the 'ringmaster' process - the 
> master process in a hod virtual cluster, or the hadoop 'jobtracker'. 
> Both this information is stored by hod for an allocated cluster. You 
> could retrieve it by using
> hod -o list,
> and
> hod -o "info cluster-dir"
> If the cluster is deallocated, you can get the ringmaster node still 
> by using torque commands:
> qstat (to get torque job ids)
> and
> qstat -n <jobid>
> This will print a list of nodes, the first one is the ringmaster.
>> It would be so nice if HOD could copy this back to the submission node.
>> I suppose we could configure syslog up but then we have to configure 
>> the syslogd on each of the submit hosts to accept from any compute 
>> node...
> Syslog configuration has a bug which will be addressed in Hadoop 0.16.1.
> Thanks
> hemanth
Jason Venner
Attributor - Publish with Confidence <http://www.attributor.com/>
Attributor is hiring Hadoop Wranglers, contact if interested

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message