hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From patf <p...@well.com>
Subject 1.0.0 Windows vs Linux
Date Mon, 02 Jan 2012 22:04:11 GMT

I'd gotten 0.22.0 to work on Windows(-cygwin) but then had all kinds of 
problems particularly (mostly) with hdfs file system permissions.

Was able to get non-daemon standalone working when I replaced the 
setPermission() at the end of RawLocalFileSystem.java with the 
corresponding code from 0.22.0 which, as best I can tell (the older 
code) calls out to chmod instead of trying to use Java native routines.  
No number of mods however succeeded in getting daemons working in 
standalone mode.  The problems were permissions and mostly it seemed 
with how the cyg_server username under which cygwin sshd runs and hdfs.

I take it that 1.0.0 is not yet fully supported for Windows?

So I wanted to see what a 'full' (standalone with daemons) looks like 
for 1.0.0 and installed Linux 1.0.0.1 into a vmware machine running CentOS.

There were a number of problems there too all of which I solved and only 
one of which I'll ask about.  Some but not all of the 5 processes 
(namenode, datanode, secondarynamenode, jobtracker and tasktracker) 
would start successfully after my first install of hadoop 1.0.0.1.  At 
some point I noticed at the bottom of start-dfs.sh and start-mapred.sh 
that some jobs start under hadoop-daemon.sh (singular) and other under 
hadoop-daemons.sh (plural).  I  determined that daemons plural is for a 
slave node and simply changed all these to daemon singular.

All 5 jobs then ran and I was able to successfully execute the validate 
routines (teragen, terasort and teravalidate).  As well as grep, pi, 
etc.  Seems to be a working installation - maybe now I can move on to 
using it for more interesting stuff however it running in a vmware vm 
and I can't imagine performance will be very good (which is much of the 
point).

So what,  in more detail, did I do by changing daemon*s* (slave) to 
daemon?  Did I take sshd out of the equation?  I do have sshd installed 
and running.

Pat


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message