hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yu-yang chen <yy...@doc.ic.ac.uk>
Subject Re: Configuration and Hadoop cluster setup
Date Fri, 25 May 2007 22:59:39 GMT
I think you are not include your nodes A, B, C in your bin/slaves file, 
that may be why.
your hadoop-site.xml seems ok to me

yu-yang

Phantom wrote:
> Here is a copy of my hadoop-site.xml. What am I doing wrong ?
>
> <configuration>
>        <property>
>                <name>fs.default.name</name>
>                <value>dev030.sctm.com:9000</value>
>        </property>
>
>        <property>
>                <name>dfs.name.dir</name>
>                <value>/tmp/hadoop</value>
>        </property>
>
>        <property>
>                <name>mapred.job.tracker</name>
>                <value>dev030.sctm.com:50029</value>
>        </property>
>
>        <property>
>                <name>mapred.job.tracker.info.port</name>
>                <value>50030</value>
>        </property>
>
>        <property>
>                <name>mapred.min.split.size</name>
>                <value>65536</value>
>        </property>
>
>        <property>
>                <name>dfs.replication</name>
>                <value>1</value>
>        </property>
>
> </configuration>
>
>
> On 5/25/07, Vishal Shah <vishals@rediff.co.in> wrote:
>>
>> Hi Avinash,
>>
>>   Can you share your hadoop-site.xml, mapred-default.xml and slaves 
>> files?
>> Most probably, you have not set the jobtracker properly in the
>> hadoop-site.xml conf file. Check the property mapred.job.tracker 
>> property
>> in
>> your file. It should look something like this:
>>
>> <property>
>>   <name>mapred.job.tracker</name>
>>   <value>fully.qualified.domainname:40000</value>
>>   <description>The host and port that the MapReduce job tracker runs
>>   at.  If "local", then jobs are run in-process as a single map
>>   and reduce task.
>>   </description>
>> </property>
>>
>> -vishal.
>>
>> -----Original Message-----
>> From: Mahadev Konar [mailto:mahadev@yahoo-inc.com]
>> Sent: Friday, May 25, 2007 5:54 AM
>> To: hadoop-user@lucene.apache.org
>> Subject: RE: Configuration and Hadoop cluster setup
>>
>> Hi,
>>   When you run the job, you need to set the environment variable
>> HADOOP_CONF_DIR to the configuration directory that has the 
>> configuration
>> file pointing to the right jobtracker.
>>
>> Regards
>> Mahadev
>>
>> > -----Original Message-----
>> > From: Phantom [mailto:ghostwhoowalks@gmail.com]
>> > Sent: Thursday, May 24, 2007 4:51 PM
>> > To: hadoop-user@lucene.apache.org
>> > Subject: Re: Configuration and Hadoop cluster setup
>> >
>> > Yes the files are the same and I am starting the tasks on the namenode
>> > server. I also figured what my problem was with respect to not being
>> able
>> > to
>> > start the namenode and job tracker on the same machine. I had to
>> reformat
>> > the file system. But the all this still doesn't cause the WordCount
>> sample
>> > to run in a distributed fashion. I can tell this because the
>> > LocalJobRunner
>> > is being used. Do I need to specify the config file to the running
>> > instance
>> > of the program ? If so how do I do that ?
>> >
>> > Thanks
>> > A
>> >
>> > On 5/24/07, Dennis Kubes <nutch-dev@dragonflymc.com> wrote:
>> > >
>> > >
>> > >
>> > > Phantom wrote:
>> > > > I am trying to run Hadoop on a cluster of 3 nodes. The namenode 
>> and
>> > the
>> > > > jobtracker web UI work. I have the namenode running on node A and
>> job
>> > > > tracker running on node B. Is it true that namenode and jobtracker
>> > > cannot
>> > > > run on the same box ?
>> > >
>> > > The namenode and the jobtracker can most definitely run on the same
>> box.
>> > >   As far as I know this is the preferred configuration.
>> > >
>> > > Also if I want to run the examples on the cluster is
>> > > > there anything special that needs to be done. When I run the 
>> example
>> > > > WordCount on a machine C (which is a task tracker and not a job
>> > tracker)
>> > > > the
>> > > > LocalJobRunner is invoked all the time. I am guessing this means
>> that
>> > > the
>> > > > map tasks are running locally. How can I distribute this on the
>> > cluster
>> > > ?
>> > > > Please advice.
>> > >
>> > > Are the conf files on machine C the same as the namenode/jobtracker?
>> > > Are they pointing to the namenode and jobtracker or are they 
>> pointing
>> to
>> > > local in the hadoop-site.xml file.  Also we have found it easier
>> > > (although not necessarily better) to start tasks on the namenode
>> server.
>> > >
>> > > It would be helpful to have more information about what is happening
>> and
>> > > your setup as that would help myself and others on the list debug 
>> what
>> > > may be occurring.
>> > >
>> > > Dennis Kubes
>> > >
>> > > >
>> > > > Thanks
>> > > > Avinash
>> > > >
>> > >
>>
>>
>


Mime
View raw message