hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pat Ferrel <...@occamsmachete.com>
Subject Re: 3 machine cluster trouble
Date Fri, 25 May 2012 01:47:49 GMT
Oops, after a few trials I got an ERROR for incompatible builds 
versions. Copied code from the master, reformatted, et voila.

On 5/24/12 11:34 AM, Pat Ferrel wrote:
> ok, so all nodes are configured the same except for master/slave 
> differences. They are all running hdfs all daemons seem to be running 
> when I do a start-all.sh from the master. However the master 
> Map/Reduce Administration page shows only two live nodes. The HDFS 
> page shows 3.
>
> Looking at the log files on the new slave node I see no outright 
> errors but see this in the tasktracker log file. All machines have 8G 
> memory. I think the important part below is TaskTracker's 
> totalMemoryAllottedForTasks is -1. I've searched for others with this 
> problem but haven't found something for my case, which is just trying 
> to startup. No tasks have been run.
>
> 2012-05-24 11:20:46,786 INFO org.apache.hadoop.mapred.TaskTracker: 
> Starting tracker tracker_occam3:localhost/127.0.0.1:45700
> 2012-05-24 11:20:46,792 INFO org.apache.hadoop.mapred.TaskTracker: 
> Starting thread: Map-events fetcher for all reduce tasks on 
> tracker_occam3:localhost/127.0.0.1:45700
> 2012-05-24 11:20:46,792 INFO org.apache.hadoop.mapred.TaskTracker:  
> Using ResourceCalculatorPlugin : 
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5abd09e8
> 2012-05-24 11:20:46,795 WARN org.apache.hadoop.mapred.TaskTracker: 
> TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is 
> disabled.
> 2012-05-24 11:20:46,795 INFO org.apache.hadoop.mapred.IndexCache: 
> IndexCache created with max memory = 10485760
> 2012-05-24 11:20:46,800 INFO org.apache.hadoop.mapred.TaskTracker: 
> Shutting down: Map-events fetcher for all reduce tasks on 
> tracker_occam3:localhost/127.0.0.1:45700
> 2012-05-24 11:20:46,800 INFO 
> org.apache.hadoop.filecache.TrackerDistributedCacheManager: Cleanup...
> java.lang.InterruptedException: sleep interrupted
>     at java.lang.Thread.sleep(Native Method)
>     at 
> org.apache.hadoop.filecache.TrackerDistributedCacheManager$CleanupThread.run(TrackerDistributedCacheManager.java:926)
> 2012-05-24 11:20:46,900 INFO org.apache.hadoop.ipc.Server: Stopping 
> server on 45700
> 2012-05-24 11:20:46,901 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 3 on 45700: exiting
> 2012-05-24 11:20:46,901 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 1 on 45700: exiting
> 2012-05-24 11:20:46,902 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 2 on 45700: exiting
> 2012-05-24 11:20:46,902 INFO org.apache.hadoop.ipc.Server: Stopping 
> IPC Server listener on 45700
> 2012-05-24 11:20:46,901 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 0 on 45700: exiting
> 2012-05-24 11:20:46,904 INFO 
> org.apache.hadoop.ipc.metrics.RpcInstrumentation: shut down
> 2012-05-24 11:20:46,904 INFO org.apache.hadoop.mapred.TaskTracker: 
> Shutting down StatusHttpServer
> 2012-05-24 11:20:46,904 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 7 on 45700: exiting
> 2012-05-24 11:20:46,903 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 6 on 45700: exiting
> 2012-05-24 11:20:46,903 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 4 on 45700: exiting
> 2012-05-24 11:20:46,904 INFO org.apache.hadoop.ipc.Server: IPC Server 
> handler 5 on 45700: exiting
> 2012-05-24 11:20:46,904 INFO org.apache.hadoop.ipc.Server: Stopping 
> IPC Server Responder
> 2012-05-24 11:20:46,909 INFO org.mortbay.log: Stopped 
> SelectChannelConnector@0.0.0.0:50060
>
>
>
> On 5/23/12 3:55 PM, James Warren wrote:
>> Hi Pat -
>>
>> The setting for hadoop.tmp.dir is used both locally and on HDFS and
>> therefore should be consistent across your cluster.
>>
>> http://stackoverflow.com/questions/2354525/what-should-be-hadoop-tmp-dir
>>
>> cheers,
>> -James
>>
>> On Wed, May 23, 2012 at 3:44 PM, Pat Ferrel<pat@occamsmachete.com>  
>> wrote:
>>
>>> I have a two machine cluster and am adding a new machine. The new 
>>> node has
>>> a different location for hadoop.tmp.dir than the other two nodes and
>>> refuses to start the datanode when started in the cluster. When I 
>>> change
>>> the location pointed to by hadoop.tmp.dir to be the same on all 
>>> machines it
>>> starts up fine on all machines.
>>>
>>> Shouldn't I be able to have the master and slave1 set as:
>>> <property>
>>> <name>hadoop.tmp.dir</name>
>>> <value>/app/hadoop/tmp</value>
>>> <description>A base for other temporary directories.</description>
>>> </property>
>>>
>>> And slave2 set as:
>>> <property>
>>> <name>hadoop.tmp.dir</name>
>>> <value>/media/d2/app/hadoop/**tmp</value>
>>> <description>A base for other temporary directories.</description>
>>> </property>
>>>
>>> ??? Slave2 runs standalone in single node mode just fine. Using 
>>> 0.20.205.
>>>

Mime
View raw message