hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Baldeschwieler <eri...@yahoo-inc.com>
Subject Re: Do people put their master node in the slave list - 0.15.1
Date Wed, 26 Dec 2007 20:38:50 GMT
We run these on separate nodes.  We saturate a 16G RAM node just  
running the name node, but we have a lot of files and clients.

On Dec 26, 2007, at 12:22 PM, Ted Dunning wrote:

>
> My namenode and jobtracker are both on a machine that is a datanode  
> and has
> a tasktracker as well.  It is also less well outfitted than yours.
>
> I have no problems, but my data is encrypted which might make the  
> CPU/disk
> trade-offs very different.
>
>
> On 12/26/07 12:11 PM, "Jason Venner" <jason@attributor.com> wrote:
>
>> This seems to be more a function of input file size than anything.  
>> I had
>> a single (uncompressed) 35gig input file Text,Value.
>>
>> Jason Venner wrote:
>>> I have been experimenting with that, and when I do, the master
>>> saturates well before the slave nodes, and the jobs start  
>>> experiencing
>>> timeouts
>>>
>>> The map task in question is the IdentityMapper, this job is a simple
>>> merge sort, combining data by key where there are duplicate keys in
>>> the input stream.
>>> There is no swapping going on in my cluster, and the machines in
>>> question are all 8 processor boxes, and the tasks.maximum was set  
>>> to 6.
>>>
>>> task_200712261033_0002_m_000078_0: Exception in thread "main"
>>> java.net.SocketTimeoutException: timed out waiting for rpc response
>>> task_200712261033_0002_m_000078_0:      at
>>> org.apache.hadoop.ipc.Client.call(Client.java:484)
>>> task_200712261033_0002_m_000078_0:      at
>>> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
>>> task_200712261033_0002_m_000078_0:      at
>>> org.apache.hadoop.mapred.$Proxy0.getTask(Unknown Source)
>>> task_200712261033_0002_m_000078_0:      at
>>> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java: 
>>> 1747)
>>> 07/12/26 10:48:03 INFO mapred.JobClient: Task Id :
>>> task_200712261033_0002_m_000081_1, Status : FAILED
>>>
>


Mime
View raw message