hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Baldeschwieler <eri...@yahoo-inc.com>
Subject Re: [jira] Commented: (HADOOP-141) Disk thrashing / task timeouts during map output copy phase
Date Fri, 21 Apr 2006 00:50:56 GMT
humm,

The client is timing out when it is getting data?  Maybe as long as  
it is getting data, it should reset its timer?  Maybe the server  
should fail a client if it is busy?  This would let you make informed  
decision.

On Apr 20, 2006, at 11:24 AM, paul sutter (JIRA) wrote:

>     [ http://issues.apache.org/jira/browse/HADOOP-141? 
> page=comments#action_12375411 ]
>
> paul sutter commented on HADOOP-141:
> ------------------------------------
>
>
> A few timeouts would be fine. The problem is when the same files  
> timeout over and over again, and progress ceases completely.
>
> I was able to make the problem go away by increasing the number of  
> mappers by 6X, making the map output files 1/6th as large, so I  
> have given up on finding the problem.
>
> So here is the summary:
>
> - with 700MB map output files (18 mappers), original code: the job  
> would never progress past reduce progress of  17% or 18%.
> - with 700MB map output files (18 mappers), large buffers: the job  
> completed in 27 hours
> - with 120MB map output files (106 mappers), and large buffers: the  
> job completed in 6 hours
>
> Im happy to share logs that include the timeouts and extended  
> logging information on MapOutputFile.java if anyone is interested,  
> but i wont post them here because they are several hundred megabytes.
>
> Otherwise I will continue to use the workaround of smaller map  
> output files.
>
>> Disk thrashing / task timeouts during map output copy phase
>> -----------------------------------------------------------
>>
>>          Key: HADOOP-141
>>          URL: http://issues.apache.org/jira/browse/HADOOP-141
>>      Project: Hadoop
>>         Type: Bug
>
>>   Components: mapred
>>  Environment: linux
>>     Reporter: paul sutter
>
>>
>> MapOutputProtocol connections cause timeouts because of system  
>> thrashing and transferring the same file over and over again,  
>> ultimately leading to making no forward progress(medium sized job,  
>> 500GB input file, map output about as large as the input, 10 node  
>> cluster).
>> There are several bugs behind this, but the following two changes  
>> improved matters considerably.
>> (1)
>> The buffersize in MapOutputFile is currently hardcoded to 8192  
>> bytes (for both reads and writes). By changing this buffer size to  
>> 256KB, the number of disk seeks are reduced and the problem went  
>> away.
>> Ideally there would be a buffer size parameter for this that is  
>> separate from the DFS io buffer size.
>> (2)
>> I also added the following code to the socket configuration in  
>> both Server.java and Client.java. No linger is a minor good idea  
>> in an enivronment with some packet loss (and you will have that  
>> when all the nodes get busy at once), but 256KB buffers is  
>> probably excessive, especially on a LAN, but it takes me two hours  
>> to test changes so I havent experimented.
>> socket.setSendBufferSize(256*1024);
>> socket.setReceiveBufferSize(256*1024);
>> socket.setSoLinger(false, 0);
>> socket.setKeepAlive(true);
>
> -- 
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the  
> administrators:
>    http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:
>    http://www.atlassian.com/software/jira
>


Mime
View raw message