hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rick Hangartner <hangart...@strands.com>
Subject Practical limits on number of blocks per datanode.
Date Fri, 21 Nov 2008 06:22:49 GMT


We are in the midst of considering Hadoop as a prototype solution for  
a system we are building.  In the abstract Hadoop and MapReduce are  
very well-suited to our computational problem.  However, this email  
exchange has caused us some concern that we are hoping the user  
community might allay.  We've searched JIRA for relevant issues but  
didn't turn up anything. (We probably aren't as adept as we might be  
at surfacing appropriate items though.)

Here are the relevant numbers for the data we are using to prototype a  
system using Hadoop 0.18.1:

We have 16,000,000 files that are 10K each, or about 160GB total.  We  
have 10 datanodes with the default replication factor of 3.   Each  
file will probably be stored as a single block, right?  This means we  
would be storing 48,000,000 blocks on 10 datanodes or 4,800,000 blocks  
per node.

At 160GB, the total data is not particularly large.  Unfortunately,  
the attached email exchange suggests  we could have a problem with the  
large number of blocks per node.  We have considered combining a  
number of small files into larger files (say concatenating  sets of  
100 files into single larger files so we have 48,000 blocks that are  
1MB in size per node.)  This would not significantly effect our  
MapReduce algorithm, but it could undesirably complicate other  
components of the system that use this data.

Thanks in advance for any insights on the match between Hadoop (0.18.x  
and later) and our particular system requirements.


Begin forwarded message:

> From: Konstantin Shvachko <shv@yahoo-inc.com>
> Date: November 17, 2008 6:27:42 PM PST
> To: core-user@hadoop.apache.org
> Subject: Re: The Case of a Long Running Hadoop System
> Reply-To: core-user@hadoop.apache.org
> Bagri,
> According to the numbers you posted your cluster has 6,000,000 block  
> replicas
> and only 12 data-nodes. The blocks are small on average about 78KB  
> according
> to fsck. So each node contains about 40GB worth of block data.
> But the number of blocks is really huge 500,000 per node. Is my math  
> correct?
> I haven't seen data-nodes that big yet.
> The problem here is that a data-node keeps a map of all its blocks  
> in memory.
> The map is a HashMap. With 500,000 entries you can get long lookup  
> times I guess.
> And also block reports can take long time.
> So I believe restarting name-node will not help you.
> You should somehow pack your small files into larger ones.
> Alternatively, you can increase your cluster size, probably 5 to 10  
> times larger.
> I don't remember whether we had any optimization patches related to  
> data-nodes
> block map since 0.15. Please advise if anybody remembers.
> Thanks,
> --Konstantin
> Abhijit Bagri wrote:
>> We do not have a secondary namenode because 0.15.3 has serious bug  
>> which truncates the namenode image if there is a failure while  
>> namenode fetches image from secondary namenode. See HADOOP-3069
>> I have a patched version of 0.15.3 for this issue. From the patch  
>> of HADOOP-3069, the changes are on namenode _and_ secondary  
>> namenode, which means I just cant fire up a seconday namenode.
>> - Bagri
>> On Nov 15, 2008, at 11:36 PM, Billy Pearson wrote:
>>> If I understand the secondary namenode merges the edits log in to  
>>> the fsimage and reduces the edit log size.
>>> Which is likely the root of your problems 8.5G seams large and  
>>> likely putting a strain on your master servers memory and io  
>>> bandwidth
>>> Why do you not have a secondary namenode?
>>> If you do not have the memory on the master I would look in to  
>>> stopping a datanode/tasktracker on a server and loading the  
>>> secondary namenode on it
>>> Let it run for a while and watch your log for the secondary  
>>> namenode you should see your edit log get smaller
>>> I am not an expert but that would be my first action.
>>> Billy

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message