hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johan Oskarsson <jo...@oskarsson.nu>
Subject Re: Practical limits on number of blocks per datanode.
Date Fri, 21 Nov 2008 10:37:48 GMT
Hi Rick,

unfortunately 4,800,000 blocks per node is going to be too much. Ideally
you'd want to merge your files into as few as possible, even 1MB per
file is quite small for Hadoop. Would it be possible to merge them into
hundreds of mbs or preferably gigabyte files?

In newer Hadoop versions there is an archive feature that can put many
files into an archive for you. This can then be processed transparently
by Hadoop. I haven't used that though so can't tell if it's worth the

I ran into issues with too many blocks per datanode before and it's not
fun, they start losing contact with the namenode with all kinds of
interesting side effects.


Rick Hangartner wrote:
> Hi,
> We are in the midst of considering Hadoop as a prototype solution for a
> system we are building.  In the abstract Hadoop and MapReduce are very
> well-suited to our computational problem.  However, this email exchange
> has caused us some concern that we are hoping the user community might
> allay.  We've searched JIRA for relevant issues but didn't turn up
> anything. (We probably aren't as adept as we might be at surfacing
> appropriate items though.)
> Here are the relevant numbers for the data we are using to prototype a
> system using Hadoop 0.18.1:
> We have 16,000,000 files that are 10K each, or about 160GB total.  We
> have 10 datanodes with the default replication factor of 3.   Each file
> will probably be stored as a single block, right?  This means we would
> be storing 48,000,000 blocks on 10 datanodes or 4,800,000 blocks per node.
> At 160GB, the total data is not particularly large.  Unfortunately, the
> attached email exchange suggests  we could have a problem with the large
> number of blocks per node.  We have considered combining a number of
> small files into larger files (say concatenating  sets of 100 files into
> single larger files so we have 48,000 blocks that are 1MB in size per
> node.)  This would not significantly effect our MapReduce algorithm, but
> it could undesirably complicate other components of the system that use
> this data.
> Thanks in advance for any insights on the match between Hadoop (0.18.x
> and later) and our particular system requirements.
> Begin forwarded message:
>> From: Konstantin Shvachko <shv@yahoo-inc.com>
>> Date: November 17, 2008 6:27:42 PM PST
>> To: core-user@hadoop.apache.org
>> Subject: Re: The Case of a Long Running Hadoop System
>> Reply-To: core-user@hadoop.apache.org
>> Bagri,
>> According to the numbers you posted your cluster has 6,000,000 block
>> replicas
>> and only 12 data-nodes. The blocks are small on average about 78KB
>> according
>> to fsck. So each node contains about 40GB worth of block data.
>> But the number of blocks is really huge 500,000 per node. Is my math
>> correct?
>> I haven't seen data-nodes that big yet.
>> The problem here is that a data-node keeps a map of all its blocks in
>> memory.
>> The map is a HashMap. With 500,000 entries you can get long lookup
>> times I guess.
>> And also block reports can take long time.
>> So I believe restarting name-node will not help you.
>> You should somehow pack your small files into larger ones.
>> Alternatively, you can increase your cluster size, probably 5 to 10
>> times larger.
>> I don't remember whether we had any optimization patches related to
>> data-nodes
>> block map since 0.15. Please advise if anybody remembers.
>> Thanks,
>> --Konstantin
>> Abhijit Bagri wrote:
>>> We do not have a secondary namenode because 0.15.3 has serious bug
>>> which truncates the namenode image if there is a failure while
>>> namenode fetches image from secondary namenode. See HADOOP-3069
>>> I have a patched version of 0.15.3 for this issue. From the patch of
>>> HADOOP-3069, the changes are on namenode _and_ secondary namenode,
>>> which means I just cant fire up a seconday namenode.
>>> - Bagri
>>> On Nov 15, 2008, at 11:36 PM, Billy Pearson wrote:
>>>> If I understand the secondary namenode merges the edits log in to
>>>> the fsimage and reduces the edit log size.
>>>> Which is likely the root of your problems 8.5G seams large and
>>>> likely putting a strain on your master servers memory and io bandwidth
>>>> Why do you not have a secondary namenode?
>>>> If you do not have the memory on the master I would look in to
>>>> stopping a datanode/tasktracker on a server and loading the
>>>> secondary namenode on it
>>>> Let it run for a while and watch your log for the secondary namenode
>>>> you should see your edit log get smaller
>>>> I am not an expert but that would be my first action.
>>>> Billy

View raw message