hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Khai Cher LIM (NYP)" <LIM_Khai_C...@nyp.edu.sg>
Subject Unable to perform terasort for 50GB of data
Date Fri, 08 Nov 2013 05:10:06 GMT
Dear all,

I have just started learning Hadoop setup and I am having problem with running terasort on
my Hadoop cluster. My input folder contains 50 GB of data but when I run the terasort, the
tasks failed and it gave me the error message as shown in the following screenshot.


I've set my dfs block size to be 128 MB. Actually with the default 64 MB, the tasks failed
also with the same reason.

Server information - HP ProLiant DL380p Gen8 (2U)
*             two Intel Xeon E5-2640 processors with 15 MB cache, 2.5Ghz, 7.2GT/s
*             48GB RAM
*             12 x 1TB (or a raw capacity of 12TB) 6G SAS 7.2K 3.5 HDD
*             RAID controller that supports RAID 5 with at least 512MB Flash-Backed Write
Cache (FBWC)
*             on-board adapter of 4 x 1GbE Ethernet port
*             2 hot-pluggable power supply units

I've configured two servers with virtual machines as decribed below:
Server 1:
1 Name Node - 32 GB RAM, 300 GB HDD space
4 Data Nodes - 16 GB RAM, 300 GB HDD space

Server 2:
1 Secondary Name Node - 32 GB RAM, 300 GB HDD space
4 Data Nodes - 16 GB RAM, 300 GB HDD space

I've checked that the diskspace used per data node is about 20% on average. Thus I couldn't
understand the error message complaining about "no space left on device".

Any help is much appreciated.

Thank you.

Khai Cher

  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message