hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Khai Cher LIM (NYP)" <LIM_Khai_C...@nyp.edu.sg>
Subject Unable to perform terasort for 50GB of data
Date Fri, 08 Nov 2013 05:10:06 GMT
Dear all,

I have just started learning Hadoop setup and I am having problem with running terasort on
my Hadoop cluster. My input folder contains 50 GB of data but when I run the terasort, the
tasks failed and it gave me the error message as shown in the following screenshot.

[cid:image001.png@01CEDC83.D789EC90]

I've set my dfs block size to be 128 MB. Actually with the default 64 MB, the tasks failed
also with the same reason.

Server information - HP ProLiant DL380p Gen8 (2U)
*             two Intel Xeon E5-2640 processors with 15 MB cache, 2.5Ghz, 7.2GT/s
*             48GB RAM
*             12 x 1TB (or a raw capacity of 12TB) 6G SAS 7.2K 3.5 HDD
*             RAID controller that supports RAID 5 with at least 512MB Flash-Backed Write
Cache (FBWC)
*             on-board adapter of 4 x 1GbE Ethernet port
*             2 hot-pluggable power supply units

I've configured two servers with virtual machines as decribed below:
Server 1:
1 Name Node - 32 GB RAM, 300 GB HDD space
4 Data Nodes - 16 GB RAM, 300 GB HDD space

Server 2:
1 Secondary Name Node - 32 GB RAM, 300 GB HDD space
4 Data Nodes - 16 GB RAM, 300 GB HDD space

I've checked that the diskspace used per data node is about 20% on average. Thus I couldn't
understand the error message complaining about "no space left on device".

Any help is much appreciated.

Thank you.

Regards,
Khai Cher


Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message