hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Richard Zhang" <richardtec...@gmail.com>
Subject Is this a design setting:hadoop can not do concurrent writing on different data nodes in the same cluster?
Date Thu, 03 Jul 2008 16:56:46 GMT
Hi Hadoop folks:
I have a 8 nodes cluster and use copyFromLocal to write some text workload
into dfs. I tried to write it on different nodes in the same cluster. That
is ,I log on to each machine in the same cluster after hadoop boots. In each
machine, I use: bin/hadoop dfs -copyFromLocal workload.txt. I was expecting
in this way, each node's local disk will be reduce because its contributed
disk space for dfs is consumed.
However, after trying a few times, I found that only the disk space of
namenode is consumed whether or not copyFromLocal is doing from namenode
machine or data node machines. Thus, it seems to have a bottleneck on the
total bytes that can write in dfs. Is this a design setting of hadoop or
there some ways to let the workload spread in non-name node's disk?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message