Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <48D11736.9000305@apache.org>
Date: Wed, 17 Sep 2008 09:41:58 -0500
From: Dennis Kubes <kubes@apache.org>
User-Agent: Thunderbird 2.0.0.16 (X11/20080724)
MIME-Version: 1.0
To: core-user@hadoop.apache.org
Subject: Re: scp to namenode faster than dfs put?
References: <200809171743.37525.pvvpr@research.iiit.ac.in>
In-Reply-To: <200809171743.37525.pvvpr@research.iiit.ac.in>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

While an scp will copy data to the namenode machine, it does *not* store 
  the data in dfs, it simply copies the data to namenode machine.   This 
is the same as copying data to any other machine.  The data isn't in DFS 
and is not accessible from DFS.  If the box running the namenode fails 
you lose your data.

The reason put is slower is that the data is actually being stored into 
the DFS on multiple machines in block format.  It is then accessible 
from programs accessing the DFS such as MR jobs.

Dennis

Prasad Pingali wrote:
> Hello,
>    I observe that scp of data to the namenode is faster than actually putting 
> into dfs (all nodes coming from same switch and have same ethernet cards, 
> homogenous nodes)? I understand that "dfs -put" breaks the data into blocks 
> and then copies to datanodes, but shouldn't that be atleast as fast as 
> copying data to namenode from a single machine, if not faster?
> 
> thanks and regards,
> Prasad Pingali,
> IIIT Hyderabad.
> 
>