hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kun Ren <ren.h...@gmail.com>
Subject Re: HDFS Federation
Date Wed, 25 May 2016 20:58:09 GMT
Yes, it is rename. Thanks for the explanation, make sense.

The rename example you gave(across namenodes) is like a distributed
transaction,  standard protocol like two-phase commit can be used to
support distributed transactions in Database world, so is that possible to
support the "atomic distributed rename" using something like two-phase
commit protocol? Or even it is possible, the implementation should be very
difficult or the performance is bad?

On Wed, May 25, 2016 at 4:26 PM, Chris Nauroth <cnauroth@hortonworks.com>
wrote:

> You might be thinking of renames.  For example, when using ViewFs with
> Federation, the client-side mount configuration might have path /data1
> backed by NameNode1 and path /data2 backed by NameNode2.  Renaming
> /data1/file1 to /data2/file1 would not be supported, because it would need
> to be copied across NameNodes, and then HDFS would not be able to satisfy
> its promise that rename is atomic.  There are more details about this in
> the ViewFs guide.
>
>
> http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/ViewFs.html
>
> --Chris Nauroth
>
> From: Kun Ren <ren.hdfs@gmail.com>
> Date: Wednesday, May 25, 2016 at 11:03 AM
> To: Chris Nauroth <cnauroth@hortonworks.com>
> Cc: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Subject: Re: HDFS Federation
>
> Thanks a lot, Chris.
>
> I remembered that Federation doesn't support some cross-namenodes
> operations, if so,  which operations that Federation doesn't support and
> why? Thanks again.
>
> On Wed, May 25, 2016 at 12:17 PM, Chris Nauroth <cnauroth@hortonworks.com>
> wrote:
>
>> Hello Kun,
>>
>> Yes, this command works with federation.  The command would copy the file
>> from NameNode 1/block pool 1 to NameNode 2/block pool 2.
>>
>> --Chris Nauroth
>>
>> From: Kun Ren <ren.hdfs@gmail.com>
>> Date: Wednesday, May 25, 2016 at 8:57 AM
>> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
>> Subject: HDFS Federation
>>
>> Hi Genius,
>>
>> Does HDFS Federation support the cross namenodes operations?
>>
>> For example:
>>
>> ./bin/hdfs dfs -cp input1/a.xml input2/b.xml
>>
>> Supposed that input1 belongs namenode 1, and input 2 belongs namenode 2,
>> does Federation support this operation? And if not, why?
>>
>> Thanks.
>>
>
>

Mime
View raw message