hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Meng QingPing <mqingp...@gmail.com>
Subject Re: Can set different dfs.replication for different dirs
Date Tue, 29 Apr 2014 08:56:10 GMT
Thanks for all replies.

The files in /tmp most are generated by hadoop jobs. Can set the
yarn/mapreduce to specify one repication for tmp files?

Thanks,
Jack


2014-04-29 16:40 GMT+08:00 sudhakara st <sudhakara.st@gmail.com>:

> Hello Nitin,
>
> HDFS replication factor is always associated with file level. When your
> copying or creating file for any directory will set to default  replicas.
> but you can specify your replication when creating or copying files
>    hadoop fs -D dfs.replication=2 -put foo.txt fsput
>
> and in java
>
> FileSystem fs = FileSystem.get(new Configuration());
> fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1);
>
>
>
>
> On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <nitinpawar432@gmail.com>wrote:
>
>> Sudhakar,
>>
>> will this set for the new files being written to those directories?
>>
>>
>> On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st <sudhakara.st@gmail.com>wrote:
>>
>>> Changes the replication factor of a file. -R option is for recursively
>>> increasing the replication factor of files within a directory.
>>>
>>> Example:
>>>
>>>    -  hadoop fs -setrep -w 3 -R /user/hadoop/dir1
>>>
>>> hadoop dfs -setrep -R -w 1 /dir/
>>>
>>>
>>> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <nitinpawar432@gmail.com>wrote:
>>>
>>>> DFS replication is set on the file level (block level) or at cluster
>>>> level (if you do not specify the replication factor while writing the file
>>>> then this one is picked).
>>>> As per my understanding, there is nothing for directories.
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing <mqingping@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as
3
>>>>> for dfs /user. How to configure?  Both /tmp and /user are generated by
>>>>> mapreduce job or hive or sqoop.
>>>>>
>>>>> Thanks,
>>>>> Jack
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Nitin Pawar
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Regards,
>>> ...sudhakara
>>>
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
>
> Regards,
> ...sudhakara
>
>



-- 
Thanks,
Qingping

Mime
View raw message