Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9D1FC10128 for ; Tue, 29 Apr 2014 08:41:11 +0000 (UTC) Received: (qmail 91718 invoked by uid 500); 29 Apr 2014 08:41:02 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 91228 invoked by uid 500); 29 Apr 2014 08:41:01 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 91219 invoked by uid 99); 29 Apr 2014 08:41:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Apr 2014 08:41:00 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of sudhakara.st@gmail.com designates 209.85.219.51 as permitted sender) Received: from [209.85.219.51] (HELO mail-oa0-f51.google.com) (209.85.219.51) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Apr 2014 08:40:54 +0000 Received: by mail-oa0-f51.google.com with SMTP id l6so827540oag.24 for ; Tue, 29 Apr 2014 01:40:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=7ZChkr1e4Smp9na+EG3HUJqhU6uZafv/F87rd0vUVB0=; b=PYgOKCBB8vMtW21W3ohpsY39XG3et4sFBA6y9fWmJ1+Des2tP7Ed5uKZjnHas4HEhX fWSw9bGGq1Ve7tDOaZPmRyaB94ZDN+ygMt+wGuFyquWBc9EA+wZLYQY+pLiRi2iJ/IIU YANPJr9ciJJNFi64aLQYyQx7lqMzwaPZ1OEH456XGqb1Nx1conQ7ajagUXd9QsEykF7a 4ZtEAVHYeI26Q0J9npXWgMMzqqAGWRez162IWY3kSRJ1gqhleguJY2YTCTyA7CSZKh7Q WJGPtV8rgsLCxLgSrLeUc0bgLBpHI8aT/CI69ZIQFC8gUigMmS03ncf3C1HJMPcOyy6M bH6A== MIME-Version: 1.0 X-Received: by 10.182.33.6 with SMTP id n6mr6472378obi.48.1398760833852; Tue, 29 Apr 2014 01:40:33 -0700 (PDT) Received: by 10.76.109.44 with HTTP; Tue, 29 Apr 2014 01:40:33 -0700 (PDT) In-Reply-To: References: Date: Tue, 29 Apr 2014 14:10:33 +0530 Message-ID: Subject: Re: Can set different dfs.replication for different dirs From: sudhakara st To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e0158a836a0ab2804f82a6260 X-Virus-Checked: Checked by ClamAV on apache.org --089e0158a836a0ab2804f82a6260 Content-Type: text/plain; charset=UTF-8 Hello Nitin, HDFS replication factor is always associated with file level. When your copying or creating file for any directory will set to default replicas. but you can specify your replication when creating or copying files hadoop fs -D dfs.replication=2 -put foo.txt fsput and in java FileSystem fs = FileSystem.get(new Configuration()); fs.setReplication(new Path("hdfs_path:/foldername/filename"), (short)1); On Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar wrote: > Sudhakar, > > will this set for the new files being written to those directories? > > > On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st wrote: > >> Changes the replication factor of a file. -R option is for recursively >> increasing the replication factor of files within a directory. >> >> Example: >> >> - hadoop fs -setrep -w 3 -R /user/hadoop/dir1 >> >> hadoop dfs -setrep -R -w 1 /dir/ >> >> >> On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar wrote: >> >>> DFS replication is set on the file level (block level) or at cluster >>> level (if you do not specify the replication factor while writing the file >>> then this one is picked). >>> As per my understanding, there is nothing for directories. >>> >>> >>> >>> >>> On Mon, Apr 28, 2014 at 2:12 PM, Meng QingPing wrote: >>> >>>> Hi, >>>> >>>> I want set dfs.replication as 1 for dfs /tmp, set dfs.replication as 3 >>>> for dfs /user. How to configure? Both /tmp and /user are generated by >>>> mapreduce job or hive or sqoop. >>>> >>>> Thanks, >>>> Jack >>>> >>> >>> >>> >>> -- >>> Nitin Pawar >>> >> >> >> >> -- >> >> Regards, >> ...sudhakara >> >> > > > > -- > Nitin Pawar > -- Regards, ...sudhakara --089e0158a836a0ab2804f82a6260 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello Nitin,

H= DFS replication factor is always associated with file level. When your copy= ing or creating file for any directory will set to default=C2=A0 replicas.<= br>
but you can specify your replication when creating or copying files=C2=A0=C2=A0 hadoop fs -D dfs.replication=3D2 -put foo.txt fsput

<= /div>and in java
FileSystem=
 fs =3D FileSystem.get(new Configuration());
fs.setReplication(new <=
span class=3D"">Path("=
hdfs_path:/foldername/filename"), (short)1);<=
/code>



On = Mon, Apr 28, 2014 at 6:11 PM, Nitin Pawar <nitinpawar432@gmail.com> wrote:
Sudhakar,

will this set for the new files being written to those directories?=C2=A0=


On Mon, Apr 28, 2014 at 5:52 PM, sudhakara st &= lt;sudhakara.st= @gmail.com> wrote:

Changes the replication factor of a file. -R option is for=20 recursively increasing the replication factor of files within a=20 directory.

Example:

  • =09 hadoop fs -setrep -w 3 -R /user/hadoop/dir1

hadoop dfs -setrep -R -w 1 /dir/



On Mon, Apr 28, 2014 at 2:29 PM, Nitin Pawar <nitinpawar432@gmail.c= om> wrote:
DFS replication is set on the file level (block level) or = at cluster level (if you do not specify the replication factor while writin= g the file then this one is picked).=C2=A0
As per my understanding, the= re is nothing for directories.=C2=A0



On Mon, Apr 28, 2014 at 2:12 PM, Meng Qing= Ping <mqingping@gmail.com> wrote:
Hi,

I want set dfs.r= eplication as 1 for dfs /tmp, set dfs.replication as 3 for dfs /user. How t= o configure?=C2=A0 Both /tmp and /user are generated by mapreduce job or hi= ve or sqoop.

Thanks,
Jack



<= font color=3D"#888888">--
Nitin Pawar



--
=C2=A0 =C2=A0 =C2=A0=C2=A0
Regards,
...sudhakara
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0=C2=A0



--
Nitin Pawar



--
=C2=A0= =C2=A0 =C2=A0=C2=A0
Regards,
<= span style=3D"color:rgb(255,0,255)">...sudhakara
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0=C2=A0
--089e0158a836a0ab2804f82a6260--