Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7140DD53C for ; Mon, 15 Oct 2012 20:20:19 +0000 (UTC) Received: (qmail 13805 invoked by uid 500); 15 Oct 2012 20:20:15 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 13711 invoked by uid 500); 15 Oct 2012 20:20:14 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 13696 invoked by uid 99); 15 Oct 2012 20:20:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Oct 2012 20:20:14 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.219.48] (HELO mail-oa0-f48.google.com) (209.85.219.48) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Oct 2012 20:20:10 +0000 Received: by mail-oa0-f48.google.com with SMTP id h2so6678268oag.35 for ; Mon, 15 Oct 2012 13:19:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=xwORwy4zM4K6aEiD///5q32pAMwgMrSNgt1ogf5OgcQ=; b=eDgxQPXEK9EgUebz6tTOoHYbV8ijqdPSBcSkeogUMURU+ga4W8Fw92IK1Bj45LxTmA lIvp/CVMXSbzjchRq0EIVoPo5E2926xlE0CS64Txl65cVVWksTGkm/sOac+YprhupcjE dUcpxxdrgkDuNXl9T1aD+cgW4J/gP7aZxMHsEE7p1Rkj9jj4sD8GrEQvTxK2d1vXIX/z qzmHfufIdEuQIgwAmiNjjRAxVPg/Y+UHTanf2HjKuHzAikny6d/UHbXCVq9kRDDKP6AW Z0X0MY94ay2pJQDAQh9GbKF5je8kNcNwsRyqYzjjKjtAjhTH5LCeSL+sK+/Y+J1Lk1MB JPlA== MIME-Version: 1.0 Received: by 10.182.152.74 with SMTP id uw10mr7232921obb.45.1350332389659; Mon, 15 Oct 2012 13:19:49 -0700 (PDT) Received: by 10.60.61.197 with HTTP; Mon, 15 Oct 2012 13:19:49 -0700 (PDT) In-Reply-To: References: Date: Mon, 15 Oct 2012 13:19:49 -0700 Message-ID: Subject: Re: final the dfs.replication and fsck From: Chris Nauroth To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d044518476a139004cc1ec297 X-Gm-Message-State: ALoCoQnQvzlbWnqUQrmct8LYwNSPd2ESsVbs5Df58QJw+wYjdwdPGjipAY6M8d6U2QYoYDNCkC8J X-Virus-Checked: Checked by ClamAV on apache.org --f46d044518476a139004cc1ec297 Content-Type: text/plain; charset=ISO-8859-1 Thank you, Harsh. I did not know about dfs.replication.max. On Mon, Oct 15, 2012 at 12:23 PM, Harsh J wrote: > Hey Chris, > > The dfs.replication param is an exception to the config > feature. If one uses the FileSystem API, one can pass in any short > value they want the replication to be. This bypasses the > configuration, and the configuration (being per-file) is also client > sided. > > The right way for an administrator to enforce a "max" replication > value at a create/setRep level, would be to set > the dfs.replication.max to a desired value at the NameNode and restart > it. > > On Tue, Oct 16, 2012 at 12:48 AM, Chris Nauroth > wrote: > > Hello Patai, > > > > Has your configuration file change been copied to all nodes in the > cluster? > > > > Are there applications connecting from outside of the cluster? If so, > then > > those clients could have separate configuration files or code setting > > dfs.replication (and other configuration properties). These would not be > > limited by final declarations in the cluster's configuration files. > > true controls configuration file resource loading, but it > > does not necessarily block different nodes or different applications from > > running with completely different configurations. > > > > Hope this helps, > > --Chris > > > > > > On Mon, Oct 15, 2012 at 12:01 PM, Patai Sangbutsarakum > > wrote: > >> > >> Hi Hadoopers, > >> > >> I have > >> > >> dfs.replication > >> 2 > >> true > >> > >> > >> set in hdfs-site.xml in staging environment cluster. while the staging > >> cluster is running the code that will later be deployed in production, > >> those code is trying to have dfs.replication of 3, 10, 50, other than > >> 2; the number that developer thought that will fit in production > >> environment. > >> > >> Even though I final the property dfs.replication in staging cluster > >> already. every time i run fsck on the staging cluster i still see it > >> said under replication. > >> I thought final keyword will not honor value in job config, but it > >> doesn't seem so when i run fsck. > >> > >> I am on cdh3u4. > >> > >> please suggest. > >> Patai > > > > > > > > -- > Harsh J > --f46d044518476a139004cc1ec297 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thank you, Harsh. =A0I did not know about dfs.replication.max.

On Mon, Oct 15, 2012 at 12:23 PM, Harsh J <harsh@= cloudera.com> wrote:
Hey Chris,

The dfs.replication param is an exception to the <final> config
feature. If one uses the FileSystem API, one can pass in any short
value they want the replication to be. This bypasses the
configuration, and the configuration (being per-file) is also client
sided.

The right way for an administrator to enforce a "max" replication=
value at a create/setRep level, would be to set
the dfs.replication.max to a desired value at the NameNode and restart
it.

On Tue, Oct 16, 2012 at 12:48 AM, Chris Nauroth
<cnauroth@hortonworks.com> wrote:
> Hello Patai,
>
> Has your configuration file change been copied to all nodes in the clu= ster?
>
> Are there applications connecting from outside of the cluster? =A0If s= o, then
> those clients could have separate configuration files or code setting<= br> > dfs.replication (and other configuration properties). =A0These would n= ot be
> limited by final declarations in the cluster's configuration files= .
> <final>true</final> controls configuration file resource l= oading, but it
> does not necessarily block different nodes or different applications f= rom
> running with completely different configurations.
>
> Hope this helps,
> --Chris
>
>
> On Mon, Oct 15, 2012 at 12:01 PM, Patai Sangbutsarakum
> <silvianhadoop@gmail.com= > wrote:
>>
>> Hi Hadoopers,
>>
>> I have
>> <property>
>> =A0 =A0 <name>dfs.replication</name>
>> =A0 =A0 <value>2</value>
>> =A0 =A0 <final>true</final>
>> =A0 </property>
>>
>> set in hdfs-site.xml in staging environment cluster. while the sta= ging
>> cluster is running the code that will later be deployed in product= ion,
>> those code is trying to have dfs.replication of 3, 10, 50, other t= han
>> 2; the number that developer thought that will fit in production >> environment.
>>
>> Even though I final the property dfs.replication in staging cluste= r
>> already. every time i run fsck on the staging cluster i still see = it
>> said under replication.
>> I thought final keyword will not honor value in job config, but it=
>> doesn't seem so when i run fsck.
>>
>> I am on cdh3u4.
>>
>> please suggest.
>> Patai
>
>



--
Harsh J

--f46d044518476a139004cc1ec297--