Return-Path: 
 <core-user-return-13520-apmail-hadoop-core-user-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-core-user-archive@www.apache.org
Received: (qmail 77797 invoked from network); 17 Apr 2009 04:27:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Apr 2009 04:27:28 -0000
Received: (qmail 48138 invoked by uid 500); 17 Apr 2009 04:27:25 -0000
Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org
Received: (qmail 48028 invoked by uid 500); 17 Apr 2009 04:27:25 -0000
Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:core-user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:core-user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:core-user@hadoop.apache.org>
List-Id: <core-user.hadoop.apache.org>
Reply-To: core-user@hadoop.apache.org
Delivered-To: mailing list core-user@hadoop.apache.org
Received: (qmail 48018 invoked by uid 99); 17 Apr 2009 04:27:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Apr 2009 04:27:25 +0000
X-ASF-Spam-Status: No, hits=4.6 required=10.0
	tests=FS_REPLICA,HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.44.28] (HELO yx-out-2324.google.com) (74.125.44.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Apr 2009 04:27:16 +0000
Received: by yx-out-2324.google.com with SMTP id 8so38779yxb.29
        for <core-user@hadoop.apache.org>;
 Thu, 16 Apr 2009 21:26:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.100.6.13 with SMTP id 13mr1576751anf.29.1239942413285;
 Thu, 16
	Apr 2009 21:26:53 -0700 (PDT)
In-Reply-To: 
 <CEDB38A443476D40901289F9F19E288A023A1441@IE10EV813.global.ds.honeywell.com>
References: 
 <CEDB38A443476D40901289F9F19E288A023231B6@IE10EV813.global.ds.honeywell.com>
	<623d9cf40904092220p28966db5m5a4a4f0294643bc0@mail.gmail.com>
	<77f4f8890904092226p130448cfwd8e15062eef20b82@mail.gmail.com>
	<CEDB38A443476D40901289F9F19E288A02323255@IE10EV813.global.ds.honeywell.com>
	<CEDB38A443476D40901289F9F19E288A023232D4@IE10EV813.global.ds.honeywell.com>
	<623d9cf40904101034r4c2ca1a8gc5eb27a6291b19c8@mail.gmail.com>
	<CEDB38A443476D40901289F9F19E288A02349B7D@IE10EV813.global.ds.honeywell.com>
	<49E4FFDE.3030502@yahoo-inc.com>
 <CEDB38A443476D40901289F9F19E288A023A1441@IE10EV813.global.ds.honeywell.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Thu, 16 Apr 2009 21:26:38 -0700
Message-ID: <d6d7c4410904162126j41db3f02iafa9a5e730ec9aec@mail.gmail.com>
Subject: Re: More Replication on dfs
To: core-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e644d6ca15f86d0467b897e6
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e644d6ca15f86d0467b897e6
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

That setting will instruct future file writes to replicate two-fold. This
has no bearing on existing files; replication can be set on a per-file
basis, so they already have their replications set in the DFS indivdually.

Use the command: bin/hadoop fs -setrep [-R] repl_factor filename...

to change the replication factor for files already in HDFS
- Aaron

On Wed, Apr 15, 2009 at 10:04 PM, Puri, Aseem <Aseem.Puri@honeywell.com>wrote:

> Hi
>        My problem is not that my data is under replicated. I have 3
> data nodes. In my hadoop-site.xml I also set the configuration as:
>
>  <property>
>  <name>dfs.replication</name>
>  <value>2</value>
>  </property>
>
> But after this also data is replicated on 3 nodes instead of two nodes.
>
> Now, please tell what can be the problem?
>
> Thanks & Regards
> Aseem Puri
>
> -----Original Message-----
> From: Raghu Angadi [mailto:rangadi@yahoo-inc.com]
> Sent: Wednesday, April 15, 2009 2:58 AM
> To: core-user@hadoop.apache.org
> Subject: Re: More Replication on dfs
>
> Aseem,
>
> Regd over-replication, it is mostly app related issue as Alex mentioned.
>
> But if you are concerned about under-replicated blocks in fsck output :
>
> These blocks should not stay under-replicated if you have enough nodes
> and enough space on them (check NameNode webui).
>
> Try grep-ing for one of the blocks in NameNode log (and datnode logs as
> well, since you have just 3 nodes).
>
> Raghu.
>
> Puri, Aseem wrote:
> > Alex,
> >
> > Ouput of $ bin/hadoop fsck / command after running HBase data insert
> > command in a table is:
> >
> > .....
> > .....
> > .....
> > .....
> > .....
> > /hbase/test/903188508/tags/info/4897652949308499876:  Under replicated
> > blk_-5193
> > 695109439554521_3133. Target Replicas is 3 but found 1 replica(s).
> > .
> > /hbase/test/903188508/tags/mapfiles/4897652949308499876/data:  Under
> > replicated
> > blk_-1213602857020415242_3132. Target Replicas is 3 but found 1
> > replica(s).
> > .
> > /hbase/test/903188508/tags/mapfiles/4897652949308499876/index:  Under
> > replicated
> >  blk_3934493034551838567_3132. Target Replicas is 3 but found 1
> > replica(s).
> > .
> > /user/HadoopAdmin/hbase table.doc:  Under replicated
> > blk_4339521803948458144_103
> > 1. Target Replicas is 3 but found 2 replica(s).
> > .
> > /user/HadoopAdmin/input/bin.doc:  Under replicated
> > blk_-3661765932004150973_1030
> > . Target Replicas is 3 but found 2 replica(s).
> > .
> > /user/HadoopAdmin/input/file01.txt:  Under replicated
> > blk_2744169131466786624_10
> > 01. Target Replicas is 3 but found 2 replica(s).
> > .
> > /user/HadoopAdmin/input/file02.txt:  Under replicated
> > blk_2021956984317789924_10
> > 02. Target Replicas is 3 but found 2 replica(s).
> > .
> > /user/HadoopAdmin/input/test.txt:  Under replicated
> > blk_-3062256167060082648_100
> > 4. Target Replicas is 3 but found 2 replica(s).
> > ...
> > /user/HadoopAdmin/output/part-00000:  Under replicated
> > blk_8908973033976428484_1
> > 010. Target Replicas is 3 but found 2 replica(s).
> > Status: HEALTHY
> >  Total size:    48510226 B
> >  Total dirs:    492
> >  Total files:   439 (Files currently being written: 2)
> >  Total blocks (validated):      401 (avg. block size 120973 B) (Total
> > open file
> > blocks (not validated): 2)
> >  Minimally replicated blocks:   401 (100.0 %)
> >  Over-replicated blocks:        0 (0.0 %)
> >  Under-replicated blocks:       399 (99.50124 %)
> >  Mis-replicated blocks:         0 (0.0 %)
> >  Default replication factor:    2
> >  Average block replication:     1.3117207
> >  Corrupt blocks:                0
> >  Missing replicas:              675 (128.327 %)
> >  Number of data-nodes:          2
> >  Number of racks:               1
> >
> >
> > The filesystem under path '/' is HEALTHY
> > Please tell what is wrong.
> >
> > Aseem
> >
> > -----Original Message-----
> > From: Alex Loddengaard [mailto:alex@cloudera.com]
> > Sent: Friday, April 10, 2009 11:04 PM
> > To: core-user@hadoop.apache.org
> > Subject: Re: More Replication on dfs
> >
> > Aseem,
> >
> > How are you verifying that blocks are not being replicated?  Have you
> > ran
> > fsck?  *bin/hadoop fsck /*
> >
> > I'd be surprised if replication really wasn't happening.  Can you run
> > fsck
> > and pay attention to "Under-replicated blocks" and "Mis-replicated
> > blocks?"
> > In fact, can you just copy-paste the output of fsck?
> >
> > Alex
> >
> > On Thu, Apr 9, 2009 at 11:23 PM, Puri, Aseem
> > <Aseem.Puri@honeywell.com>wrote:
> >
> >> Hi
> >>        I also tried the command $ bin/hadoop balancer. But still the
> >> same problem.
> >>
> >> Aseem
> >>
> >> -----Original Message-----
> >> From: Puri, Aseem [mailto:Aseem.Puri@Honeywell.com]
> >> Sent: Friday, April 10, 2009 11:18 AM
> >> To: core-user@hadoop.apache.org
> >> Subject: RE: More Replication on dfs
> >>
> >> Hi Alex,
> >>
> >>        Thanks for sharing your knowledge. Till now I have three
> >> machines and I have to check the behavior of Hadoop so I want
> >> replication factor should be 2. I started my Hadoop server with
> >> replication factor 3. After that I upload 3 files to implement word
> >> count program. But as my all files are stored on one machine and
> >> replicated to other datanodes also, so my map reduce program takes
> > input
> >> from one Datanode only. I want my files to be on different data node
> > so
> >> to check functionality of map reduce properly.
> >>
> >>        Also before starting my Hadoop server again with replication
> >> factor 2 I formatted all Datanodes and deleted all old data manually.
> >>
> >> Please suggest what I should do now.
> >>
> >> Regards,
> >> Aseem Puri
> >>
> >>
> >> -----Original Message-----
> >> From: Mithila Nagendra [mailto:mnagendr@asu.edu]
> >> Sent: Friday, April 10, 2009 10:56 AM
> >> To: core-user@hadoop.apache.org
> >> Subject: Re: More Replication on dfs
> >>
> >> To add to the question, how does one decide what is the optimal
> >> replication
> >> factor for a cluster. For instance what would be the appropriate
> >> replication
> >> factor for a cluster consisting of 5 nodes.
> >> Mithila
> >>
> >> On Fri, Apr 10, 2009 at 8:20 AM, Alex Loddengaard <alex@cloudera.com>
> >> wrote:
> >>
> >>> Did you load any files when replication was set to 3?  If so, you'll
> >> have
> >>> to
> >>> rebalance:
> >>>
> >>>
> >
> <http://hadoop.apache.org/core/docs/r0.19.1/commands_manual.html#balance
> >> r>
> >>> <
> >>>
> >
> http://hadoop.apache.org/core/docs/r0.19.1/hdfs_user_guide.html#Rebalanc
> >> er
> >>> Note that most people run HDFS with a replication factor of 3.
> > There
> >> have
> >>> been cases when clusters running with a replication of 2 discovered
> >> new
> >>> bugs, because replication is so often set to 3.  That said, if you
> > can
> >> do
> >>> it, it's probably advisable to run with a replication factor of 3
> >> instead
> >>> of
> >>> 2.
> >>>
> >>> Alex
> >>>
> >>> On Thu, Apr 9, 2009 at 9:56 PM, Puri, Aseem
> > <Aseem.Puri@honeywell.com
> >>>> wrote:
> >>>> Hi
> >>>>
> >>>>            I am a new Hadoop user. I have a small cluster with 3
> >>>> Datanodes. In hadoop-site.xml values of dfs.replication property
> > is
> >> 2
> >>>> but then also it is replicating data on 3 machines.
> >>>>
> >>>>
> >>>>
> >>>> Please tell why is it happening?
> >>>>
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> Aseem Puri
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
>
>

--0016e644d6ca15f86d0467b897e6--