Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAGiZwvO5f5xnh+=iQKLOXvLJHj_zq0XapV+qirM979yjaQfjcA@mail.gmail.com>
References: <CAGiZwvO5f5xnh+=iQKLOXvLJHj_zq0XapV+qirM979yjaQfjcA@mail.gmail.com>
From: Ravi Prakash <ravihadoop@gmail.com>
Date: Mon, 13 Feb 2017 09:31:55 -0800
Message-ID: <CAMs9kVjs=KQQaFddk1_Zk6zX33vkAr_Z3K6n4pgSsoNRSREyqg@mail.gmail.com>
Subject: Re: How to fix "HDFS Missing replicas"
To: Ascot Moss <ascot.moss@gmail.com>
Cc: user <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=94eb2c07dd06e818c805486cd2ad
archived-at: Mon, 13 Feb 2017 17:32:17 -0000

--94eb2c07dd06e818c805486cd2ad
Content-Type: text/plain; charset=UTF-8

Hi Ascot!

Just out of curiosity, which version of hadoop are you using?

fsck has some other options (e.g. -blocks will print out the block report
too, -list-corruptfileblocks prints out the list of missing blocks and
files they belong to) . I suspect you may also want to specify the
-openforwrite option.

In any case, missing blocks are a pretty bad symptom. There's a high
likelihood that you've lost data. If you can't find the blocks on any of
the datanodes, you would want to delete the files on HDFS and recreate them
(however they were originally created). In my experience I've seen missing
files which were never closed. This used to happen in older versions when
an rsync via HDFS NFS / HDFS FUSE is cancelled / fails.

HTH
Ravi


On Sun, Feb 12, 2017 at 4:15 AM, Ascot Moss <ascot.moss@gmail.com> wrote:

> Hi,
>
> After running 'hdfs fsck /blocks' to check the cluster, I got
> 'Missing replicas:              441 (0.24602923 %)"
>
> How to fix HDFS missing replicas?
> Regards
>
>
>
>
> (detailed output)
>
> Status: HEALTHY
>
>  Total size:    3375617914739 B (Total open files size: 68183613174 B)
>
>  Total dirs:    2338
>
>  Total files:   39960
>
>  Total symlinks:                0 (Files currently being written: 60)
>
>  Total blocks (validated):      59493 (avg. block size 56739749 B) (Total
> open file blocks (not validated): 560)
>
>  Minimally replicated blocks:   59493 (100.0 %)
>
>  Over-replicated blocks:        0 (0.0 %)
>
>  Under-replicated blocks:       111 (0.18657658 %)
>
>  Mis-replicated blocks:         0 (0.0 %)
>
>  Default replication factor:    3
>
>  Average block replication:     3.0054965
>
>  Corrupt blocks:                0
>
>  Missing replicas:              441 (0.24602923 %)
>
>  Number of data-nodes:          7
>
>  Number of racks:               1
>
>
>

--94eb2c07dd06e818c805486cd2ad
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Hi Ascot!<br><br>Just out of curiosity, which version=
 of hadoop are you using?<br><br></div><div>fsck has some other options (e.=
g. -blocks will print out the block report too, -list-corruptfileblocks pri=
nts out the list of missing blocks and files they belong to) . I suspect yo=
u may also want to specify the -openforwrite option.<br><br></div><div>In a=
ny case, missing blocks are a pretty bad symptom. There&#39;s a high likeli=
hood that you&#39;ve lost data. If you can&#39;t find the blocks on any of =
the datanodes, you would want to delete the files on HDFS and recreate them=
 (however they were originally created). In my experience I&#39;ve seen mis=
sing files which were never closed. This used to happen in older versions w=
hen an rsync via HDFS NFS / HDFS FUSE is cancelled / fails.<br><br></div><d=
iv>HTH<br></div><div>Ravi<br></div><div><br><br></div><div class=3D"gmail_e=
xtra"><br><div class=3D"gmail_quote">On Sun, Feb 12, 2017 at 4:15 AM, Ascot=
 Moss <span dir=3D"ltr">&lt;<a href=3D"mailto:ascot.moss@gmail.com" target=
=3D"_blank">ascot.moss@gmail.com</a>&gt;</span> wrote:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr">Hi,<div><br></div><div>After running &#39;hd=
fs fsck /blocks&#39; to check the cluster, I got<br>&#39;Missing replicas:=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 441 (0.24602923 %)&quot;=
=C2=A0</div><div><br>How to fix HDFS missing=C2=A0replicas?<br>Regards</div=
><div><br></div><div><br><br><br>(detailed output)<br>


<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">Status: HEALTHY</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Total size:=C2=A0 =C2=A0 3375617914739 B (Total open fi=
les size: 68183613174 B)</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Total dirs:=C2=A0 =C2=A0 2338</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Total files: =C2=A0 39960</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Total symlinks:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 0 (Files currently being written: 60)</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Total blocks (validated):=C2=A0 =C2=A0 =C2=A0 59493 (av=
g. block size 56739749 B) (Total open file blocks (not validated): 560)</sp=
an></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Minimally replicated blocks: =C2=A0 59493 (100.0 %)</sp=
an></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Over-replicated blocks:=C2=A0 =C2=A0 =C2=A0 =C2=A0 0 (0=
.0 %)</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Under-replicated blocks: =C2=A0 =C2=A0 =C2=A0 111 (0.18=
657658 %)</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Mis-replicated blocks: =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 (0=
.0 %)</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Default replication factor:=C2=A0 =C2=A0 3</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Average block replication: =C2=A0 =C2=A0 3.0054965</spa=
n></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Corrupt blocks:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 0</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Missing replicas:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 441 (0.24602923 %)</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Number of data-nodes:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 7</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><span class=3D"m_236946477315337=
0559gmail-s1">=C2=A0Number of racks: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 1</span></p>
<p class=3D"m_2369464773153370559gmail-p1"><br></p></div></div>
</blockquote></div><br></div></div>

--94eb2c07dd06e818c805486cd2ad--