Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
To: user@hadoop.apache.org
From: David Watzke <david@watzke.cz>
Subject: datanode directory structure mess-up
Message-ID: <56DAF18A.20203@watzke.cz>
Date: Sat, 5 Mar 2016 15:47:38 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.6.0
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="------------000602060908080003060105"

--------------000602060908080003060105
Content-Type: text/plain; charset=iso-8859-2; format=flowed
Content-Transfer-Encoding: 7bit

Hi list,

I ran into trouble because I accidentally usedthis tool 
https://github.com/killerwhile/volume-balancerwith Hadoop 2.6.0 (just 
like that page warns you not to -- I used it successfully before and 
didn't think to check that page before using it again) and it messed up 
my datadirs because as I understand it that software now makes invalid 
assumptions about what directory moves can it do. Now the datanode logs 
are filled with these:

WARN org.apache.hadoop.hdfs.server.datanode.VolumeScanner: I/O error 
while finding block 
BP-680964103-A.B.C.D-1375882473930:blk_5822441067008155275_0 on volume 
/xyz/dfs/dn

What can I do to fix this? I don't know what files/dirs were moved and 
from where but is there a reasonable way out of this? Such as editing 
VERSION file to a previous version when DN is down so that it fixes the 
layout by itself - would that work?

Please note that I've lost the other replica due to a filesystem error 
so I can't just ignore it. This is literally my only option to recover 
some missing blocks.

Thanks,

-- 
David Watzke


--------------000602060908080003060105
Content-Type: text/html; charset=iso-8859-2
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=iso-8859-2">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <meta http-equiv="content-type" content="text/html;
      charset=iso-8859-2">
    <p style="margin: 0px; padding: 0px; color: rgb(51, 51, 51);
      font-family: Arial, sans-serif; font-size: 14px; font-style:
      normal; font-variant: normal; font-weight: normal; letter-spacing:
      normal; line-height: 20px; orphans: auto; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255);">Hi list,<br>
      <br>
    </p>
    <p style="margin: 0px; padding: 0px; color: rgb(51, 51, 51);
      font-family: Arial, sans-serif; font-size: 14px; font-style:
      normal; font-variant: normal; font-weight: normal; letter-spacing:
      normal; line-height: 20px; orphans: auto; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255);">I ran into trouble because
      I accidentally used<span class="Apple-converted-space"> this tool
      </span><a href="https://github.com/killerwhile/volume-balancer"
        class="external-link" rel="nofollow" style="color: rgb(59, 115,
        175); text-decoration: none;">https://github.com/killerwhile/volume-balancer</a><span
        class="Apple-converted-space">�</span>with Hadoop 2.6.0 (just
      like that page warns you not to -- I used it successfully before
      and didn't think to check that page before using it again) and it
      messed up my datadirs because as I understand it that software now
      makes invalid assumptions about what directory moves can it do.
      Now the datanode logs are filled with these:</p>
    <p style="margin: 10px 0px 0px; padding: 0px; color: rgb(51, 51,
      51); font-family: Arial, sans-serif; font-size: 14px; font-style:
      normal; font-variant: normal; font-weight: normal; letter-spacing:
      normal; line-height: 20px; orphans: auto; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255);">WARN
      org.apache.hadoop.hdfs.server.datanode.VolumeScanner: I/O error
      while finding block
      BP-680964103-A.B.C.D-1375882473930:blk_5822441067008155275_0 on
      volume /xyz/dfs/dn</p>
    <p style="margin: 10px 0px 0px; padding: 0px; color: rgb(51, 51,
      51); font-family: Arial, sans-serif; font-size: 14px; font-style:
      normal; font-variant: normal; font-weight: normal; letter-spacing:
      normal; line-height: 20px; orphans: auto; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255);">What can I do to fix this?
      I don't know what files/dirs were moved and from where but is
      there a reasonable way out of this? Such as editing VERSION file
      to a previous version when DN is down so that it fixes the layout
      by itself - would that work?</p>
    <p style="margin: 10px 0px 0px; padding: 0px; color: rgb(51, 51,
      51); font-family: Arial, sans-serif; font-size: 14px; font-style:
      normal; font-variant: normal; font-weight: normal; letter-spacing:
      normal; line-height: 20px; orphans: auto; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255);">Please note that I've lost
      the other replica due to a filesystem error so I can't just ignore
      it. This is literally my only option to recover some missing
      blocks.</p>
    <p style="margin: 10px 0px 0px; padding: 0px; color: rgb(51, 51,
      51); font-family: Arial, sans-serif; font-size: 14px; font-style:
      normal; font-variant: normal; font-weight: normal; letter-spacing:
      normal; line-height: 20px; orphans: auto; text-align: start;
      text-indent: 0px; text-transform: none; white-space: normal;
      widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px;
      background-color: rgb(255, 255, 255);">Thanks,</p>
    <pre class="moz-signature" cols="72">-- 
David Watzke</pre>
  </body>
</html>

--------------000602060908080003060105--