Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of airbots@gmail.com designates
 74.125.82.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=kxFz6gMBFs+i+ynYn5CbjQ6tjv04f0dwDkKfW0aF7w4Hy06gp9Ov+a8F4zPG+6aX9s
         Fi9iC3M3NnlTAUAKkWLuXX5Uz5Z99zxQO6VL3FxYwSXSAuAMLOKQSKN5xmzrbwJtosDY
         LKvW9DICRMEztWWm0IHLCkutPQPU1N6GHPV60=
MIME-Version: 1.0
Date: Tue, 1 Mar 2011 10:40:02 -0600
Message-ID: <AANLkTikLpjA5-Wh_ZtWVMfSiXVWwUxRoK2CjKtQC36Nk@mail.gmail.com>
Subject: hadoop balancer
From: He Chen <airbots@gmail.com>
To: common-dev@hadoop.apache.org, common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64c1a26aa0427049d6e72fa

--0016e64c1a26aa0427049d6e72fa
Content-Type: text/plain; charset=ISO-8859-1

Hi all

I met a problem when I try to balance certain hdfs directory among the
clusters. For example, I have a directory "/user/xxx/", and there 100
blocks. I want to balance them among my 5 nodes clusters. Each node has 40
blocks (2 replicas). The problem is about transfer block from one datanode
to another. Actually, I followed the balancer's method. However, it always
waits for the response of destination datanode and halt. I attached the
code:
.................................................

                                      Socket sock = new Socket();

      DataOutputStream out = null;

      DataInputStream in = null;

      try{

        sock.connect(NetUtils.createSocketAddr(

            target.getName()), HdfsConstants.READ_TIMEOUT);

        sock.setKeepAlive(true);

        System.out.println(sock.isConnected());

        out = new DataOutputStream( new BufferedOutputStream(

            sock.getOutputStream(), FSConstants.BUFFER_SIZE));

        out.writeShort(DataTransferProtocol.DATA_TRANSFER_VERSION);

        out.writeByte(DataTransferProtocol.OP_REPLACE_BLOCK);

        out.writeLong(block2move.getBlockId());

        out.writeLong(block2move.getGenerationStamp());

        Text.writeString(out, source.getStorageID());

        System.out.println("Ready to move");

        source.write(out);

        System.out.println("Write to output Stream");

        out.flush();

        System.out.println("out has been flushed!");

        in = new DataInputStream( new BufferedInputStream(

            sock.getInputStream(), FSConstants.BUFFER_SIZE));

        It stop here and wait for response.

        short status = in.readShort();

        System.out.println("Got the response from input stream!"+status);

        if (status != DataTransferProtocol.OP_STATUS_SUCCESS) {

           throw new IOException("block move is failed\t"+status);

        }


      } catch (IOException e) {

        LOG.warn("Error moving block "+block2move.getBlockId()+

            " from " + source.getName() + " to " +

            target.getName() + " through " +

            source.getName() +

            ": "+e.toString());


       } finally {

        IOUtils.closeStream(out);

        IOUtils.closeStream(in);

        IOUtils.closeSocket(sock);
       }
..........................................

Any reply will be appreciated. Thank you in advance!

Chen

--0016e64c1a26aa0427049d6e72fa--