Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 67EA9116EE for ; Thu, 10 Jul 2014 04:34:38 +0000 (UTC) Received: (qmail 13247 invoked by uid 500); 10 Jul 2014 04:34:33 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 13114 invoked by uid 500); 10 Jul 2014 04:34:33 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 13104 invoked by uid 99); 10 Jul 2014 04:34:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jul 2014 04:34:33 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of aagarwal@hortonworks.com designates 209.85.219.52 as permitted sender) Received: from [209.85.219.52] (HELO mail-oa0-f52.google.com) (209.85.219.52) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jul 2014 04:34:27 +0000 Received: by mail-oa0-f52.google.com with SMTP id j17so9182220oag.39 for ; Wed, 09 Jul 2014 21:34:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=myiFG2BBLY1RPK5p1azFNMIm2yaec6fGrWbdtxq4O6E=; b=ETFlZErlJSCrlFJddbWXLwuX16krZgSedelFgZDoVoZIiuxA2wFN/aBhD26OxDHiGj J2FzOiqZMRyl56uzIw4z3aCwY8wuWfXi5t6KlEShg16/hszLSxZY3gQoITe+9mhKBkI6 fR3FHo3LFLUCc5zEzVsTsBpDLkZSo3xBrNqyWUmShfOTaj9bJyww/x2LTX6Tj0S1UXF2 8Cg2uAAwCjE8GdJw/KrZLxEIumi/JldpDa8VUYcuA12jHd7eibeOsINwNRo5gUatneWr 7l7W6ZHJti1oSMobBwq2aMkm3bn7Q2FWIgCqtTIfPtQ2ub5YW8XNIKht5zwyRZPB+eWx uyeQ== X-Gm-Message-State: ALoCoQlGF2kagFiQ80Oe/bZRDoSa+xgjOl5GI5Jv45BzxGRdF0cz9JGZvC0txV7BGdn/uQjM/U9htg+mpNmtal1Ku0sc5B3ZHtRzH8+klkFClJJqd5VVhPs= MIME-Version: 1.0 X-Received: by 10.60.59.130 with SMTP id z2mr51337413oeq.4.1404966846804; Wed, 09 Jul 2014 21:34:06 -0700 (PDT) Received: by 10.76.167.199 with HTTP; Wed, 9 Jul 2014 21:34:06 -0700 (PDT) In-Reply-To: References: Date: Wed, 9 Jul 2014 21:34:06 -0700 Message-ID: Subject: Re: Copy hdfs block from one data node to another From: Arpit Agarwal To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e013d0800d3464304fdcf55f1 X-Virus-Checked: Checked by ClamAV on apache.org --089e013d0800d3464304fdcf55f1 Content-Type: text/plain; charset=UTF-8 The balancer does something similar. It uses DataTransferProtocol.replaceBlock. On Wed, Jul 9, 2014 at 9:20 PM, sudhakara st wrote: > You can get info about all blocks stored in perticuler data node, i,e > block report. But you to handle, move in block level not in file or start > and end bytes level. > > > On Thu, Jul 10, 2014 at 2:49 AM, Chris Mawata > wrote: > >> Haven't looked at the source but the thing you are trying to do sounds >> similar to what happens when you are decommissioning a datanode. I would >> hunt for that code. >> Cheers >> Chris >> On Jul 9, 2014 3:41 PM, "Yehia Elshater" wrote: >> >>> Hi Chris, >>> >>> Actually I need this functionality for my research, basically for fault >>> tolerance. I can calculate some failure probability for some data nodes >>> after certain unit of time. So I need to copy all the blocks reside on >>> these nodes to another nodes. >>> >>> Thanks >>> Yehia >>> >>> >>> On 7 July 2014 20:45, Chris Mawata wrote: >>> >>>> Can you outline why one would want to do that? The blocks are >>>> disposable so it is strange to manipulate them directly. >>>> On Jul 7, 2014 8:16 PM, "Yehia Elshater" >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> How can copy a certain hdfs block (given the file name, start and end >>>>> bytes) from one node to another node ? >>>>> >>>>> Thanks >>>>> Yehia >>>>> >>>> >>> > > > -- > > Regards, > ...sudhakara > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. --089e013d0800d3464304fdcf55f1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
The balancer does something similar. It uses DataTran= sferProtocol.replaceBlock.

On Wed, Jul 9, 2014 at 9:20 PM, sudhakara st <= span dir=3D"ltr"><sudhakara.st@gmail.com> wrote:
You can get info about all = blocks stored in perticuler data node, i,e block report. But you to handle,= move in block level not in file or start and end bytes level.


On Thu, Jul 10, 2014 at 2:49 AM, Chris Mawata <chris.mawata@gmail.com> wrote:

Haven't looked at the source but the thing you are tryin= g to do sounds similar to what happens when you are decommissioning a datan= ode. I would hunt for that code.
Cheers
Chris

On Jul 9, 2014 3:41 PM, "Yehia Elshater&quo= t; <y.z.elsh= ater@gmail.com> wrote:
Hi Chris,

Actually I need this function= ality for my research, basically for fault tolerance. I can calculate some = failure probability for some data nodes after certain unit of time. So I ne= ed to copy all the blocks reside on these nodes to another nodes.=C2=A0

Thanks
Yehia


On 7 July 2014 20:45, Chris Mawata= <chris.mawata@gmail.com> wrote:

Can you outline why one would= want to do that? The blocks are disposable so it is strange to manipulate = them directly.

On Jul 7, 2014 8:16 PM, "Yehia Elshater&quo= t; <y.z.elsh= ater@gmail.com> wrote:
Hi All,
=
How ca= n copy a certain hdfs block (given the file name, start and end bytes) from= one node to another node ?=C2=A0

Thanks
Yehia




--
=C2=A0 =C2= =A0 =C2=A0=C2=A0
Regards, .= ..sudhakara
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0=C2=A0


CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --089e013d0800d3464304fdcf55f1--