Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E9031193CE for ; Tue, 12 Apr 2016 10:14:46 +0000 (UTC) Received: (qmail 14990 invoked by uid 500); 12 Apr 2016 10:14:41 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 14868 invoked by uid 500); 12 Apr 2016 10:14:41 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 14857 invoked by uid 99); 12 Apr 2016 10:14:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Apr 2016 10:14:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 5C8B4180220 for ; Tue, 12 Apr 2016 10:14:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id I4RC1TGMt2N1 for ; Tue, 12 Apr 2016 10:14:38 +0000 (UTC) Received: from mail-lf0-f46.google.com (mail-lf0-f46.google.com [209.85.215.46]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id B0D255F20E for ; Tue, 12 Apr 2016 10:14:37 +0000 (UTC) Received: by mail-lf0-f46.google.com with SMTP id c126so18223329lfb.2 for ; Tue, 12 Apr 2016 03:14:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=9SiI+JlnsfljqhQX86iZ+S1BkdYhpI1jzoJ5BvUqZ30=; b=HAwbjQCfdqrS5ZxghmPPrHm+fvZNklJrjf+YijuTWgB6jQMmNb3f2PxDyxOUVVUSCf 2LzHAk6N9OmiklxqlyOhmAkuW9T4jJYmM7PpKrrdVDx5/lGBHlW2iALLIMhjhXUTl7Gf FtjhfkOiRG3i/0VqiP79GnzrNzHjfebfpSH7zDGqs6z8VIXzO15f3ovzMcZFsN7TFl5M Uvl61nCsnotK7N0XkKrtLbW+7goSu3oekgsEVppH7gWq6pAjQsjbgrMUZLA7nYOF8ANS 0hArwAESO8EmnWK7Vt5bsoUBB+lhLVtGNru5/HD514h3bM535/y1CJAl4N2ZtMGXxfU/ roaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=9SiI+JlnsfljqhQX86iZ+S1BkdYhpI1jzoJ5BvUqZ30=; b=DPMpNtE/mG5Vhl0o+oxeJ6Yh9MDckonW00pyxY89bWlOlFwj6LE++1zJBGXbqo9gya QhBvWfOLHH/2xmjQwSx5gAA7vW5eMD0WdScERkQ+7H5z9cdEbMPpxPlu40qy+VXGBUZs u5u4PTI41/maqgq27sCmu0w8dOKhvdIZ2o/u4fcM5iR30IGysz+VMhY4VyfifPj+Enxo hwaDK+KxQn+nEtmoIIKlV6hvkLETevNdOslsGJa2U7Bth7svDuoBt8tMSUcRVOkWxsqP AwRMP2JpEW2XUkHSplkQMnCgdvYO0wsYRi7rQVOYMiPFtUyiTJHh4kVGseIESdus4zyX V11A== X-Gm-Message-State: AOPr4FW+ksOQApKs0DrYccIyBz/oHkcCLPR4QnadezpHEKDN4jWMbjHir/3om66OaxgCqs47KLRyL3PEcHkw6Q== MIME-Version: 1.0 X-Received: by 10.25.161.75 with SMTP id k72mr1085206lfe.86.1460456077169; Tue, 12 Apr 2016 03:14:37 -0700 (PDT) Received: by 10.112.143.231 with HTTP; Tue, 12 Apr 2016 03:14:37 -0700 (PDT) In-Reply-To: References: Date: Tue, 12 Apr 2016 11:14:37 +0100 Message-ID: Subject: Re: Best way to migrate PB scale data between live cluster? From: cs user To: raymond Cc: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a114111bcb09ce8053046edaf --001a114111bcb09ce8053046edaf Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi there, At some point in the near future we are also going to require exactly what you describe. We had hope to use distcp. You mentioned: 1. it do not handle data delete distcp has a -delete flag which says - "Delete the files existing in the dst but not in src" Does this not help with handling deleted data? I believe there is an issue if data is removed during a distcp run, so for example at the start of the run it captures all the files it needs to sync. If some files are deleted during the run, it may lead to errors. Is there a way to ignore these errors and have distcp retry on the next run? I'd be interested in how you manage to eventually accomplish the syncing between the two clusters, because we also need to solve the very same problem :-) Perhaps others on the mailing list have experience with this? Thanks! On Tue, Apr 12, 2016 at 10:44 AM, raymond wrote: > Hi > > > We have a hadoop cluster with several PB data. and we need to migrate it > to a new cluster across datacenter for larger volume capability. > We estimate that the data copy itself might took near a month to finish. > So we are seeking for a sound solution. The requirement is as below: > 1. we cannot bring down the old cluster for such a long time ( of course)= , > and a couple of hours is acceptable. > 2. we need to mirror the data, it means that we not only need to copy the > new data, but also need to delete the deleted data happened during the > migration period. > 3. we don=E2=80=99t have much space left on the old cluster, say 30% room= . > > > regarding distcp, although it might be the easiest way , but > > > 1. it do not handle data delete > 2. it handle newly appended file by compare file size and overwrite it ( > well , it might waste a lot of bandwidth ) > 3. error handling base on file is triffle. > 4 load control is difficult ( we still have heavy work load on old > cluster) you can just try to split your work manually and make it small > enough to achieve the flow control goal. > > > In one word, for a long time mirror work. It won't do well by itself. > > > The are some possible works might need to be done : > > > We can: > > > > 1. Do some wrap work around distcp to make it works better. ( say > error handling, check results. Extra code for sync deleted files etc. = ) > 2. Utilize Snapshot mechanisms for better identify files need to be > copied and deleted. Or renamed. > > > Or > > > > 1. Forget about distcp. Use FSIMAGE and editlog as a change history > source, and write our own code to replay the operation. Handle each fi= le > one by one. ( better per file error handling could be achieved), but t= his > might need a lot of dev works. > > > > > Btw. The closest thing I could found is facebook migration 30PB hive > warehouse: > > > > https://www.facebook.com/notes/facebook-engineering/moving-an-elephant-la= rge-scale-hadoop-data-migration-at-facebook/10150246275318920/ > > > They modifiy the distcp to do a initial bulk load (to better handling > large files and very small files, for load balance I guess.) , and a > replication system (not much detail on this part) to mirror the changes. > > > But it is not clear that how they handle those shortcomings of distcp I > mentioned above. And do they utilize snapshot mechanism. > > > So , does anyone have experience on this kind of work? What do you think > might be the best approaching for our case? Is there any ready works been > done that we can utilize? Is there any works have been done around snapsh= ot > mechanism to easy data migration? > --001a114111bcb09ce8053046edaf Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi there,=C2=A0

At some point in the ne= ar future we are also going to require exactly what you describe. We had ho= pe to use distcp.=C2=A0

You mentioned:
<= br>
1. it= do not handle data delete

<= span style=3D"font-size:14.6667px">distcp has a -delete flag which says -= =C2=A0

"Delete the = files existing in the dst but not in src"

Does this not help with handling= deleted data?

I believe there is an issue if data is removed during a distcp r= un, so for example at the start of the run it captures all the files it nee= ds to sync. If some files are deleted during the run, it may lead to errors= . Is there a way to ignore these errors and have distcp retry on the next r= un?

I'd be interested in how you manage to eventually=C2=A0accomplish= =C2=A0the syncing between the two clusters, because we also need to solve t= he very same problem :-)

Perhaps others on the mailing list have experie= nce with this?


Thanks!
<= span style=3D"color:rgb(0,0,0);font-family:Verdana,Helvetica,sans-serif;fon= t-size:12.8px">

On Tue, Apr 12, 2016 at 10:44 AM, raymond <rgbbones@16= 3.com> wrote:
Hi

=C2=A0

We have a h= adoop cluster with several PB data. and we need to migrate it to a new cluster ac= ross datacenter for larger volume capability.
We estimate that the data copy itself might took near a month to finish. So we are seeking for a sound solution. The requirement is as below:
1. we cannot bring down the old cluster for such a long time ( of course), and a couple of hou= rs is acceptable.
2. we need to = mirror the data, it means that we not only need to copy the new data, but also nee= d to delete the deleted data happened during the migration period.
3. we don=E2=80=99t have much space left on the old cluster, say 30% room.

=C2=A0

regarding distcp, although it might be the easiest way , but=C2=A0

=C2=A0

1. it = do not handle data delete
2. it handle newl= y appended file by compare file size and overwrite it ( well , it might waste= a lot of bandwidth )
3. error h= andling base on file is triffle.=C2=A0
4 load control is difficult ( we still have heavy work load on old cluster) you can just try = to split your work manually and make it small enough to achieve the flow contr= ol goal.

=C2=A0

In one word, for a long time mirror work. It won't do well by itself.

=C2=A0

= The are some possible works might need to be done :

=C2=A0

We can:

=C2=A0

  1. Do=C2=A0 some wrap work around distcp to = make it works better. ( say error handling, check results. Extra code for sync deleted files etc. )
  2. Utilize Snapshot mechanisms for better identify files need to be copied and deleted. Or renamed.

=C2=A0

Or

=C2=A0

  1. Forget about distcp. Use FSIMAGE and editlog as a change history source, and write our own code to replay the operation. Handle each file one by on= e. ( better per file error handling could be achieved), but this might need= a lot of dev works.

=C2=A0

=C2=A0

Btw. The closest thing I could found is facebook migration 30PB hive warehouse:

=C2=A0

=C2=A0

The= y modifiy the distcp to do a initial bulk load (to better handling large file= s and very small files, for load balance I guess.) , and a replication system (not much detail on this part) to mirror the changes.

=C2=A0

But it is not clear that how they handle those shortcomings of distcp I mentioned abo= ve. And do they utilize snapshot mechanism.

=C2=A0

= So , does anyone have experience on this kind of work? What do you think might be the best approaching for our case? Is there any ready works been done that we c= an utilize? Is there any works have been done around snapshot mechanism to eas= y data migration?

--001a114111bcb09ce8053046edaf--