Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of hadoopman@gmail.com designates
 209.85.212.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject
         :content-type:content-transfer-encoding;
        b=Xl+Jkt/sdHqzVQasjeJGQ58BVkqUnNLtZM/5IVPqrJdf31AGavVjdB07CgpKODzN4R
         uxfWbQh/2pQVVfDqqHoRH5Zpsywsir/1rwaRXbS2HsUajBkrVS74cDstK5RJRJlvWtKT
         3PpVFqQ0qlgSwL+k2JHqcHjahYWrn7EakPX4M=
Message-ID: <4CF4769A.7080901@gmail.com>
Date: Mon, 29 Nov 2010 20:59:22 -0700
From: hadoopman <hadoopman@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
 rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: common-user@hadoop.apache.org
Subject: HDFS Rsync process??
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

We have two Hadoop clusters in two separate buildings.  Both clusters 
are loading the same data from the same sources (the second cluster is 
for DR).

We're looking at how we can recover the primary cluster and catch it 
back up again as new data will continue to feed into the DR cluster.  
It's been suggested we use rsync across the network however my concern 
is the amount of data we would have to copy over would take several days 
(at a minimum) to sync them even with our dual bonded 1 gig network cards.

I'm curious if anyone has come up with a solution short of just loading 
the source logs into HDFS.  Is there a way to even rsync two clusters 
and get them in sync?  Been googling around.  Haven't found anything of 
substances yet.

Thanks!