Return-Path: 
 <user-return-34971-apmail-hbase-user-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-user-archive@www.apache.org
Delivered-To: apmail-hbase-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6BC05FFE0
	for <apmail-hbase-user-archive@www.apache.org>;
 Sun, 24 Mar 2013 12:46:53 +0000 (UTC)
Received: (qmail 81731 invoked by uid 500); 24 Mar 2013 12:46:51 -0000
Delivered-To: apmail-hbase-user-archive@hbase.apache.org
Received: (qmail 81644 invoked by uid 500); 24 Mar 2013 12:46:51 -0000
Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hbase.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hbase.apache.org>
List-Post: <mailto:user@hbase.apache.org>
List-Id: <user.hbase.apache.org>
Reply-To: user@hbase.apache.org
Delivered-To: mailing list user@hbase.apache.org
Received: (qmail 81632 invoked by uid 99); 24 Mar 2013 12:46:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 24 Mar 2013 12:46:51 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ogdude@googlemail.com
 designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-la0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 24 Mar 2013 12:46:46 +0000
Received: by mail-la0-f48.google.com with SMTP id fq13so9634974lab.7
        for <user@hbase.apache.org>; Sun, 24 Mar 2013 05:46:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=20120113;
        h=mime-version:x-received:date:message-id:subject:from:to
         :content-type;
        bh=HBz6s+yQXoaTFMnILCedqtkHh1qC5cHYJUFw4nsDl0I=;
        b=AGqr7sGqYAXsNkoDn/0xr+BtsZ1tNP+Qm1zSaKo9znl5+alVfDVK5qLULpRmTPY/1y
         SiFAyxzIAFnx3Bw7m+mYJt8l4cKc/voY6c2cI/gyiuyoqSCiDrD2jWtMvGmKs2sAzygI
         PdmlKtPEh6XJybnQeiOQwg1y3ahE3tYAPJ81tQ978oYB18up5ZJufqnbYqATykjWQTg4
         U4h/8nsZKGdgkrYFkUO6hCITjkXll71JxKf+BjsZziz96PmcwWOD5uu/dCCuFbF2Fa0n
         NzLKeAzNhJMBh2JDyiqhLQcKSWJkWof9F0jaiP95K7ML2JIHYCrA2WAp+IFzg3mlWAXU
         5tJg==
MIME-Version: 1.0
X-Received: by 10.112.23.193 with SMTP id o1mr4174804lbf.66.1364129185291;
 Sun, 24 Mar 2013 05:46:25 -0700 (PDT)
Received: by 10.152.6.40 with HTTP; Sun, 24 Mar 2013 05:46:25 -0700 (PDT)
Date: Sun, 24 Mar 2013 13:46:25 +0100
Message-ID: 
 <CAE24rAcLN6XvRJUFLhrHUvw_WB+xAUxD9g22YC2S7ZYB1wbcTw@mail.gmail.com>
Subject: hbase.mapred.output.quorum ignored in Mapper job with HDFS source and
 HBase sink
From: David Koch <ogdude@googlemail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=485b390f7c00846cdb04d8ab1300
X-Virus-Checked: Checked by ClamAV on apache.org

--485b390f7c00846cdb04d8ab1300
Content-Type: text/plain; charset=ISO-8859-1

Hello,

I want to import a file on HDFS from one cluster A (source) into HBase
tables on a different cluster B (destination) using a Mapper job with an
HBase sink. Both clusters run HBase.

This setup works fine:
- Run Mapper job on cluster B (destination)
- "mapred.input.dir" --> hdfs://<cluster-A>/<path-to-file> (file on source
cluster)
- "hbase.zookeeper.quorum" --> <quorum-hostname-B>
- "hbase.zookeeper.property.clientPort" --> <quorum-port-B>

I thought it should be possible to run the job on cluster A (source) and
using "hbase.mapred.output.quorum" to insert into the tables on cluster B.
This is what the CopyTable utility does. However, the following does not
work. HBase looks for the destination table(s) on cluster A and NOT cluster
B:
- Run Mapper job on cluster A (source)
- "mapred.input.dir" --> hdfs://<cluster-A>/<path-to-file> (file is local)
- "hbase.zookeeper.quorum" --> <quorum-hostname-A>
- "hbase.zookeeper.property.clientPort" --> <quorum-port-A>
- "hbase.mapred.output.quorum" -> <quorum-hostname-B>:2181:/hbase (same as
--peer.adr argument for CopyTable)

Job setup inside the class MyJob is as follows, note I am using
MultiTableOutputFormat.

Configuration conf = HBaseConfiguration.addHbaseResources(getConf());
Job job = new Job(conf);
job.setJarByClass(MyJob.class);
job.setMapperClass(JsonImporterMapper.class);
// Note, several output tables!
job.setOutputFormatClass(MultiTableOutputFormat.class);
job.setNumReduceTasks(0);
TableMapReduceUtil.addDependencyJars(job);
TableMapReduceUtil.addDependencyJars(job.getConfiguration());

Where The Mapper class has the following frame:

public static class JsonImporterMapper extends
    Mapper<LongWritable, Text, ImmutableBytesWritable, Put> { }

Is this expected behaviour? How can I get the second scenario using
hbase.mapred.output.quorum" to work? Could the fact I am using
MultiTableOutputFormat instead of TableOutputFormat play a part? I am using
HBase 0.92.1.

Thank you,

/David

--485b390f7c00846cdb04d8ab1300--