Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com
 designates 65.55.111.109 as permitted sender)
Message-ID: <BLU436-SMTP225D4BC57680FAB63BF1C318F2D0@phx.gbl>
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Mac OS X Mail 7.3 \(1878.2\))
Subject: Re: HBase export limit bandwith
From: Michael Segel <michael_segel@hotmail.com>
In-Reply-To: 
 <CALte62ykK26PoA3uUzJoBzGSDaHf+0v6OgGo4_K7uf3wDnZBcA@mail.gmail.com>
Date: Thu, 5 Jun 2014 17:07:41 -0500
Content-Transfer-Encoding: quoted-printable
References: <538F139E.6050305@viadeoteam.com>
 <BLU436-SMTP51744DA75755BD110E13B08F2D0@phx.gbl>
 <CALte62ykK26PoA3uUzJoBzGSDaHf+0v6OgGo4_K7uf3wDnZBcA@mail.gmail.com>
To: user@hbase.apache.org

I guess you could say snap shot as in a point in time M/R job that =
exports all of the rows in the table before a specified time X which =
could default to the start time of the job.=20

Since you're running your export to the same cluster (but a different =
directory from /hbase, you don't really have to worry about the number =
of mappers.=20
However, since its a backup... you may want to reduce the number of =
region files so you could reduce the data set to 10, 100, etc... =
depending on the size of the underlying table and then as you write out =
from the reducer you could write to S3 directly, but if you want more =
control... you reduce to the local HDFS, then in a separate job or =
single threaded program, you could open up a file at a time and trickle =
it in. (Or write a map only job that has a set number of mappers defined =
to run in parallel.=20

The only caveat is that you need to make sure you have enough disk space =
to store the local copy until you complete the S3 write.=20

Of course there are other permutations.... like if you have a NAS/SAN =
you could move the export there.=20
(Hot =3D=3D Hbase table. Warm =3D=3D HDFS outside of HBase, Luke Warm =3D=3D=
 local attached disks, Cold =3D S3...)=20

Again, it depends on the resources available to you and your enterprise. =
 YMMV.=20
On Jun 5, 2014, at 9:15 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> bq. take a snapshot and write the file(s)
>=20
> Is the above referring to hbase snapshot ?
> hbase 0.92.x doesn't support snapshot.
>=20
> FYI
>=20
>=20
> On Thu, Jun 5, 2014 at 5:11 AM, Michael Segel =
<michael_segel@hotmail.com>
> wrote:
>=20
>> Ok...
>>=20
>> So when the basic tools don't work...
>> How about roll your own?
>>=20
>> Step 1 take a snapshot and write the file(s) to a different location
>> outside of /hbase.
>> (Export to local disk on the cluster)
>>=20
>> Step 2 write your own M/R job and control the number of mappers who =
read
>> from HDFS and write to S3.
>> Assuming you want a block for block match. If you want to change the
>> #files since each region would be a separate file, you could do the =
write
>> to S3 in the reduce phase.
>> (Which is what you want.)
>>=20
>>=20
>> On Jun 4, 2014, at 7:39 AM, Damien Hardy <dhardy@viadeoteam.com> =
wrote:
>>=20
>>> Hello,
>>>=20
>>> We are trying to export HBase table on S3 for backup purpose.
>>> By default export tool run a map per region and we want to limit =
output
>>> bandwidth on internet (to amazon s3).
>>>=20
>>> We were thinking in adding some reducer to limit the number of =
writers
>>> but this is explicitly hardcoded to 0 in Export class
>>> ```
>>>   // No reducers. Just write straight to output files.
>>>   job.setNumReduceTasks(0);
>>> ```
>>>=20
>>> Is there an other way (propertie?) in hadoop to limit output =
bandwidth ?
>>>=20
>>> --
>>> Damien
>>>=20
>>=20
>> The opinions expressed here are mine, while they may reflect a =
cognitive
>> thought, that is purely accidental.
>> Use at your own risk.
>> Michael Segel
>> michael_segel (AT) hotmail.com
>>=20
>>=20
>>=20
>>=20
>>=20
>>=20

The opinions expressed here are mine, while they may reflect a cognitive =
thought, that is purely accidental.=20
Use at your own risk.=20
Michael Segel
michael_segel (AT) hotmail.com