hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vladrodio...@gmail.com>
Subject Re: [DISCUSSION] MR jobs started by Master or RS
Date Thu, 22 Sep 2016 23:25:40 GMT
Forgot WALPlayer :)

-Vlad

On Thu, Sep 22, 2016 at 4:21 PM, Vladimir Rodionov <vladrodionov@gmail.com>
wrote:

> >> and
> >> backups too, but don't want to bother having to install and configure
> YARN
> >> just for that, as well as removing resources from HBase to give it to
>
> Any suggestions on how to do bulk data move with transformation from/to
> HBase cluster w/o MapReduce?
>
> Opposition to M/R does not make sense imo, as since we have a lot of tools
> in HBase which depend on MapReduce:
>
> CountRows
> CountCells
> Import
> Export
> ImportTsv
> ExportTsv
> CopyTable
> VerifyReplication
> ExportSnapshot
>
> and new backup create/restore of course.
>
>
> -Vlad
>
>
>
>
> On Thu, Sep 22, 2016 at 4:15 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> My 2ยข: I have a strong preference for NOT having a dependency on MR
>> anywhere :( I run my HBase cluste without YARN. Just HBase and HDFS. I
>> like
>> all the features that we built. Would love to be able to use MOBs and
>> backups too, but don't want to bother having to install and configure YARN
>> just for that, as well as removing resources from HBase to give it to
>> yarn....
>>
>> JMS
>>
>> 2016-09-22 18:44 GMT-04:00 Matteo Bertozzi <theo.bertozzi@gmail.com>:
>>
>> > just a remark. my query was not about tools using MR (everyone i think
>> is
>> > ok with those).
>> > the topic was about: "are we ok with running MR jobs from Master and RSs
>> > code?" since this will be the first time we do this
>> >
>> > Matteo
>> >
>> >
>> > On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <ddas@hortonworks.com>
>> wrote:
>> >
>> > > Very much agree; for tools like ExportSnapshot / Backup / Restore,
>> it's
>> > > fine to be dependent on MR. MR is the right framework for such. We
>> should
>> > > also do compactions using MR (just saying :) )
>> > > ________________________________________
>> > > From: Ted Yu <yuzhihong@gmail.com>
>> > > Sent: Thursday, September 22, 2016 2:00 PM
>> > > To: dev@hbase.apache.org
>> > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>> > >
>> > > I agree - backup / restore is in the same category as import / export.
>> > >
>> > > On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>> > andrew.purtell@gmail.com>
>> > > wrote:
>> > >
>> > > > Backup is extra tooling around core in my opinion. Like import or
>> > export.
>> > > > Or the optional MOB tool. It's fine.
>> > > >
>> > > > > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
>> mbertozzi@apache.org>
>> > > > wrote:
>> > > > >
>> > > > > What's the latest opinion around running MR jobs from hbase
>> (Master
>> > or
>> > > > RS)?
>> > > > >
>> > > > > I remember in the past that there was discussion about not having
>> MR
>> > > has
>> > > > > direct dependency of hbase.
>> > > > >
>> > > > > I think some of discussion where around MOB that had a MR job
to
>> > > compact,
>> > > > > that later was transformed in a non-MR job to be merged, I think
>> we
>> > > had a
>> > > > > similar discussion for log split/replay.
>> > > > >
>> > > > > the latest is the new Backup feature (HBASE-7912), that runs
a MR
>> job
>> > > > from
>> > > > > the master to copy data or restore data.
>> > > > > (backup is also "not really core" as in.. if you don't use backup
>> > > you'll
>> > > > > not end up running MR jobs, but this was probably true for MOB
as
>> in
>> > > "if
>> > > > > you don't enable MOB you don't need MR")
>> > > > >
>> > > > > any thoughts? do we a rule that says "we don't want to have hbase
>> run
>> > > MR
>> > > > > jobs, only tool started manually by the user can do that". or
can
>> we
>> > > > start
>> > > > > adding MR calls around without problems?
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message