Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
From: Andrew Purtell <andrew.purtell@gmail.com>
Content-Type: multipart/alternative;
	boundary=Apple-Mail-3B869125-C3D8-492B-87EB-69C777B62080
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (1.0)
Date: Thu, 22 Sep 2016 20:16:08 -0700
Subject: Re: [DISCUSSION] MR jobs started by Master or RS
Message-Id: <CCC8D028-D6B5-4ADD-ADEE-E7AC89C1BC3C@gmail.com>
References: <CAFb6Hi+cu1=qgYtBLEq1JXE5tWO1C_WVym+scBhnmWOia2c1tA@mail.gmail.com> <0D1ED081-8BBB-4486-87E3-8F4E3BDFC638@gmail.com> <CALte62y5zmgTC08RpJXLzAk2u7xpyLhz321SXoFnFTFpTx9KRg@mail.gmail.com> <1474580970259.54191@hortonworks.com> <CAFb6HiKUOAmE-whbe6uGBfTgsfma4u2CT4HO+uKgpWh8i+GB-A@mail.gmail.com> <54FE16D7-96FB-4853-8864-1008DD61320D@gmail.com> <CAAAYAnPFGnSmfGGbF6sgT4O-_6LoyrP-Y87cZzsXHa2WZU3swQ@mail.gmail.com> <CAAAYAnOc9nD2MtsdOWEY3cMXwGPq4N-tNWGGXKAseu8S1i3s-Q@mail.gmail.com> <CAKKbPbKGghziThHn-26EpCEiCnke3kQdafRVpmVfm_Wd1RNMQw@mail.gmail.com> <CALte62xQO54KwNjxtEeygcv_ygkSiscKU7ZwxKXAnOCKr8QxSw@mail.gmail.com> <CAKKbPbL3YnvSUnqZ+1v=Qw+2RCTgz+ujL-4P=K94TjOsUHYzEw@mail.gmail.com> <CAAg3a2qV3-WnvTxr2w=KpUWLHDYrLjbQUVCdYKQOAX97QGGiXQ@mail.gmail.com> <CAAg3a2qWLwRBxqQJT3pFPfjcdPrwLna=NJy5TEGEagktMyKVLg@mail.gmail.com>
In-Reply-To: <CAAg3a2qWLwRBxqQJT3pFPfjcdPrwLna=NJy5TEGEagktMyKVLg@mail.gmail.com>
To: dev@hbase.apache.org
archived-at: Fri, 23 Sep 2016 03:16:18 -0000

--Apple-Mail-3B869125-C3D8-492B-87EB-69C777B62080
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

Agreed, this would be interesting to contemplate.=20

On Sep 22, 2016, at 8:03 PM, Vladimir Rodionov <vladrodionov@gmail.com> wrot=
e:

>>> No, never.
>=20
> No need for M/R here, just a simple compaction-server colocated with RS on=

> a same node.
> You save a lot on GC in RS. Ideally, it can be IO "nice" in Linux (by
> setting IO priority). But offtopic, of course :)
>=20
> -Vlad
>=20
> On Thu, Sep 22, 2016 at 7:57 PM, Vladimir Rodionov <vladrodionov@gmail.com=
>
> wrote:
>=20
>>>> And if MR not deployed,  Backup/Restore feature could not be used,
>> right?
>>=20
>> Yes.
>>=20
>> On Thu, Sep 22, 2016 at 7:53 PM, Heng Chen <heng.chen.1986@gmail.com>
>> wrote:
>>=20
>>> {quote}
>>> If MR framework is not deployed in the cluster, hbase still functions
>>> normally (post merge).
>>> {quote}
>>>=20
>>> If MR is not strong dependency for Master/RS,  it is OK for me.
>>> And if MR not deployed,  Backup/Restore feature could not be used, right=
?
>>>=20
>>> 2016-09-23 10:49 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
>>>> If MR framework is not deployed in the cluster, hbase still functions
>>>> normally (post merge).
>>>>=20
>>>> In terms of build time dependency, we have long been depending on
>>>> mapreduce. Take a look at ExportSnapshot.
>>>>=20
>>>> Cheers
>>>>=20
>>>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <heng.chen.1986@gmail.com>
>>> wrote:
>>>>=20
>>>>> In our production cluster,  it is a common case we just have HDFS and
>>>>> HBase deployed.
>>>>> If our Master/RS depend on MR framework (especially some features we
>>>>> have not used at all),  it introduced another cost for maintain.  I
>>>>> don't think it is a good idea.
>>>>>=20
>>>>> 2016-09-23 10:28 GMT+08:00 =E5=BC=A0=E9=93=8E <palomino219@gmail.com>:=

>>>>>> To be specific, for example, our nice Backup/Restore feature, if we
>>> think
>>>>>> this is not a core feature of HBase, then we could make it depend on
>>> MR,
>>>>>> and start a standalone BackupManager instance that submits MR jobs
>>> to do
>>>>>> periodical maintenance job. And if we think this is a core feature
>>> that
>>>>>> everyone should use it, then we'd better implement it without MR
>>>>>> dependency, like DLS.
>>>>>>=20
>>>>>> Thanks.
>>>>>>=20
>>>>>> 2016-09-23 10:11 GMT+08:00 =E5=BC=A0=E9=93=8E <palomino219@gmail.com>=
:
>>>>>>=20
>>>>>>> I=E2=80=98m -1 on let master or rs launch MR jobs. It is OK that som=
e of our
>>>>>>> features depend on MR but I think the bottom line is that we should
>>>>> launch
>>>>>>> the jobs from outside manually or by other services.
>>>>>>>=20
>>>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <andrew.purtell@gmail.com
>>>> :
>>>>>>>=20
>>>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a fair
>>>>>>>> question.
>>>>>>>>=20
>>>>>>>> Can this be driven by a utility derived from Tool like our other MR=

>>>>> apps?
>>>>>>>> The issue is needing the AccessController to decide if allowed? But=

>>>>> nothing
>>>>>>>> prevents the user from running the job manually/independently,
>>> right?
>>>>>>>>=20
>>>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
>>>>> theo.bertozzi@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>=20
>>>>>>>>> just a remark. my query was not about tools using MR (everyone i
>>>>> think
>>>>>>>> is
>>>>>>>>> ok with those).
>>>>>>>>> the topic was about: "are we ok with running MR jobs from Master
>>> and
>>>>> RSs
>>>>>>>>> code?" since this will be the first time we do this
>>>>>>>>>=20
>>>>>>>>> Matteo
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
>>> ddas@hortonworks.com>
>>>>>>>> wrote:
>>>>>>>>>>=20
>>>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
>>> Restore,
>>>>> it's
>>>>>>>>>> fine to be dependent on MR. MR is the right framework for such.
>>> We
>>>>>>>> should
>>>>>>>>>> also do compactions using MR (just saying :) )
>>>>>>>>>> ________________________________________
>>>>>>>>>> From: Ted Yu <yuzhihong@gmail.com>
>>>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
>>>>>>>>>> To: dev@hbase.apache.org
>>>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>>>>>>>>>>=20
>>>>>>>>>> I agree - backup / restore is in the same category as import /
>>>>> export.
>>>>>>>>>>=20
>>>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>>>>>>>> andrew.purtell@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>=20
>>>>>>>>>>> Backup is extra tooling around core in my opinion. Like import
>>> or
>>>>>>>> export.
>>>>>>>>>>> Or the optional MOB tool. It's fine.
>>>>>>>>>>>=20
>>>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
>>>>> mbertozzi@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>=20
>>>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
>>>>> (Master
>>>>>>>> or
>>>>>>>>>>> RS)?
>>>>>>>>>>>>=20
>>>>>>>>>>>> I remember in the past that there was discussion about not
>>> having
>>>>> MR
>>>>>>>>>> has
>>>>>>>>>>>> direct dependency of hbase.
>>>>>>>>>>>>=20
>>>>>>>>>>>> I think some of discussion where around MOB that had a MR job
>>> to
>>>>>>>>>> compact,
>>>>>>>>>>>> that later was transformed in a non-MR job to be merged, I
>>> think
>>>>> we
>>>>>>>>>> had a
>>>>>>>>>>>> similar discussion for log split/replay.
>>>>>>>>>>>>=20
>>>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that runs
>>> a MR
>>>>> job
>>>>>>>>>>> from
>>>>>>>>>>>> the master to copy data or restore data.
>>>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
>>> backup
>>>>>>>>>> you'll
>>>>>>>>>>>> not end up running MR jobs, but this was probably true for
>>> MOB as
>>>>> in
>>>>>>>>>> "if
>>>>>>>>>>>> you don't enable MOB you don't need MR")
>>>>>>>>>>>>=20
>>>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
>>> hbase
>>>>> run
>>>>>>>>>> MR
>>>>>>>>>>>> jobs, only tool started manually by the user can do that". or
>>> can
>>>>> we
>>>>>>>>>>> start
>>>>>>>>>>>> adding MR calls around without problems?
>>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>=20
>>>>>>>=20
>>>>>=20
>>>=20
>>=20
>>=20

--Apple-Mail-3B869125-C3D8-492B-87EB-69C777B62080--