hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anoop John <anoop.hb...@gmail.com>
Subject Re: meta region written by 2.0 cannot be opened by 1.x
Date Tue, 20 Dec 2016 04:21:11 GMT
So what is the conclusion here ?  We agree to the fact that, when 2.0
upgrade is required, first 1.x upgrade needed (The latest stable 1.x.y
version?) and then to 2.0.   We have to make sure to doc this clearly
for 2.0 release.  Thanks..

-Anoop-

On Wed, Dec 7, 2016 at 4:28 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> I agree.
>
> Just marked 15265 and 10800 incompatible change.
>
>> On Dec 7, 2016, at 2:38 AM, 张铎 <palomino219@gmail.com> wrote:
>>
>> I think the imcompatible change is HBASE-15265? HBASE-14949 aims to fix the
>> incompatible... Just like HBASE-16189 to HBASE-10800.
>>
>> I think we should mark HBASE-15265 and HBASE-10800 as Incompatible Change?
>>
>> 2016-12-07 18:29 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
>>
>>> Shouldn't HBASE-14949 be marked Incompatible Change ?
>>>
>>>> On Wed, Dec 7, 2016 at 2:17 AM, 张铎 <palomino219@gmail.com> wrote:
>>>>
>>>> See this one
>>>>
>>>> https://issues.apache.org/jira/browse/HBASE-14949
>>>>
>>>> 2016-12-07 18:09 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
>>>>
>>>>> Which other features ?
>>>>> Can you name them ?
>>>>>
>>>>> Thanks
>>>>>
>>>>>> On Wed, Dec 7, 2016 at 2:06 AM, 张铎 <palomino219@gmail.com>
wrote:
>>>>>>
>>>>>> I believe there are other features that require upgrading to a
>>> specific
>>>>>> version in each 1.x release before rolling upgrading to 2.0? I think
>>>> this
>>>>>> is a typical trick to add incompatible changes...
>>>>>>
>>>>>> 2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan <
>>>>>> ramkrishna.s.vasudevan@gmail.com>:
>>>>>>
>>>>>>> If its going to be a pain then we need to support writing the
older
>>>>> class
>>>>>>> name in the FFT (trailer) and then use our internal mapping.
>>>>>>>
>>>>>>>> On Wed, Dec 7, 2016 at 3:22 PM, Anoop John <anoop.hbase@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I read the comments in the issue 16189..  This concern was
raised
>>>>>>>> there already..   This is what I replied there.
>>>>>>>> "Ya we may need to make such a need. User on 1.x has to first
>>>> upgrade
>>>>>>>> (rolling upgrade supported) to latest version in 1.x and
then to
>>>> 2.0.
>>>>>>>> This was discussed some other place also. I believe in the
mail
>>>>> thread
>>>>>>>> where we discussed abt rolling upgrade support in 2.0"
>>>>>>>>
>>>>>>>> Not able to find the original mail thread which discussed
abt
>>>> rolling
>>>>>>>> upgrade support in 2.0    Pls correct if took some thing
wrong
>>>> way..
>>>>>>>> Ya I agree that this way of indirection might be inconvenience
>>> for
>>>>>>>> users.
>>>>>>>>
>>>>>>>>
>>>>>>>> -Anoop-
>>>>>>>>
>>>>>>>> On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu <yuzhihong@gmail.com>
>>>> wrote:
>>>>>>>>> bq. write old name only and within code we will have
to map it
>>>> with
>>>>>> new
>>>>>>>>> name.
>>>>>>>>>
>>>>>>>>> This is a viable approach.
>>>>>>>>> Even if we document upgrading to 1.2.3+, some users may
not
>>>> perform
>>>>>>> this
>>>>>>>>> step before upgrading to 2.0
>>>>>>>>>
>>>>>>>>> On Tue, Dec 6, 2016 at 10:04 PM, Anoop John <
>>>> anoop.hbase@gmail.com
>>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Ya that bug raised by Enis was very valid..  So this
means
>>> when
>>>>>>>>>> rolling upgrade happens, if some of the other RSs
with version
>>>>>> <1.2.3
>>>>>>>>>> , which is not having this fix,  this issue might
come up!
>>>>>>>>>> How to address then?   Do we need to enforce a 1.2.3+
upgrade
>>>> 1st
>>>>>> and
>>>>>>>>>> then only 2.0 rolling upgrade?
>>>>>>>>>>
>>>>>>>>>> Or else we will need to fix in 2.0..  We write the
new
>>>> Comparator
>>>>>>>>>> class name in FFT (trunk code)   To fix, we might
have to
>>> write
>>>>> old
>>>>>>>>>> name only and within code we will have to map it
with new
>>> name.
>>>> It
>>>>>>>>>> will be ugly! It can be fixed only in 3.0 may be..
 But that
>>> can
>>>>>> make
>>>>>>>>>> the rolling upgrade story easier for users..   Just
saying the
>>>>>>>>>> possibilities..
>>>>>>>>>>
>>>>>>>>>> Thanks Ted to bring it up again.
>>>>>>>>>>
>>>>>>>>>> -Anoop-
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
>>>>>>>>>> <ramkrishna.s.vasudevan@gmail.com> wrote:
>>>>>>>>>>> I think when Enis reported the issue (
>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-16189),
in
>>>> rolling
>>>>>>>> upgrade
>>>>>>>>>>> regions could move around. So a region created
in 2.0 moved
>>>> to a
>>>>>>>> server
>>>>>>>>>>> with 1.x.
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Ram
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Dec 7, 2016 at 1:27 AM, Stack <stack@duboce.net>
>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu <
>>> yuzhihong@gmail.com
>>>>>
>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Looking at http://hbase.apache.org/book.
>>>>>> html#executing.the.0.96.
>>>>>>>>>> upgrade
>>>>>>>>>>>> ,
>>>>>>>>>>>>> there is step of running "bin/hbase upgrade
-check"
>>>>>>>>>>>>>
>>>>>>>>>>>>> How about adding a sample hfile which
contains
>>>>>>>>>>>>> CellComparator$MetaCellComparator
>>>>>>>>>>>>> so that the upgrade check can read and
verify ?
>>>>>>>>>>>>
>>>>>>>>>>>> We don't support downgrade. Never have.
>>>>>>>>>>>> St.Ack
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu
<
>>>> yuzhihong@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The build I used yesterday didn't
include HBASE-16189
>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/HBASE-16189>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Once it is included, the cluster
can be downgraded
>>> fine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I wonder how users would know that
their existing
>>>>> deployment
>>>>>>> has
>>>>>>>>>>>>>> HBASE-16189 <https://issues.apache.org/
>>>>>> jira/browse/HBASE-16189
>>>>>>>>
>>>>>>>>>> before
>>>>>>>>>>>>>> upgrading to 2.0 release.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Dec 6, 2016 at 2:29 AM, ramkrishna
vasudevan <
>>>>>>>>>>>>>> ramkrishna.s.vasudevan@gmail.com>
wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @Ted
>>>>>>>>>>>>>>> Does your version have this fix
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-16189
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>> Ram
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Dec 6, 2016 at 3:56 PM,
Ted Yu <
>>>>> yuzhihong@gmail.com
>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is the assumption that hbase:meta
would not split ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In other thread, Francis
Liu was proposing
>>> supporting
>>>>>>>> splittable
>>>>>>>>>>>>>>>> hbase:meta in 2.0 release.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Dec 6, 2016, at 2:20
AM, 张铎 <
>>>> palomino219@gmail.com
>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Could this be solved
by hosting meta only on
>>> master?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> BTW, MetaCellComparator
is introduced in
>>>> HBASE-10800.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2016-12-06 17:44 GMT+08:00
Ted Yu <
>>>>> yuzhihong@gmail.com
>>>>>>> :
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>> When I restarted
a cluster with 1.1 , I found
>>> that
>>>>>>>> hbase:meta
>>>>>>>>>>>>> region
>>>>>>>>>>>>>>>>>> (written to by the
previously deployed 2.0)
>>>> couldn't
>>>>> be
>>>>>>>>>> opened:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Caused by: java.io.IOException:
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.io.
>>>>>> hfile.CorruptHFileException:
>>>>>>>>>> Problem
>>>>>>>>>>>>>>> reading
>>>>>>>>>>>>>>>>>> HFile Trailer from
file hdfs://
>>>> yz1.xx.com:8020/apps/
>>>>>>>>>>>>>>> hbase/data/data/
>>>>>>>>>>>>>>>>>> hbase/meta/1588230740/info/
>>>>>>> 599fc8a37311414e876803312009a9
>>>>>>>> 86
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HStore.
>>>>>>>> openStoreFiles(
>>>>>>>>>>>>>>> HStore.java:
>>>>>>>>>>>>>>>>>> 579)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HStore.
>>>>>>>> loadStoreFiles(
>>>>>>>>>>>>>>> HStore.java:
>>>>>>>>>>>>>>>>>> 534)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.
>>>> regionserver.HStore.<init>(
>>>>>>>>>>>>> HStore.java:275)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.
>>>>>>>> instantiateHSto
>>>>>>>>>>>>>>> re(HRegion.
>>>>>>>>>>>>>>>>>> java:5150)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.
>>>> regionserver.HRegion$1.call(
>>>>>>>> HRegion.
>>>>>>>>>>>>>>> java:912)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.
>>>> regionserver.HRegion$1.call(
>>>>>>>> HRegion.
>>>>>>>>>>>>>>> java:909)
>>>>>>>>>>>>>>>>>>       at java.util.concurrent.
>>>>>>> FutureTask.run(FutureTask.
>>>>>>>>>>>> java:266)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>> call(
>>>>>>>>>>>>>>> Executors.java:511)
>>>>>>>>>>>>>>>>>>       at java.util.concurrent.
>>>>>>> FutureTask.run(FutureTask.
>>>>>>>>>>>> java:266)
>>>>>>>>>>>>>>>>>>       ... 3 more
>>>>>>>>>>>>>>>>>> Caused by: org.apache.hadoop.hbase.io.
>>>>>>>>>>>> hfile.CorruptHFileException:
>>>>>>>>>>>>>>>> Problem
>>>>>>>>>>>>>>>>>> reading HFile Trailer
from file hdfs://
>>>>>>>>>>>>>>>>>> yz1.xx.com:8020/apps/hbase/data/data/hbase/
>>>>>>>> meta/1588230740/
>>>>>>>>>>>>>>>>>> info/599fc8a37311414e876803312009a986
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.io.
>>>>>>> hfile.HFile.pickReaderVersion(
>>>>>>>>>>>>>>>> HFile.java:483)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.io.
>>>> hfile.HFile.createReader(
>>>>>>>>>>>>> HFile.java:511)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.
>>>>> regionserver.StoreFile$Reader.
>>>>>>>>>>>>>>>>>> <init>(StoreFile.java:1128)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.
>>>> regionserver.StoreFileInfo.
>>>>>>>>>>>>>>>>>> open(StoreFileInfo.java:267)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.
>>>> regionserver.StoreFile.open(
>>>>>>>> StoreFil
>>>>>>>>>>>>>>> e.java:409)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.StoreFile.
>>>>>>>>>>>>>>>>>> createReader(StoreFile.java:517)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HStore.
>>>>>>>> createStoreFileA
>>>>>>>>>>>>>>> ndReader(
>>>>>>>>>>>>>>>>>> HStore.java:687)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.
>>>> regionserver.HStore.access$
>>>>>> 000(
>>>>>>>>>>>>>>> HStore.java:130)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.
>>>> regionserver.HStore$1.call(
>>>>>>>>>>>>> HStore.java:554)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.
>>>> regionserver.HStore$1.call(
>>>>>>>>>>>>> HStore.java:551)
>>>>>>>>>>>>>>>>>>       ... 6 more
>>>>>>>>>>>>>>>>>> Caused by: java.io.IOException:
java.lang.
>>>>>>>>>>>> ClassNotFoundException:
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.CellComparator$
>>>>>>> MetaCellComparator
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.io.
>>> hfile.FixedFileTrailer.
>>>>>>>> getCompara
>>>>>>>>>>>>>>> torClass(
>>>>>>>>>>>>>>>>>> FixedFileTrailer.java:581)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.io.
>>> hfile.FixedFileTrailer.
>>>>>>>>>>>>> deserializeFromPB(
>>>>>>>>>>>>>>>>>> FixedFileTrailer.java:300)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.io.
>>> hfile.FixedFileTrailer.
>>>>>>>>>>>>>>>>>> deserialize(FixedFileTrailer.java:242)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.io.
>>> hfile.FixedFileTrailer.
>>>>>>>>>>>> readFromStream(
>>>>>>>>>>>>>>>>>> FixedFileTrailer.java:407)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.io.
>>>>>>> hfile.HFile.pickReaderVersion(
>>>>>>>>>>>>>>>> HFile.java:468)
>>>>>>>>>>>>>>>>>>       ... 15 more
>>>>>>>>>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.CellComparator$
>>>>>>> MetaCellComparator
>>>>>>>>>>>>>>>>>>       at java.net.URLClassLoader.
>>>>>>>>>> findClass(URLClassLoader.java:
>>>>>>>>>>>>> 381)
>>>>>>>>>>>>>>>>>>       at java.lang.ClassLoader.
>>>>>>>> loadClass(ClassLoader.java:
>>>>>>>>>> 424)
>>>>>>>>>>>>>>>>>>       at sun.misc.Launcher$
>>>>> AppClassLoader.loadClass(
>>>>>>>>>>>>> Launcher.java:
>>>>>>>>>>>>>>> 331)
>>>>>>>>>>>>>>>>>>       at java.lang.ClassLoader.
>>>>>>>> loadClass(ClassLoader.java:
>>>>>>>>>> 357)
>>>>>>>>>>>>>>>>>>       at java.lang.Class.forName0(Native
>>> Method)
>>>>>>>>>>>>>>>>>>       at java.lang.Class.forName(Class.
>>> java:264)
>>>>>>>>>>>>>>>>>>       at
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.io.
>>> hfile.FixedFileTrailer.
>>>>>>>> getCompara
>>>>>>>>>>>>>>> torClass(
>>>>>>>>>>>>>>>>>> FixedFileTrailer.java:579)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> When user does rolling
upgrade from 1.1 to 2.0,
>>> the
>>>>>> above
>>>>>>>> may
>>>>>>>>>>>> cause
>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>>>> if hbase:meta region
is updated by server running
>>>> 2.0
>>>>>> but
>>>>>>>>>> later
>>>>>>>>>>>>>>>> assigned to
>>>>>>>>>>>>>>>>>> a region server which
still runs 1.1 (due to
>>> crash
>>>> of
>>>>>> the
>>>>>>>>>> server
>>>>>>>>>>>>>>> running
>>>>>>>>>>>>>>>>>> 2.0, e.g.)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I want to get community
feedback on the severity
>>> of
>>>>>> this
>>>>>>>>>> issue.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks
>>>

Mime
View raw message