hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys
Date Sat, 27 Aug 2016 00:01:43 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15440250#comment-15440250
] 

Andrew Purtell commented on HBASE-16232:
----------------------------------------

I have also been chasing sporadic failures due to regions in transition. I was seeing it for
a while on a clusterdock virtual cluster hosted on a d2.4xlarge EC2 instance, so definitely
at smaller scale. 1.2.0 and 1.1.6rc1 (and 0.98.21) were always good but for a time 1.2.2 was
failing reliably. I tried to bisect between 1.2.0 and 1.2.2 but couldn't nail it down. Now
I can no longer reproduce at all. Shame I didn't save logs when I could. I'll be looking at
branch-1 head soon and if it turns up there will save off all logs

> ITBLL fails on branch-1.3, now loosing actual keys
> --------------------------------------------------
>
>                 Key: HBASE-16232
>                 URL: https://issues.apache.org/jira/browse/HBASE-16232
>             Project: HBase
>          Issue Type: Bug
>          Components: dataloss, integration tests
>    Affects Versions: 1.3.0, 1.2.1
>            Reporter: Mikhail Antonov
>             Fix For: 1.3.0
>
>
> So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix for fake keys
showing up in the scans) with increased number of regions per regionserver and seeing the
following.
> {quote} 
> $Verify‚Äč$Counts	
> REFERENCED	0	4,999,999,994	4,999,999,994
> UNDEFINED	0	3	3
> UNREFERENCED	0	3	3
> {quote}
> So we're loosing some keys. This time those aren't fake:
> {quote}
> undef	
> \x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F	0	1	1
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE3	0	1	1
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A	0	1	1
> unref	
> \x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF	0	1	1
> \xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC	0	1	1
> U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15
> {quote}
> Re-running verify step with CM off still shows this issue. Search tool reports:
> {quote}
> Total
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE3	5	0	5
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A	4	0	4
> CELL_WITH_MISSING_ROW	15	0	15
> {quote}
> Will post more as I dig into.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message