hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tianying Chang <tych...@gmail.com>
Subject Re: Major compaction cannot remove deleted rows until the region is split. Strange!
Date Tue, 31 May 2016 22:43:02 GMT
Hi, Stack

Based on the log, the major compaction was run, and it took 5+ hours.  And
I also manually run major_compact from hbase shell explicitly to verify.

I just moved the region to a different RS and issued a major_compact on
that region again, let me see if the major compaction can succeed and will
report back.

Thanks
Tian-Ying

On Sun, May 29, 2016 at 4:35 PM, Stack <stack@duboce.net> wrote:

> On Fri, May 27, 2016 at 3:17 PM, Tianying Chang <tychang@gmail.com> wrote:
>
> > Yes, it is 94.26.  By a quick glance, I didn't  see any put that is older
> > than the delete marker's TS, which could go as far as about couple weeks
> > ago since major compaction on it for long time seems.
> >
> Also it is really strange that if the region is split, then seems
> > everything is working as expected. Also we noticed, the same region
> > replicated at the slave side is totally normal, i.e. at 20+G....
> >
> >
> If you move the region to another server, does that work?
>
> Looking in 0.94 codebase, I see this in Compactor#compact
>
>
>       // For major compactions calculate the earliest put timestamp
>
>       // of all involved storefiles. This is used to remove
>
>       // family delete marker during the compaction.
>
>       if (majorCompaction) {
>
>         tmp = fileInfo.get(StoreFile.EARLIEST_PUT_TS);
>
>         if (tmp == null) {
>
>           // There's a file with no information, must be an old one
>
>           // assume we have very old puts
>
>           earliestPutTs = HConstants.OLDEST_TIMESTAMP;
>
>         } else {
>
>           earliestPutTs = Math.min(earliestPutTs, Bytes.toLong(tmp));
>
>         }
>
>       }
>
>
> The above is followed by this log line:
>
>
>       if (LOG.isDebugEnabled()) {
>
>         LOG.debug("Compacting " + file +
>
>           ", keycount=" + keyCount +
>
>           ", bloomtype=" + r.getBloomFilterType().toString() +
>
>           ", size=" + StringUtils.humanReadableInt(r.length()) +
>
>           ", encoding=" + r.getHFileReader().getEncodingOnDisk() +
>
>           (majorCompaction? ", earliestPutTs=" + earliestPutTs: ""));
>
>       }
>
> This prints out earliestPutTs. You see that in the logs?  You running with
> DEBUG? Does the earliest put ts preclude our dropping delete family?
>
>
> Looking more in code, we retain deletes in following circumstances:
>
>
>     this.retainDeletesInOutput = scanType == ScanType.MINOR_COMPACT || scan
> .isRaw();
>
>
> So, for sure we are running major compaction?
>
> Otherwise, have to dig in a bit more here.. This stuff is a little
> involved.
> St.Ack
>
>
>
>
> > On Fri, May 27, 2016 at 3:13 PM, Stack <stack@duboce.net> wrote:
> >
> > > On Fri, May 27, 2016 at 2:32 PM, Tianying Chang <tychang@gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > We saw a very strange case in one of our production cluster. A couple
> > > > regions cannot get their deleted rows or delete marker removed even
> > after
> > > > major compaction. However when the region triggered split (we set
> 100G
> > > for
> > > > auto split), the deletion worked. The 100G region becomes two 10G
> > > daughter
> > > > regions, and all the delete marker are gone.
> > > >
> > > > Also, the same region in the slave cluster (through replication) have
> > > > normal size at about 20+G.
> > > >
> > > > BTW, the delete marker in the regions are mostly deleteFamily if it
> > > > matters.
> > > >
> > > > This is really weird. Anyone has any clue for this strange behavior?
> > > >
> > > > Thanks
> > > > Tian-Ying
> > > >
> > > > These 0.94 Tian-Ying?
> > >
> > > It looks like the DeleteFamily is retained only; do you see incidence
> > where
> > > there may have been versions older than the DeleteFamily that are also
> > > retained post-major-compaction?
> > >
> > > St.Ack
> > >
> > >
> > >
> > > > A snippet of the HFile generated by the major compaction:
> > > >
> > > > : \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > > \x10\x00?PF/d:/1459808114380/DeleteFamily/vlen=0/ts=2292870047
> > > > V:
> > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > > \x10\x00?PF/d:/1459808114011/DeleteFamily/vlen=0/ts=2292869794
> > > > V:
> > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > > \x10\x00?PF/d:/1459805381104/DeleteFamily/vlen=0/ts=2291072240
> > > > V:
> > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > > \x10\x00?PF/d:/1459805380673/DeleteFamily/vlen=0/ts=2291071997
> > > > V:
> > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > > \x10\x00?PF/d:/1459802643449/DeleteFamily/vlen=0/ts=2290248886
> > > > V:
> > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > > \x10\x00?PF/d:/1459802643246/DeleteFamily/vlen=0/ts=2290248786
> > > > V:
> > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > > \x10\x00?PF/d:/1459799913003/DeleteFamily/vlen=0/ts=2289446916
> > > > V:
> > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > > \x10\x00?PF/d:/1459797181831/DeleteFamily/vlen=0/ts=2288670451
> > > > V:
> > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > > \x10\x00?PF/d:/1459794447388/DeleteFamily/vlen=0/ts=2287911443
> > > > V:
> > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > > \x10\x00?PF/d:/1459791708803/DeleteFamily/vlen=0/ts=2287213792
> > > > V:
> > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > > \x10\x00?PF/d:/1459788978387/DeleteFamily/vlen=0/ts=2286488738
> > > > V:
> > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > > \x10\x00?PF/d:/1459786243642/DeleteFamily/vlen=0/ts=2285778927
> > > > V:
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message