Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 3 May 2017 17:01:04 +0000 (UTC)
From: "Sean Busbey (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13022091.1479705316000.117110.1493830864474@Atlassian.JIRA>
In-Reply-To: <JIRA.13022091.1479705316000@Atlassian.JIRA>
References: <JIRA.13022091.1479705316000@Atlassian.JIRA> <JIRA.13022091.1479705316244@jira-lw-us.apache.org>
Subject: [jira] [Updated] (HBASE-17138) Backport read-path offheap
 (HBASE-11425) to branch-1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 03 May 2017 17:01:12 -0000


     [ https://issues.apache.org/jira/browse/HBASE-17138?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Busbey updated HBASE-17138:
--------------------------------
    Component/s: regionserver

> Backport read-path offheap (HBASE-11425) to branch-1
> ----------------------------------------------------
>
>                 Key: HBASE-17138
>                 URL: https://issues.apache.org/jira/browse/HBASE-17138
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>            Reporter: Yu Li
>            Assignee: Yu Sun
>         Attachments: 0001-fix-EHB-511-Resolve-client-compatibility-issue-=
introduced-by-offheap-change.patch, 0001-to-EHB-446-offheap-hfile-format-sh=
ould-keep-compatible-v3.patch, 0001-to-EHB-456-Cell-should-be-compatible-wi=
th-branch-1.1.2.patch
>
>
> From the [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/2016=
11.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gma=
il.com%3E] of sharing our experience and performance data of read-path offh=
eap usage in Alibaba search, we could see people are positive to have HBASE=
-11425 in branch-1, so I'd like to create a JIRA and move the discussion an=
d decision making here.
> Echoing some comments from the mail thread:
> Bryan:
> Is the backported patch available anywhere? If it ends up not getting off=
icially backported to branch-1 due to 2.0 around the corner, some of us who=
 build our own deploy may want to integrate into our builds
> Andrew:
> Yes, please, the patches will be useful to the community even if we decid=
e not to backport into an official 1.x release.
> Enis:
> I don't see any reason why we cannot backport to branch-1.
> Ted:
> Opening a JIRA would be fine. This makes it easier for people to obtain t=
he patch(es)
> Nick:
> From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
> support 1.x releases for some time... I would guess these will be
> maintained until 2.2 at least. Therefore, offheap patches that have seen
> production exposure seem like a reasonable candidate for backport, perhap=
s in a 1.4 or 1.5 release timeframe.
> Anoop:
> Because of some compatibility issues, we decide that this will be done in=
 2.0 only..  Ya as Andy said, it would be great to share the 1.x backported=
 patches.
> The following is all the jira ids we have back ported:
> HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Ce=
lls (Ram)
> HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and Abstract=
HFileReader; ditto for Scanners and BlockReader, etc.
> HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner.
> HBASE-13450 - Purge RawBytescomparator from the writers and readers for H=
BASE-10800 (Ram)
> HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo.
> HBASE-12048 Remove deprecated APIs from Filter.
> HBASE-10800 - Use CellComparator instead of KVComparator (Ram)
> HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byt=
e[], int, int.
> HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with by=
te[],int,int args in favor of taking Cell arg.
> HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int=
 length) in favor of filterRowKey(Cell firstRowCell).
> HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner.
> HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[],=
 int, int.
> HBASE-11911 Break up tests into more fine grained categories (Alex Newman=
)
> HBASE-12059 Create hbase-annotations module
> HBASE-12106 Move test annotations to test artifact (Enis Soztutar)
> HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers.
> HBASE-15679 Assertion on wrong variable in TestReplicationThrottler#testT=
hrottling
> HBASE-13931 Move Unsafe based operations to UnsafeAccess.
> HBASE-12345 Unsafe based ByteBuffer Comparator.
> HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, i=
nt llength, byte[] right, int roffset, int rlength).
> HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc =
warn
> HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport=
 this patch)
> HBASE-13448 New Cell implementation with cached component offsets/lengths=
.
> HBASE-13387 Add ByteBufferedCell an extension to Cell.
> HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum.
> HBASE-12650 Move ServerName to hbase-common module (partially backport th=
is patch)
> HBASE-12296 Filters should work with ByteBufferedCell.
> HBASE-14120 ByteBufferUtils#compareTo small optimization.
> HBASE-13510 - Purge ByteBloomFilter (Ram)
> HBASE-13451 - Make the HFileBlockIndex blockKeys to Cells so that it coul=
d be easy to use in the CellComparators (Ram)
> HBASE-13614 - Avoid temp KeyOnlyKeyValue temp objects creations in read h=
ot path (Ram)
> HBASE-13939 - Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell =
(Ram)
> HBASE-13307 Making methods under ScannerV2#next inlineable, faster
> HBASE-14020 Unsafe based optimized write in ByteBufferOutputStream.
> HBASE-13977 - Convert getKey and related APIs to Cell (Ram)
> HBASE-11927 Use Native Hadoop Library for HFile checksum. (Apekshit)
> HBASE-12213 HFileBlock backed by Array of ByteBuffers (Ram)
> HBASE-12084 Remove deprecated APIs from Result.
> HBASE-12084 Remove deprecated APIs from Result - shell addendum
> HBASE-13754 Allow non KeyValue Cell types also to oswrite.
> HBASE-14047 - Cleanup deprecated APIs from Cell class (Ashish Singhi)
> HBASE-13817 ByteBufferOuputStream - add writeInt support.
> HBASE-12374 Change DBEs to work with new BB based cell.
> HBASE-14116 Change ByteBuff.getXXXStrictlyForward to relative position ba=
sed reads
> HBASE-14073 TestRemoteTable.testDelete failed in the latest trunk code.(J=
ingcheng)
> HBASE-13926 Close the scanner only after Call#setResponse.
> HBASE-12295 Prevent block eviction under us if reads are in progress from=
 the BBs (Ram)
> HBASE-12295 - Addendum for multiGets to add the call back(Ram)
> HBASE-14001 Optimize write(OutputStream out, boolean withTags) for SizeCa=
chedNoTagsKeyValue.
> HBASE-14063 Use BufferBackedCell in read path after HBASE-12213 and HBASE=
-12295 (ram)
> HBASE-14202 Reduce garbage we create.
> HBASE-14099 StoreFile.passesKeyRangeFilter need not create Cells from the=
 Scan's start and stop Row (Ram)
> HBASE-14144 - Bloomfilter path to work with Byte buffered cells (Ram)
> HBASE-14395 Short circuit last byte check in CellUtil#matchingXXX methods=
 for ByteBufferedCells.
> HBASE-12298 Support BB usage in PrefixTree (Ram)
> HBASE-14398 - Create the fake keys required in the scan path to avoid cop=
y to byte[] (Ram)
> HBASE-14480 Small optimization in SingleByteBuff.
> HBASE-14590 Shorten ByteBufferedCell#getXXXPositionInByteBuffer method na=
me (Anoop Sam John)
> HBASE-14636 Clear HFileScannerImpl#prevBlocks in between Compaction flow.
> HBASE-14832 Ensure write paths work with ByteBufferedCells in case of com=
paction (Ram)
> HBASE-14524 Short-circuit comparison of rows in CellComparator. (Lars Fra=
ncke)
> HBASE-14188 - Read path optimizations after HBASE-11425 profiling (Ram)
> HBASE-14188- Read path optimizations after HBASE-11425 profiling-Addendum=
(Ram)
> HBASE-12593 Tags to work with ByteBuffer.
> HBASE-15077 Support OffheapKV write in compaction with out copying data o=
n heap.
> HBASE-14660 AssertionError found when using offheap BucketCache with asse=
rtion enabled (ram)
> HBASE-15076 Add getScanner(Scan scan, List<KeyValueScanner> additionalSca=
nners) API into Region interface (Anoop Sam John
> HBASE-15735(also particial HBASE-13893) Tightening of the CP contract.
> HBASE-15785 Unnecessary lock in ByteBufferArray.
> HBASE-15760 TestBlockEvictionFromClient#testParallelGetsAndScanWithWrappe=
dRegionScanner fails in master branch (Ram)
> HBASE-15379 Fake cells created in read path not implementing SettableSequ=
enceId
> HBASE-16609 Fake cells EmptyByteBufferedCell created in read path not imp=
lementing SettableSequenceId
> HBASE-14940 Make our unsafe based ops more safe
> HBASE-15063 Bug in MultiByteBuf#toBytes. (deepankar)
> HBASE-15064 BufferUnderflowException after last Cell fetched from an HFil=
e Block served from L2 offheap cache.
> HBASE-15064 BufferUnderflowException after last Cell fetched from an HFil=
e Block served from L2 offheap cache - Addendum.
> HBASE-15253 Small bug in CellUtil.matchingRow(Cell, byte[]) (Ram)
> HBASE-16704 Scan will be broken while working with DBE and KeyValueCodecW=
ithTags
> the above jira ids are listed in chronological order I have backport=EF=
=BC=8Cand there are also some patch i dont list, such as:
> 1. keep Cell api compatible with our existing code.
> 2. hfile format related compatibility issues.
> 3. client compatibility issue


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)