hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From OpenInx <open...@gmail.com>
Subject Re: Please help to review patch of the HDFS-14585 (backport Byte
Date Fri, 28 Jun 2019 02:57:37 GMT
> What are your challenges upgrading to Hadoop 3 ?

It's  a major release upgradation from hadoop 2 to hadoop 3.  we upstreams
will need some time to test the stability
and performance, so I think the HBase will not upgrade to hadoop 3 so quick
in short time. Backporting the ByteBuffer
pread to hadoop 2.9 is more suitable for now (it's a minor feature, should
be OK to port in branch-2.9). I believe in future
HBase release we will include the hadoop 3

> Did you verify if your problem statement valid, if you use Ozone API ?
We have some work about the object-store api [1]. but I'm not quite sure
how will it affect the GC.  will take a look about that.

Thanks for your reply.

[1]. https://issues.apache.org/jira/browse/HBASE-22149
On Fri, Jun 28, 2019 at 7:28 AM Sree Vaddi <sree_at_chess@yahoo.com> wrote:

> What are your challenges upgrading to Hadoop 3 ?
> Did you verify if your problem statement valid, if you use Ozone API ?
> Sent from Yahoo Mail on Android
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature>
> On Thu, Jun 27, 2019 at 9:20 AM, OpenInx
> <openinx@gmail.com> wrote:
> Dear HDFS Team:
> I'm apache HBase member and working on HBASE-21879 [1]:  it's a feature
> which help to optimize the
> HBase GC issues, we redesigned the HBase read(Scan/Get) path by reading
> HDFS block into offheap
> ByteBuffers and it had great performance improvement in our benchmark test
> [2].  While the offheap
> HDFS block reading highly depend on the HDFS ByteBuffer interfaces, such
> as: ByteBuffer stream
> read and ByteBuffer positional read.
> The problem is ByteBuffer pread interface: we've finished the HDFS-3246[3]
> for Hadoop 3.3.0 version. but
> the Apache HBase is depending on Hadoop 2.8.5 version, and seems will not
> upgrade the dependency to
> Hadoop 3.3.0 soon.  We have to backport HDFS-3246 to Hadoop branch-2.9  so
> that in the next hadoop
> 2.9.3 can include it , and our Apache HBase can upgrade the dependency to
> 2.9.3.
> Now all the patches of HBASE-21879 have been merged into HBase master
> branch.  If we cannot upgrade to
> the Hadoop which include the ByteBuffer pread implementation,  then I
> believe almost all of work from HBASE-21879
> will worth nothing because we still read heap block and still high GC
> pressure in some special case ( such as
> cacheHitRatio is not very high).  That will be quite frustrating for us.
> Actually, we have been pushing the backport for the past few weeks. our
> XiaoMi HDFS team have prepared the
> backport patches, but seems the guys we ping didn't have the bandwidth to
> help review the patches. we
> expect the backport can be included in Hadoop 2.9.3 so that our upstream
> can benefit from this as soon as
> possible.
> If any HDFS committer or PMC have some bandwidth to review the backport
> patches, please give us a hand.
> The Apache HBase team really need your help.
> The ByteBuffer pread backport issues are:
> https://issues.apache.org/jira/browse/HDFS-14585
> https://issues.apache.org/jira/browse/HDFS-14483
> Thank you very much.
> [1]. https://issues.apache.org/jira/browse/HBASE-21879
> [2]. https://www.mail-archive.com/dev@hbase.apache.org/msg55335.html
> [3]. https://issues.apache.org/jira/browse/HDFS-3246

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message