hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil Govindan <sun...@apache.org>
Subject Re: [DISCUSS] ARM/aarch64 support for Hadoop
Date Thu, 05 Sep 2019 08:57:34 GMT
Thanks Vinay for starting the thread.

I agree to Anu's view point related to protobuf. And with the suggestion
pointed out by Duo Zhang, if we can make use
of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
of protobuf will also be more easier.

However i think its better to do this effort in trunk itself.
In offline talks, few members were interested to start 3.3.0 release. And
given that happens soon, I feel its better
we do this task in trunk itself as branch diverge is very much possible.
And to bring to call a merge on such a big branch will be even more tough
task.

my 2 cents.

Thanks
Sunil

On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <palomino219@gmail.com> wrote:

> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to generate
> the protobuf code. It will download the protoc binary from the maven
> central so we do not need to install protoc on the build machine any more.
>
> Zhenyu Zheng <zhengzhenyulixi@gmail.com> 于2019年9月4日周三 下午5:27写道:
>
> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
> for
> > over 2 month related to the Protobuf problem .
> > According to the latest successful build log:
> >
> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
> > the
> > os version was ubuntu 14.04 and for the jobs that are failling now such
> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
> > the os version is 18.04. I'm not very familiar with the version changing
> > for the jobs but I did a little search, according to:
> >
> >
> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
> > &
> >
> >
> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
> > it both said that the version of libprotc-dev and protobuf-compiler
> > available for ubuntu 18.04 is 3.0.0
> >
> >
> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ayushtkn@gmail.com> wrote:
> >
> >> Thanx Vinay for the initiative, Makes sense to add support for different
> >> architectures.
> >>
> >> +1, for the branch idea.
> >> Good Luck!!!
> >>
> >> -Ayush
> >>
> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <palomino219@gmail.com>
> >> wrote:
> >> >
> >> > For HBase, we purged all the protobuf related things from the public
> >> API,
> >> > and then upgraded to a shaded and relocated version of protobuf. We
> have
> >> > created a repo for this:
> >> >
> >> > https://github.com/apache/hbase-thirdparty
> >> >
> >> > But since the hadoop dependencies still pull in the protobuf 2.5 jars,
> >> our
> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
> >> discuss
> >> > on how to deal with the upgrading of coprocessor. Glad to see that the
> >> > hadoop community is also willing to solve the problem.
> >> >
> >> > Anu Engineer <aengineer@cloudera.com.invalid> 于2019年9月3日周二
上午1:23写道:
> >> >
> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
> that
> >> >> Hadoop and the downstream projects work correctly after you upgrade
> >> core
> >> >> components like Protobuf.
> >> >> So while branching and working on a branch is easy, merging back
> after
> >> you
> >> >> upgrade some of these core components is insanely hard. You might
> want
> >> to
> >> >> make sure that community buys into upgrading these components in the
> >> trunk.
> >> >> That way we will get testing and downstream components will notice
> when
> >> >> things break.
> >> >>
> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
> long
> >> >> time; I have argued that 2.5 is out of support and we cannot stay on
> >> that
> >> >> branch forever; or we need to take ownership of the Protobuf 2.5 code
> >> base.
> >> >> It has been rightly pointed to me that while all the arguments I make
> >> is
> >> >> correct; it is a very complicated task to upgrade Protobuf, and the
> >> worst
> >> >> part is we will not even know what breaks until downstream projects
> >> pick up
> >> >> these changes and work against us.
> >> >>
> >> >> If we work off the Hadoop version 3 — and assume that we have
> >> "shading" in
> >> >> place for all deployments; it might be possible to get there; still
a
> >> >> daunting task.
> >> >>
> >> >> So best of luck with the branch approach — But please remember,
> Merging
> >> >> back will be hard, Just my 2 cents.
> >> >>
> >> >> — Anu
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
> zhengzhenyulixi@gmail.com
> >> >
> >> >> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea.
A
> >> separate
> >> >>> branch with it's own ARM CI seems a really good idea.
> >> >>> By doing this we won't break any of the undergoing development
in
> >> trunk
> >> >> and
> >> >>> a CI can be a very good way to show what are the
> >> >>> current problems and what have been fixed, it will also provide
a
> very
> >> >> good
> >> >>> view for contributors that are intrested to working on
> >> >>> this. We can finally merge back the branch to trunk until the
> >> community
> >> >>> thinks it is good enough and stable enough. We can donate
> >> >>> ARM machines to the existing CI system for the job.
> >> >>>
> >> >>> I wonder if this approch possible?
> >> >>>
> >> >>> BR,
> >> >>>
> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <liusheng2048@gmail.com
> >
> >> >>> wrote:
> >> >>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
> community
> >> >>>> mentioned by Vinay. I am working on building and
> >> >>>> testing Hadoop components on aarch64 server these days, besides
the
> >> >>> missing
> >> >>>> dependices of ARM platform issues #1 #2 #3
> >> >>>> mentioned by Vinay, other similar issue has also be found,
such as
> >> the
> >> >>>> "PhantomJS" dependent package also missing for aarch64.
> >> >>>>
> >> >>>> To promote the ARM support for Hadoop, we have discussed and
hoped
> to
> >> >> add
> >> >>>> an ARM specific CI to Hadoop repo. we are not
> >> >>>> sure about if there is any potential effect or confilict on
the
> trunk
> >> >>>> branch, so maybe creating a ARM specific branch for doing these
> stuff
> >> >>>> is a better choice, what do you think?
> >> >>>>
> >> >>>> Hope to hear thoughts from you :)
> >> >>>>
> >> >>>> BR,
> >> >>>> Liu sheng
> >> >>>>
> >> >>>> Vinayakumar B <vinayakumarb@apache.org> 于2019年8月27日周二
上午5:34写道:
> >> >>>>
> >> >>>>> Hi Folks,
> >> >>>>>
> >> >>>>> ARM is becoming famous lately in its processing capability
and has
> >> >> got
> >> >>>> the
> >> >>>>> potential to run Bigdata workloads.
> >> >>>>> Many users have been moving to ARM machines due to its
low cost.
> >> >>>>>
> >> >>>>> In the past there were attempts to compile Hadoop on ARM
(Rasberry
> >> >> PI)
> >> >>>> for
> >> >>>>> experimental purposes. Today ARM architecture is taking
some of
> the
> >> >>>>> serverside processing as well. So there will be/is a real
need of
> >> >>> Hadoop
> >> >>>> to
> >> >>>>> support ARM architecture as well.
> >> >>>>>
> >> >>>>> There are bunch of users who are trying out building Hadoop
on
> ARM,
> >> >>>> trying
> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some
> >> >>>>>
> >> >>>>> As of today, Hadoop does not compile on ARM due to below
issues,
> >> >> found
> >> >>>> from
> >> >>>>> testing done in openlab in [2].
> >> >>>>>
> >> >>>>> 1. Protobuf :
> >> >>>>> -------------------
> >> >>>>>     Hadoop project (also some downstream projects) stuck
to
> protobuf
> >> >>>> 2.5.0
> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0
is
> >> not
> >> >>>> being
> >> >>>>> maintained in the community. While protobuf 3.x is being
actively
> >> >>> adopted
> >> >>>>> widely, still protobuf 3.x provides wire compatibility
for proto2
> >> >>>> messages.
> >> >>>>> Due to some compilation issues in the generated java code,
which
> can
> >> >>>> induce
> >> >>>>> problems in downstream. Due to this reason protobuf upgrade
from
> >> >> 2.5.0
> >> >>>> was
> >> >>>>> not taken up.
> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries
to avoid
> >> >>> classpath
> >> >>>>> problem in downstream projects.
> >> >>>>>    There are patches available to fix compilation in Hadoop.
But
> >> >> need
> >> >>> to
> >> >>>>> find a way to upgrade protobuf to latest version and still
> maintain
> >> >> the
> >> >>>>> downstream's classpath using shading feature of Hadoop
build.
> >> >>>>>
> >> >>>>>     There is a Jira for protobuf upgrade[3] created even
before
> >> >> shade
> >> >>>>> support was added to Hadoop. Now need to revisit the Jira
and
> >> >> continue
> >> >>>>> explore possibilities.
> >> >>>>>
> >> >>>>> 2. leveldbjni:
> >> >>>>> ---------------
> >> >>>>>    Current leveldbjni used in YARN doesnot support ARM
> architecture,
> >> >>>> need
> >> >>>>> to check whether any of the future versions support ARM
and can
> >> >> hadoop
> >> >>>>> upgrade to that version.
> >> >>>>>
> >> >>>>>
> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
> >> >>>>> -------------------------
> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable
by
> >> >>> default
> >> >>>> in
> >> >>>>> the maven repository. Workaround is to build it locally
and keep
> in
> >> >>> local
> >> >>>>> maven repository.
> >> >>>>> Need to check whether any future versions of
> 'protoc-gen-grpc-java'
> >> >> is
> >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade
it?
> >> >>>>>
> >> >>>>>
> >> >>>>> Once the compilation issues are solved, then there might
be many
> >> >> native
> >> >>>>> code related issues due to different architectures.
> >> >>>>> So to explore everything, need to join hands together and
proceed.
> >> >>>>>
> >> >>>>>
> >> >>>>> Let us discuss and check, whether any body else out there
who also
> >> >> need
> >> >>>> the
> >> >>>>> support of Hadoop on ARM architectures and ready to lend
their
> hands
> >> >>> and
> >> >>>>> time in this work.
> >> >>>>>
> >> >>>>>
> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
> >> >>>>> [2]
> >> >>
> >>
> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
> >> >>>>>
> >> >>>>> -Vinay
> >> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message