hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinayakumar B <vinayakum...@apache.org>
Subject Re: [DISCUSS] ARM/aarch64 support for Hadoop
Date Thu, 05 Sep 2019 10:18:52 GMT
Hi all,

Thanks for the response.
As I see, protobuf upgrade is long pending and most awaited one.

@Sunil
Protobuf upgrade looks to be a non-trivial task.
Thanks @Duo Zhang for the suggestion of
'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the problem
of dependency on build environment.
However more problem lies in upgrade protobuf without breaking the
downstream builds.
Reason is many downstream projects directly refers server specific jars and
they expect protobuf-2.5.0 jar to get added to classpath by transitive
dependency.

There are some historical discussions and suggestions on
https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
upgrade.
Recommended path for solution is, try to upgrade protobuf using shading of
latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
dependencies for downstreams.
I am trying out ideas on this and if it gets completed within time, may be
we can target trunk itself for the protobuf upgrade.

separate branch idea is suggested for the overall ARM support including
protobuf upgrade and other problems mentioned in the discussion above.

I dont expect separate branch to have a huge changes, apart from bug fixes,
since there are no separate features specific to ARM is being planned.
So timely rebase of separate branch would reduce the overhead on branch
review/merge task.

Still, if the solution to protobuf upgrade winds up early, without any side
effects, I am more than happy to land it in trunk itself.

Thanks,
Vinay
On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <sunilg@apache.org> wrote:

> Thanks Vinay for starting the thread.
>
> I agree to Anu's view point related to protobuf. And with the suggestion
> pointed out by Duo Zhang, if we can make use
> of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0
> of protobuf will also be more easier.
>
> However i think its better to do this effort in trunk itself.
> In offline talks, few members were interested to start 3.3.0 release. And
> given that happens soon, I feel its better
> we do this task in trunk itself as branch diverge is very much possible.
> And to bring to call a merge on such a big branch will be even more tough
> task.
>
> my 2 cents.
>
> Thanks
> Sunil
>
> On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <palomino219@gmail.com>
> wrote:
>
>> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to
>> generate
>> the protobuf code. It will download the protoc binary from the maven
>> central so we do not need to install protoc on the build machine any more.
>>
>> Zhenyu Zheng <zhengzhenyulixi@gmail.com> 于2019年9月4日周三 下午5:27写道:
>>
>> > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling
>> for
>> > over 2 month related to the Protobuf problem .
>> > According to the latest successful build log:
>> >
>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull
>> > the
>> > os version was ubuntu 14.04 and for the jobs that are failling now such
>> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console,
>> > the os version is 18.04. I'm not very familiar with the version changing
>> > for the jobs but I did a little search, according to:
>> >
>> >
>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names
>> > &
>> >
>> >
>> https://packages.ubuntu.com/search?suite=default&section=all&arch=any&keywords=libprotoc-dev&searchon=names
>> > it both said that the version of libprotc-dev and protobuf-compiler
>> > available for ubuntu 18.04 is 3.0.0
>> >
>> >
>> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ayushtkn@gmail.com> wrote:
>> >
>> >> Thanx Vinay for the initiative, Makes sense to add support for
>> different
>> >> architectures.
>> >>
>> >> +1, for the branch idea.
>> >> Good Luck!!!
>> >>
>> >> -Ayush
>> >>
>> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <palomino219@gmail.com>
>> >> wrote:
>> >> >
>> >> > For HBase, we purged all the protobuf related things from the public
>> >> API,
>> >> > and then upgraded to a shaded and relocated version of protobuf. We
>> have
>> >> > created a repo for this:
>> >> >
>> >> > https://github.com/apache/hbase-thirdparty
>> >> >
>> >> > But since the hadoop dependencies still pull in the protobuf 2.5
>> jars,
>> >> our
>> >> > coprocessors are still on protobuf 2.5. Recently we have opened a
>> >> discuss
>> >> > on how to deal with the upgrading of coprocessor. Glad to see that
>> the
>> >> > hadoop community is also willing to solve the problem.
>> >> >
>> >> > Anu Engineer <aengineer@cloudera.com.invalid> 于2019年9月3日周二
上午1:23写道:
>> >> >
>> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
>> that
>> >> >> Hadoop and the downstream projects work correctly after you upgrade
>> >> core
>> >> >> components like Protobuf.
>> >> >> So while branching and working on a branch is easy, merging back
>> after
>> >> you
>> >> >> upgrade some of these core components is insanely hard. You might
>> want
>> >> to
>> >> >> make sure that community buys into upgrading these components in
the
>> >> trunk.
>> >> >> That way we will get testing and downstream components will notice
>> when
>> >> >> things break.
>> >> >>
>> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
>> long
>> >> >> time; I have argued that 2.5 is out of support and we cannot stay
on
>> >> that
>> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>> code
>> >> base.
>> >> >> It has been rightly pointed to me that while all the arguments
I
>> make
>> >> is
>> >> >> correct; it is a very complicated task to upgrade Protobuf, and
the
>> >> worst
>> >> >> part is we will not even know what breaks until downstream projects
>> >> pick up
>> >> >> these changes and work against us.
>> >> >>
>> >> >> If we work off the Hadoop version 3 — and assume that we have
>> >> "shading" in
>> >> >> place for all deployments; it might be possible to get there; still
>> a
>> >> >> daunting task.
>> >> >>
>> >> >> So best of luck with the branch approach — But please remember,
>> Merging
>> >> >> back will be hard, Just my 2 cents.
>> >> >>
>> >> >> — Anu
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>> zhengzhenyulixi@gmail.com
>> >> >
>> >> >> wrote:
>> >> >>
>> >> >>> Hi,
>> >> >>>
>> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea.
A
>> >> separate
>> >> >>> branch with it's own ARM CI seems a really good idea.
>> >> >>> By doing this we won't break any of the undergoing development
in
>> >> trunk
>> >> >> and
>> >> >>> a CI can be a very good way to show what are the
>> >> >>> current problems and what have been fixed, it will also provide
a
>> very
>> >> >> good
>> >> >>> view for contributors that are intrested to working on
>> >> >>> this. We can finally merge back the branch to trunk until the
>> >> community
>> >> >>> thinks it is good enough and stable enough. We can donate
>> >> >>> ARM machines to the existing CI system for the job.
>> >> >>>
>> >> >>> I wonder if this approch possible?
>> >> >>>
>> >> >>> BR,
>> >> >>>
>> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>> liusheng2048@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> Hi,
>> >> >>>>
>> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>> community
>> >> >>>> mentioned by Vinay. I am working on building and
>> >> >>>> testing Hadoop components on aarch64 server these days,
besides
>> the
>> >> >>> missing
>> >> >>>> dependices of ARM platform issues #1 #2 #3
>> >> >>>> mentioned by Vinay, other similar issue has also be found,
such as
>> >> the
>> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >> >>>>
>> >> >>>> To promote the ARM support for Hadoop, we have discussed
and
>> hoped to
>> >> >> add
>> >> >>>> an ARM specific CI to Hadoop repo. we are not
>> >> >>>> sure about if there is any potential effect or confilict
on the
>> trunk
>> >> >>>> branch, so maybe creating a ARM specific branch for doing
these
>> stuff
>> >> >>>> is a better choice, what do you think?
>> >> >>>>
>> >> >>>> Hope to hear thoughts from you :)
>> >> >>>>
>> >> >>>> BR,
>> >> >>>> Liu sheng
>> >> >>>>
>> >> >>>> Vinayakumar B <vinayakumarb@apache.org> 于2019年8月27日周二
上午5:34写道:
>> >> >>>>
>> >> >>>>> Hi Folks,
>> >> >>>>>
>> >> >>>>> ARM is becoming famous lately in its processing capability
and
>> has
>> >> >> got
>> >> >>>> the
>> >> >>>>> potential to run Bigdata workloads.
>> >> >>>>> Many users have been moving to ARM machines due to
its low cost.
>> >> >>>>>
>> >> >>>>> In the past there were attempts to compile Hadoop on
ARM
>> (Rasberry
>> >> >> PI)
>> >> >>>> for
>> >> >>>>> experimental purposes. Today ARM architecture is taking
some of
>> the
>> >> >>>>> serverside processing as well. So there will be/is
a real need of
>> >> >>> Hadoop
>> >> >>>> to
>> >> >>>>> support ARM architecture as well.
>> >> >>>>>
>> >> >>>>> There are bunch of users who are trying out building
Hadoop on
>> ARM,
>> >> >>>> trying
>> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also
some
>> >> >>>>>
>> >> >>>>> As of today, Hadoop does not compile on ARM due to
below issues,
>> >> >> found
>> >> >>>> from
>> >> >>>>> testing done in openlab in [2].
>> >> >>>>>
>> >> >>>>> 1. Protobuf :
>> >> >>>>> -------------------
>> >> >>>>>     Hadoop project (also some downstream projects)
stuck to
>> protobuf
>> >> >>>> 2.5.0
>> >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0
is
>> >> not
>> >> >>>> being
>> >> >>>>> maintained in the community. While protobuf 3.x is
being actively
>> >> >>> adopted
>> >> >>>>> widely, still protobuf 3.x provides wire compatibility
for proto2
>> >> >>>> messages.
>> >> >>>>> Due to some compilation issues in the generated java
code, which
>> can
>> >> >>>> induce
>> >> >>>>> problems in downstream. Due to this reason protobuf
upgrade from
>> >> >> 2.5.0
>> >> >>>> was
>> >> >>>>> not taken up.
>> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries
to avoid
>> >> >>> classpath
>> >> >>>>> problem in downstream projects.
>> >> >>>>>    There are patches available to fix compilation in
Hadoop. But
>> >> >> need
>> >> >>> to
>> >> >>>>> find a way to upgrade protobuf to latest version and
still
>> maintain
>> >> >> the
>> >> >>>>> downstream's classpath using shading feature of Hadoop
build.
>> >> >>>>>
>> >> >>>>>     There is a Jira for protobuf upgrade[3] created
even before
>> >> >> shade
>> >> >>>>> support was added to Hadoop. Now need to revisit the
Jira and
>> >> >> continue
>> >> >>>>> explore possibilities.
>> >> >>>>>
>> >> >>>>> 2. leveldbjni:
>> >> >>>>> ---------------
>> >> >>>>>    Current leveldbjni used in YARN doesnot support
ARM
>> architecture,
>> >> >>>> need
>> >> >>>>> to check whether any of the future versions support
ARM and can
>> >> >> hadoop
>> >> >>>>> upgrade to that version.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
>> >> >>>>> -------------------------
>> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM
executable by
>> >> >>> default
>> >> >>>> in
>> >> >>>>> the maven repository. Workaround is to build it locally
and keep
>> in
>> >> >>> local
>> >> >>>>> maven repository.
>> >> >>>>> Need to check whether any future versions of
>> 'protoc-gen-grpc-java'
>> >> >> is
>> >> >>>>> having ARM executable and whether hadoop-yarn-csi can
upgrade it?
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Once the compilation issues are solved, then there
might be many
>> >> >> native
>> >> >>>>> code related issues due to different architectures.
>> >> >>>>> So to explore everything, need to join hands together
and
>> proceed.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Let us discuss and check, whether any body else out
there who
>> also
>> >> >> need
>> >> >>>> the
>> >> >>>>> support of Hadoop on ARM architectures and ready to
lend their
>> hands
>> >> >>> and
>> >> >>>>> time in this work.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358
>> >> >>>>> [2]
>> >> >>
>> >>
>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
>> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363
>> >> >>>>>
>> >> >>>>> -Vinay
>> >> >>
>> >>
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message