infra-hadoop-migrations mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zoltan Haindrich <k...@rxd.hu>
Subject Re: HIVE building on ARM
Date Thu, 18 Jun 2020 10:05:02 GMT
Hey Chinna!

On 6/18/20 11:43 AM, Chinna Rao Lalam wrote:
> As you said, migrating this job to the new ci-hadoop instance looks good as
> Hadoop also shares the same armN slaves.

Sounds great!

> I am able to login the new ci-hadoop instance with Apache LDAP credentials,
> but i am not able to see the job creation option. Should I request access
> or the process for creation of a job is different than jenkin?.
> Please guide me to create the new job in the ci-hadoop instance. I will
> migrate this job after connecting the armN slaves to the new system.


I've also logged in - and apparently I've create job rights; I'm happy to help, but the best
would be to self-service yourselft :)
I think you may miss the "hudson-jobadmin" privilege.
Probably Gavin (or someone on the infra team) could help you with that..
to talk to them quickly - you can reach them on the #asfinfra channel (on the asf-slack).

The migration effort is coordinated thru the hadoop-migrations mailing list (I've cc-ed that
list)
you may want to subscribe to it by sending a mail to: hadoop-migrations-subscribe@infra.apache.org

cheers,
Zoltan



> 
> Thanks
> Chinna
> 
> On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng <zhengzhenyulixi@gmail.com>
> wrote:
> 
>> Hi Zoltan,
>>
>> Thanks alot for the information, so looks like one possible solution is as
>> you suggest, move the current ARM2 and ARM3 (those two were donate to
>> builds.apache.org by us) to the new ci-hadoop cluster and set up the jobs
>> just as what has been done in current jenkins.
>>
>> I will also ask our team member works on other projects to find out what
>> the status of other projects is.
>>
>> BR,
>>
>> On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich <kirk@rxd.hu> wrote:
>>
>>> Hey,
>>>
>>> There is an effort by the Apache Infra to change the way Jenkins stuff is
>>> organized; a couple months ago Gavin wrote an email about it:
>>>
>>> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3CCAN0Gg1dOdEPzatJz9bOFE-2veR7QG7H0HMvYjmsLdgjr8_RPew@mail.gmail.com%3E
>>> The resources for running these jobs are coming from the H0~H21 slaves
>>> which will be migrated to the new jenkins master eventually.
>>>
>>>   >> So please
>>>   >> suggest a way which direction we can move and can you share some
>>> details
>>>   >> about the new ci-hadoop instance.
>>>
>>> Since Hadoop testing is also happening on ARM - I think the best would be
>>> to also migrate the armN slaves and the Hive arm nightly over to the new
>>> ci-hadoop instance.
>>>
>>> On 6/16/20 8:40 AM, Zhenyu Zheng wrote:
>>>> Thanks for the info, I wonder if where does the resource of ci-hadoop
>>> and
>>>> hive-test-kube come from? Do they include ARM resources?
>>>
>>> Interesting question; the resources for Hive testing are donated by
>>> Cloudera.
>>> About the ARM workers I think Chinna could provide more details.
>>> ...I've no idea don't know who sponsors the Hxx slaves
>>>
>>>> Can you provide some more information about how the new hive-test-kube
>>> is
>>>> running?
>>> It's basically a Jenkins instance which is using kubernetes pods to run
>>> things.
>>> The whole thing is running on a GKE cluster.
>>> While I was working on it I collected stuff needed for it in this repo:
>>> https://github.com/kgyrtkirk/hive-test-kube/
>>> it should be possible to start a new deployment using that stuff
>>>
>>> cheers,
>>> Zoltan
>>>
>>>>
>>>> BR,
>>>> Kevin Zheng
>>>>
>>>> On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam <
>>>> lalamchinnarao13@gmail.com> wrote:
>>>>
>>>>> Hi Zoltan,
>>>>>
>>>>> Thanks for the update.
>>>>>
>>>>> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is
>>>>> targeting to run hive tests daily on "arm" slaves, it is using 2 arm
>>>>> slaves.
>>>>> To find any potential issues with "arm" and fix the issues. So please
>>>>> suggest a way which direction we can move and can you share some
>>> details
>>>>> about the new ci-hadoop instance.
>>>>>
>>>>> Thanks,
>>>>> Chinna
>>>>>
>>>>> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <kirk@rxd.hu>
wrote:
>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> In an ticket (INFRA-20416) Gavin asked me if we are completely off
>>>>>> builds.apache.org - when I went over the jobs I've saw that
>>>>>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there
>>>>>> once a day.
>>>>>>
>>>>>> Since builds.apache.org will be shut down in sometime in the future
>>> - we
>>>>>> should move this job to the new ci-hadoop instance or to
>>> hive-test-kube.
>>>>>> The key feature of the job is that it runs the test on the "armX"
>>> slaves;
>>>>>> which are statically configured on b.a.o.
>>>>>> Not sure which way to go - but we will have to move in some direction.
>>>>>>
>>>>>> cheers,
>>>>>> Zoltan
>>>>>>
>>>>>>
>>>>>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote:
>>>>>>> Hi Chinna,
>>>>>>>
>>>>>>> Thanks alot for the reply, I uploaded a patch and also a github
PR
>>> for
>>>>>>> https://issues.apache.org/jira/browse/HIVE-21939 .
>>>>>>> In the patch, I bumped the protobuf used in standalone-metadata
to
>>> 2.6.1
>>>>>>> and added a new profile, this profile will identify
>>>>>>> the hardware architecture and if it is Aarch64, it will override
the
>>>>>>> protobuf group.id and package to com.github.os72 which
>>>>>>> includes ARM support. For X86 platform, Hive will still download
the
>>>>>>> protobuf packages from org.google repo. I think with
>>>>>>> this method, we can keep the influence to existing x86 users
to the
>>>>>>> minimum. I hope this could be a acceptable short-term
>>>>>>> solution.
>>>>>>>
>>>>>>> I've manually tested on my machine and the github PR travis CI
test
>>> has
>>>>>>> already passed, so the build process is OK, so let's
>>>>>>> wait for the full test result from builds.apache.org.
>>>>>>>
>>>>>>> BR,
>>>>>>>
>>>>>>> Zhenyu
>>>>>>>
>>>>>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam <
>>>>>> lalamchinnarao13@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Zhenyu,
>>>>>>>>
>>>>>>>> Until HBase dependency resolved, without effecting the existing
code
>>>>>> on X86
>>>>>>>> i suggest create a separate profile with "os72" repo.
>>>>>>>>
>>>>>>>> Down the line we should have common version for both X86
and ARM.
>>>>>>>>
>>>>>>>> Hope It Helps,
>>>>>>>> Chinna
>>>>>>>>
>>>>>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <
>>>>>> zhengzhenyulixi@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Chinna, David and others might interested,
>>>>>>>>>
>>>>>>>>> Thanks for bring this up, we are currently working on
improving
>>>>>> enabling
>>>>>>>>> big-data software on the ARM platform,
>>>>>>>>> we have already done fixes and providing CIs to some
of the
>>> well-know
>>>>>>>>> projects like:
>>>>>>>>> 1. Hadoop:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>>>>>>>>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/
>>>>>>>>> 3. HBase:
>>>>>>>>>
>>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/
>>>>>>>>>
>>>>>>>>> And we are now working on projects including Hive, Kudu,
etc.
>>>>>>>>>
>>>>>>>>> Regarding to the protobuf upgrades in Hive, except upgrading
to 3.x
>>>>>> and
>>>>>>>>> break dependency for HBase, there can
>>>>>>>>> be some possible short-term plan(or walk-arounds), doing
thes can
>>> make
>>>>>>>> Hive
>>>>>>>>> work on ARM without break any
>>>>>>>>> dependencies, and then we can interact with Hbase project
to see
>>> how
>>>>>> can
>>>>>>>> we
>>>>>>>>> both upgrade to 3.x(since this
>>>>>>>>> make take some time).
>>>>>>>>>
>>>>>>>>> Those possible solutions can be:
>>>>>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support
>>>>>>>>> from org.openlabtesting repo, some projects(HBase did
>>>>>>>>> this: https://github.com/apache/hbase/pull/959, and we
will add a
>>>>>>>> profile
>>>>>>>>> for this, it will detact the host arch and
>>>>>>>>> will only use this pre-patched protobuf when the host
arch is ARM
>>> so
>>>>>> that
>>>>>>>>> nothing will be affected for existing x86
>>>>>>>>> users; Spark and Hadoop used some other package from
this repo as
>>> they
>>>>>>>> have
>>>>>>>>> already upgraded protobuf).
>>>>>>>>>
>>>>>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's
repo:
>>>>>>>>>
>>>>>>
>>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3
>>>>>>>>> Hive already used ``protoc-jar-maven-plugin`` from this
repo:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472
>>>>>>>>> so it is a minor update.
>>>>>>>>>
>>>>>>>>> I've tested both ways, worked OK on the ARM machine.
>>>>>>>>>
>>>>>>>>> So my ideas will be that we make this a two-step-work,
first step
>>> we
>>>>>> use
>>>>>>>>> one of the short-term plan to enable
>>>>>>>>> Hive on ARM, and 2nd Step, we look for the long run to
discuss with
>>>>>> HBase
>>>>>>>>> to find out a fesiable solution.
>>>>>>>>>
>>>>>>>>> BR,
>>>>>>>>>
>>>>>>>>> Zhenyu Zheng
>>>>>>>>>
>>>>>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam <
>>>>>>>>> lalamchinnarao13@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks David for the quick reply.
>>>>>>>>>> Yes currently HBase protobuf version mismatch is
the issue.
>>>>>>>>>>
>>>>>>>>>> We are investigating any common version of protobuf
is there to
>>>>>> support
>>>>>>>>> for
>>>>>>>>>> HBase an HIVE.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati
>>>>>>>>> <dlavati@cloudera.com.invalid
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I recently investigated this in
>>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359
and it didn't
>>> look
>>>>>>>>>>> feasible, as protobuf has to be in sync across
the whole stack,
>>> but
>>>>>>>> the
>>>>>>>>>>> external protobuf version for hbase is on 2.5.0
for hbase-2.x,
>>> see
>>>>>> my
>>>>>>>>>>> comment on the jira and
>>>>>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414.
>>>>>>>>>>>
>>>>>>>>>>> Kind Regards,
>>>>>>>>>>> David
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam
<
>>>>>>>>>>> lalamchinnarao13@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi All,
>>>>>>>>>>>>
>>>>>>>>>>>> We are working on HIVE building and running
on ARM
>>> architecture. As
>>>>>>>>>> part
>>>>>>>>>>> of
>>>>>>>>>>>> this i want to add one periodic HIVE JENKINS
JOB on ARM
>>> machines.
>>>>>>>>> Which
>>>>>>>>>>>> helps to identity gaps for ARM, If any issues
identified we will
>>>>>>>> keep
>>>>>>>>>> on
>>>>>>>>>>>> working to fix those issues.
>>>>>>>>>>>>
>>>>>>>>>>>> Currently to run HIVE on aarch64, we need
to upgrade to
>>>>>>>>> protobuf-3.7.1.
>>>>>>>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939>
>>>>>>>>> upgraded
>>>>>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have
mainly 2 failures. We
>>> are
>>>>>>>>>>> working
>>>>>>>>>>>> on this.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Chinna Rao Lalam
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Hope It Helps,
>>>>>>>>>> Chinna
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Hope It Helps,
>>>>>>>> Chinna
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Hope It Helps,
>>>>> Chinna
>>>>>
>>>>
>>>
>>
> 

Mime
View raw message