From hadoop-migrations-return-20-archive-asf-public=cust-asf.ponee.io@infra.apache.org Thu Jun 18 10:05:07 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3FA3218065B for ; Thu, 18 Jun 2020 12:05:07 +0200 (CEST) Received: (qmail 40735 invoked by uid 500); 18 Jun 2020 10:05:06 -0000 Mailing-List: contact hadoop-migrations-help@infra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-migrations@infra.apache.org Delivered-To: mailing list hadoop-migrations@infra.apache.org Received: (qmail 40708 invoked by uid 99); 18 Jun 2020 10:05:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jun 2020 10:05:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 9A04FC09BB for ; Thu, 18 Jun 2020 10:05:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.011 X-Spam-Level: X-Spam-Status: No, score=0.011 tagged_above=-999 required=6.31 tests=[KAM_DMARC_STATUS=0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id K2YlH24b0uGL for ; Thu, 18 Jun 2020 10:05:03 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=94.199.53.98; helo=matrix.larten.hu; envelope-from=kirk@rxd.hu; receiver= Received: from matrix.larten.hu (matrix.larten.hu [94.199.53.98]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTP id 0A4DC7D3FB for ; Thu, 18 Jun 2020 10:05:02 +0000 (UTC) Received: from [192.168.128.70] (89-147-69-235.pool.digikabel.hu [89.147.69.235]) by matrix.larten.hu (Postfix) with ESMTPSA id A4E524BE25; Thu, 18 Jun 2020 12:05:02 +0200 (CEST) Subject: Re: HIVE building on ARM To: dev@hive.apache.org, Chinna Rao Lalam , hadoop-migrations@infra.apache.org References: <0da0caee-9c1d-23a4-44ca-ea7d5a89c8fb@rxd.hu> <8e290e49-41e8-32eb-e168-16d4e1888c18@rxd.hu> From: Zoltan Haindrich Message-ID: Date: Thu, 18 Jun 2020 12:05:02 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Hey Chinna! On 6/18/20 11:43 AM, Chinna Rao Lalam wrote: > As you said, migrating this job to the new ci-hadoop instance looks good as > Hadoop also shares the same armN slaves. Sounds great! > I am able to login the new ci-hadoop instance with Apache LDAP credentials, > but i am not able to see the job creation option. Should I request access > or the process for creation of a job is different than jenkin?. > Please guide me to create the new job in the ci-hadoop instance. I will > migrate this job after connecting the armN slaves to the new system. I've also logged in - and apparently I've create job rights; I'm happy to help, but the best would be to self-service yourselft :) I think you may miss the "hudson-jobadmin" privilege. Probably Gavin (or someone on the infra team) could help you with that.. to talk to them quickly - you can reach them on the #asfinfra channel (on the asf-slack). The migration effort is coordinated thru the hadoop-migrations mailing list (I've cc-ed that list) you may want to subscribe to it by sending a mail to: hadoop-migrations-subscribe@infra.apache.org cheers, Zoltan > > Thanks > Chinna > > On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng > wrote: > >> Hi Zoltan, >> >> Thanks alot for the information, so looks like one possible solution is as >> you suggest, move the current ARM2 and ARM3 (those two were donate to >> builds.apache.org by us) to the new ci-hadoop cluster and set up the jobs >> just as what has been done in current jenkins. >> >> I will also ask our team member works on other projects to find out what >> the status of other projects is. >> >> BR, >> >> On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich wrote: >> >>> Hey, >>> >>> There is an effort by the Apache Infra to change the way Jenkins stuff is >>> organized; a couple months ago Gavin wrote an email about it: >>> >>> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3CCAN0Gg1dOdEPzatJz9bOFE-2veR7QG7H0HMvYjmsLdgjr8_RPew@mail.gmail.com%3E >>> The resources for running these jobs are coming from the H0~H21 slaves >>> which will be migrated to the new jenkins master eventually. >>> >>> >> So please >>> >> suggest a way which direction we can move and can you share some >>> details >>> >> about the new ci-hadoop instance. >>> >>> Since Hadoop testing is also happening on ARM - I think the best would be >>> to also migrate the armN slaves and the Hive arm nightly over to the new >>> ci-hadoop instance. >>> >>> On 6/16/20 8:40 AM, Zhenyu Zheng wrote: >>>> Thanks for the info, I wonder if where does the resource of ci-hadoop >>> and >>>> hive-test-kube come from? Do they include ARM resources? >>> >>> Interesting question; the resources for Hive testing are donated by >>> Cloudera. >>> About the ARM workers I think Chinna could provide more details. >>> ...I've no idea don't know who sponsors the Hxx slaves >>> >>>> Can you provide some more information about how the new hive-test-kube >>> is >>>> running? >>> It's basically a Jenkins instance which is using kubernetes pods to run >>> things. >>> The whole thing is running on a GKE cluster. >>> While I was working on it I collected stuff needed for it in this repo: >>> https://github.com/kgyrtkirk/hive-test-kube/ >>> it should be possible to start a new deployment using that stuff >>> >>> cheers, >>> Zoltan >>> >>>> >>>> BR, >>>> Kevin Zheng >>>> >>>> On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam < >>>> lalamchinnarao13@gmail.com> wrote: >>>> >>>>> Hi Zoltan, >>>>> >>>>> Thanks for the update. >>>>> >>>>> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is >>>>> targeting to run hive tests daily on "arm" slaves, it is using 2 arm >>>>> slaves. >>>>> To find any potential issues with "arm" and fix the issues. So please >>>>> suggest a way which direction we can move and can you share some >>> details >>>>> about the new ci-hadoop instance. >>>>> >>>>> Thanks, >>>>> Chinna >>>>> >>>>> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich wrote: >>>>> >>>>>> Hey all, >>>>>> >>>>>> In an ticket (INFRA-20416) Gavin asked me if we are completely off >>>>>> builds.apache.org - when I went over the jobs I've saw that >>>>>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there >>>>>> once a day. >>>>>> >>>>>> Since builds.apache.org will be shut down in sometime in the future >>> - we >>>>>> should move this job to the new ci-hadoop instance or to >>> hive-test-kube. >>>>>> The key feature of the job is that it runs the test on the "armX" >>> slaves; >>>>>> which are statically configured on b.a.o. >>>>>> Not sure which way to go - but we will have to move in some direction. >>>>>> >>>>>> cheers, >>>>>> Zoltan >>>>>> >>>>>> >>>>>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote: >>>>>>> Hi Chinna, >>>>>>> >>>>>>> Thanks alot for the reply, I uploaded a patch and also a github PR >>> for >>>>>>> https://issues.apache.org/jira/browse/HIVE-21939 . >>>>>>> In the patch, I bumped the protobuf used in standalone-metadata to >>> 2.6.1 >>>>>>> and added a new profile, this profile will identify >>>>>>> the hardware architecture and if it is Aarch64, it will override the >>>>>>> protobuf group.id and package to com.github.os72 which >>>>>>> includes ARM support. For X86 platform, Hive will still download the >>>>>>> protobuf packages from org.google repo. I think with >>>>>>> this method, we can keep the influence to existing x86 users to the >>>>>>> minimum. I hope this could be a acceptable short-term >>>>>>> solution. >>>>>>> >>>>>>> I've manually tested on my machine and the github PR travis CI test >>> has >>>>>>> already passed, so the build process is OK, so let's >>>>>>> wait for the full test result from builds.apache.org. >>>>>>> >>>>>>> BR, >>>>>>> >>>>>>> Zhenyu >>>>>>> >>>>>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam < >>>>>> lalamchinnarao13@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Zhenyu, >>>>>>>> >>>>>>>> Until HBase dependency resolved, without effecting the existing code >>>>>> on X86 >>>>>>>> i suggest create a separate profile with "os72" repo. >>>>>>>> >>>>>>>> Down the line we should have common version for both X86 and ARM. >>>>>>>> >>>>>>>> Hope It Helps, >>>>>>>> Chinna >>>>>>>> >>>>>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng < >>>>>> zhengzhenyulixi@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Chinna, David and others might interested, >>>>>>>>> >>>>>>>>> Thanks for bring this up, we are currently working on improving >>>>>> enabling >>>>>>>>> big-data software on the ARM platform, >>>>>>>>> we have already done fixes and providing CIs to some of the >>> well-know >>>>>>>>> projects like: >>>>>>>>> 1. Hadoop: >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ >>>>>>>>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ >>>>>>>>> 3. HBase: >>>>>>>>> >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/ >>>>>>>>> >>>>>>>>> And we are now working on projects including Hive, Kudu, etc. >>>>>>>>> >>>>>>>>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x >>>>>> and >>>>>>>>> break dependency for HBase, there can >>>>>>>>> be some possible short-term plan(or walk-arounds), doing thes can >>> make >>>>>>>> Hive >>>>>>>>> work on ARM without break any >>>>>>>>> dependencies, and then we can interact with Hbase project to see >>> how >>>>>> can >>>>>>>> we >>>>>>>>> both upgrade to 3.x(since this >>>>>>>>> make take some time). >>>>>>>>> >>>>>>>>> Those possible solutions can be: >>>>>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support >>>>>>>>> from org.openlabtesting repo, some projects(HBase did >>>>>>>>> this: https://github.com/apache/hbase/pull/959, and we will add a >>>>>>>> profile >>>>>>>>> for this, it will detact the host arch and >>>>>>>>> will only use this pre-patched protobuf when the host arch is ARM >>> so >>>>>> that >>>>>>>>> nothing will be affected for existing x86 >>>>>>>>> users; Spark and Hadoop used some other package from this repo as >>> they >>>>>>>> have >>>>>>>>> already upgraded protobuf). >>>>>>>>> >>>>>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo: >>>>>>>>> >>>>>> >>> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3 >>>>>>>>> Hive already used ``protoc-jar-maven-plugin`` from this repo: >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472 >>>>>>>>> so it is a minor update. >>>>>>>>> >>>>>>>>> I've tested both ways, worked OK on the ARM machine. >>>>>>>>> >>>>>>>>> So my ideas will be that we make this a two-step-work, first step >>> we >>>>>> use >>>>>>>>> one of the short-term plan to enable >>>>>>>>> Hive on ARM, and 2nd Step, we look for the long run to discuss with >>>>>> HBase >>>>>>>>> to find out a fesiable solution. >>>>>>>>> >>>>>>>>> BR, >>>>>>>>> >>>>>>>>> Zhenyu Zheng >>>>>>>>> >>>>>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam < >>>>>>>>> lalamchinnarao13@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks David for the quick reply. >>>>>>>>>> Yes currently HBase protobuf version mismatch is the issue. >>>>>>>>>> >>>>>>>>>> We are investigating any common version of protobuf is there to >>>>>> support >>>>>>>>> for >>>>>>>>>> HBase an HIVE. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati >>>>>>>>> >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I recently investigated this in >>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't >>> look >>>>>>>>>>> feasible, as protobuf has to be in sync across the whole stack, >>> but >>>>>>>> the >>>>>>>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x, >>> see >>>>>> my >>>>>>>>>>> comment on the jira and >>>>>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414. >>>>>>>>>>> >>>>>>>>>>> Kind Regards, >>>>>>>>>>> David >>>>>>>>>>> >>>>>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam < >>>>>>>>>>> lalamchinnarao13@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi All, >>>>>>>>>>>> >>>>>>>>>>>> We are working on HIVE building and running on ARM >>> architecture. As >>>>>>>>>> part >>>>>>>>>>> of >>>>>>>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM >>> machines. >>>>>>>>> Which >>>>>>>>>>>> helps to identity gaps for ARM, If any issues identified we will >>>>>>>> keep >>>>>>>>>> on >>>>>>>>>>>> working to fix those issues. >>>>>>>>>>>> >>>>>>>>>>>> Currently to run HIVE on aarch64, we need to upgrade to >>>>>>>>> protobuf-3.7.1. >>>>>>>>>>>> HIVE-21939 >>>>>>>>> upgraded >>>>>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We >>> are >>>>>>>>>>> working >>>>>>>>>>>> on this. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Chinna Rao Lalam >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Hope It Helps, >>>>>>>>>> Chinna >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Hope It Helps, >>>>>>>> Chinna >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Hope It Helps, >>>>> Chinna >>>>> >>>> >>> >> >