Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <OF26BC1840.351BA404-ON00258013.001CD365-65258013.001CE6ED@LocalDomain>
Subject: Re: Contributions to Cloudera Impala
To: "dev@impala" <dev@impala.incubator.apache.org>,
        Jim Apple
 <jbapple@cloudera.com>
Cc: "Manish Patil" <patil2@us.ibm.com>, Silvius Rus <srus@cloudera.com>,
        "Valencia
 Serrao" <vserrao@us.ibm.com>,
        "Sudarshan Jagadale"
 <jagadale@us.ibm.com>,
        "Nishidha Panpaliya" <nishidha@us.ibm.com>,
        "Zhi Zhi NY
 Yang" <zhizhiy@cn.ibm.com>
From: "Nishidha Panpaliya" <nishidha@us.ibm.com>
Date: Fri, 19 Aug 2016 16:23:40 +0530
References: <OF96B7B863.5384BD9A-ON65258012.002E232A-65258012.0041005F@notes.na.collabserv.com>
 <CAC-pSX05WhRD8BqUpXGxUsYSdzMnztBbyZ3xW=dE+UPwC9i12A@mail.gmail.com>
 <OF26BC1840.351BA404-ON00258013.001CD365-65258013.001CE6ED@LocalDomain>
Content-type: multipart/related;
	Boundary="0__=EABB0A87DFA1E50D8f9e8a93df938690918cEABB0A87DFA1E50D"
Message-Id: <OF9AC72F56.CE1B481B-ON65258014.0032639D-65258014.003BD867@notes.na.collabserv.com>
archived-at: Fri, 19 Aug 2016 10:53:57 -0000

--0__=EABB0A87DFA1E50D8f9e8a93df938690918cEABB0A87DFA1E50D
Content-type: multipart/alternative;
	Boundary="1__=EABB0A87DFA1E50D8f9e8a93df938690918cEABB0A87DFA1E50D"


--1__=EABB0A87DFA1E50D8f9e8a93df938690918cEABB0A87DFA1E50D
Content-Transfer-Encoding: quoted-printable
Content-type: text/plain; charset=US-ASCII


Hi Jim,

Thanks a lot for your response.

Please see my comments inline.

Adding Zhi Zhi, our colleague from China in the thread.

Regards,
Nishidha


From:	Sudarshan Jagadale/Austin/Contr/IBM
To:	Jim Apple <jbapple@cloudera.com>
Cc:	"dev@impala" <dev@impala.incubator.apache.org>, Manish
            Patil/Austin/Contr/IBM@IBMUS, Silvius Rus <srus@cloudera.com>,
            Valencia Serrao/Austin/Contr/IBM@IBMUS, Nishidha
            Panpaliya/Austin/Contr/IBM@IBMUS
Date:	08/18/2016 10:45 AM
Subject:	Re: Contributions to Cloudera Impala


Hi Jim,

+Nishidha

Thank you for your inputs...

Thanks and Regards
Sudarshan Jagadale
Power Open Source Solutions


From:	Jim Apple <jbapple@cloudera.com>
To:	"dev@impala" <dev@impala.incubator.apache.org>
Cc:	Silvius Rus <srus@cloudera.com>, Sudarshan
            Jagadale/Austin/Contr/IBM@IBMUS, Manish
            Patil/Austin/Contr/IBM@IBMUS, Valencia
            Serrao/Austin/Contr/IBM@IBMUS
Date:	08/17/2016 09:43 PM
Subject:	Re: Contributions to Cloudera Impala


> I'm glad to tell you that we are able to build and test Impala on Ubuntu
> linux ppc64le with the great support from the Cloudera Community.

Excellent!

> Our next action is to upstream all our changes to Cloudera Impala.

Great!

Cloudera has donated Impala to the Apache Software Foundation (aka
"ASF"). Cloudera now contributes to the project, and the project is
managed by the Impala community.

> With
> this, our plan is to start building latest Impala on Power8 as we'd been
> porting quite an old version (code from cdh5-trunk branch till 23rd
March,
> 2016). Since then, I know there have been many many changes happened
which
> are yet to be ported, specially kudu stuff.

Yes, there have been many changes. One is that Impala is now hosted on
ASF-owned git. Please see
https://cwiki.apache.org/confluence/display/IMPALA/How+to+switch+to
+Apache-hosted+git
[Nishidha] I've read a few pages from this Confluence. Indeed, very useful
to start with.

>    We know we need CLA to be signed to start contributions. We have
already
>    initiated the process and hoping to get it done soon.

I think the right thing to do here is use the Apache CLAs. See

https://www.apache.org/licenses/cla-corporate.txt

and

https://www.apache.org/licenses/icla.txt
[Nishidha] We'll start with this.

>     By the time we get CLA signed, we would start porting the changes
done
>    in last 5 months. So, I wanted to know which tag/branch should we take
>    up for this.

This is a question we could all discuss together, and it might end up
being a decision made by the Project Management Committee (PMC).

This is a big question about how Apache Impala will evolve. Our bylaws
state:

"Significant, pervasive features may be developed in a speculative
branch of the repository. The PMC may grant commit rights on the
branch to its consistent contributors for the duration of the
initiative. Branch committers are responsible for shepherding their
feature into an active release and do not cast binding votes or vetoes
in the project."

So perhaps this should happen on a separate branch?

One question the community should also consider, IMHO, is whether the
community will have sufficient resources to maintain a working ppc64le
codebase indefinitely into the future.
[Nishidha] We found two new source code URLs as one mentioned in Confluence
https://git-wip-us.apache.org/repos/asf/incubator-impala.git and another to
be https://github.com/apache/incubator-impala.
Commits wise both look same, though former one says "wip" in the URL.
Please suggest the URL to be forked and worked upon. We didn't know if we
could directly work on a separate branch on apache's Impala. We thought of
forking first into our repo and then working on it.

> Working on cdh5-trunk will put us into an unending loop of
>    porting as it is being modified everyday. We are thinking to create a
>    branch from cdh5.8.0-release tag and start working on it. Please
suggest
>    us the best way to do this.

Since Impala is now developed on Apache infrastructure, we have
switched branching schemas. Our main branch is now "master". We do not
have any release branches yet.
[Nishidha] Okay. So, after CLA, can we work directly on Apache's Impala or
we'll need to fork it into our repository and create a new branch from
master, and then generate PRs/Gerrit code reviews from it?

>    Verifying all the changes on x86 platforms ourself here will also be
>    time consuming and add potential delays in upstreaming. So, we were
>    thinking if we can get a job on Cloudera's Continuous integration
server
>    which would simply fetch our branch and verify it on all the supported
>    platforms and do all the required checks. I'm not sure if this is
>    feasible but just a thought. Any other suggestions  to foster this
>    activity would be appreciated.

We are working on making a publicly-available CI setup, but we aren't done
yet.

Do you have a CI setup and x86-64 machines that your CI workers can run on?
[Nishidha] Which CI do you have or working on? We can setup Jenkins here
and can get x86-64 machines too. What is the expected timeline for your CI
to be publicly available?

>    For every Pull Request, what are the basic sanity tests required to be
>    ensured? Do you test all BE, FE, End-to-End tests, Custom cluster
tests?

Patches are sent to gerrit for review. Before they are merged, all
tests must pass in "core" (but not "exhaustive") mode.
[Nishidha] Sure.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

If I were in your shoes, I might take the following steps:

1. Start a discussion on dev@ about whether a new branch is the right
way to develop.

2. Work out long-term maintenance plans and commitments and CI plans

3. Do the arduous work of rebasing on a recent HEAD.
[Nishidha] Thanks for this suggestion. Yes, definitely, I would like to
follow this.
1. Would you suggest me to start a new thread for branch discussion,
although I created a gist in github.com
https://gist.github.com/npanpaliya/bd58e554370455babc5c4f290e4b1723 for the
same?
2. For long-term maintenance plans, commitments and CI plans, I would start
a separate thread once we get clarity on above.
3. We would definitely need to rebase on a recent HEAD to submit changes
upstream. (This would be again challenging as kudu, the newly added
dependency is even tougher to build on Power). We have also started looking
at building our native toolchain of Impala Pre-requisites for Power.


--1__=EABB0A87DFA1E50D8f9e8a93df938690918cEABB0A87DFA1E50D
Content-Transfer-Encoding: quoted-printable
Content-type: text/html; charset=US-ASCII
Content-Disposition: inline

<html><body><p>Hi Jim,<br><br>Thanks a lot for your response.<br><br>Please=
 see my comments inline.<br><br>Adding Zhi Zhi, our colleague from China in=
 the thread.<br><br>Regards,<br>Nishidha<br><br><img width=3D"16" height=3D=
"16" src=3D"cid:1=5F=5F=3DEABB0A87DFA1E50D8f9e8a93df938690918cEAB@" border=
=3D"0" alt=3D"Inactive hide details for Sudarshan Jagadale---08/18/2016 10:=
45:47 AM---Hi Jim,  +Nishidha"><font color=3D"#424282">Sudarshan Jagadale--=
-08/18/2016 10:45:47 AM---Hi Jim,  +Nishidha</font><br><br><font size=3D"2"=
 color=3D"#5F5F5F">From:        </font><font size=3D"2">Sudarshan Jagadale/=
Austin/Contr/IBM</font><br><font size=3D"2" color=3D"#5F5F5F">To:        </=
font><font size=3D"2">Jim Apple &lt;jbapple@cloudera.com&gt;</font><br><fon=
t size=3D"2" color=3D"#5F5F5F">Cc:        </font><font size=3D"2">&quot;dev=
@impala&quot; &lt;dev@impala.incubator.apache.org&gt;, Manish Patil/Austin/=
Contr/IBM@IBMUS, Silvius Rus &lt;srus@cloudera.com&gt;, Valencia Serrao/Aus=
tin/Contr/IBM@IBMUS, Nishidha Panpaliya/Austin/Contr/IBM@IBMUS</font><br><f=
ont size=3D"2" color=3D"#5F5F5F">Date:        </font><font size=3D"2">08/18=
/2016 10:45 AM</font><br><font size=3D"2" color=3D"#5F5F5F">Subject:       =
 </font><font size=3D"2">Re: Contributions to Cloudera Impala</font><br><hr=
 width=3D"100%" size=3D"2" align=3D"left" noshade style=3D"color:#8091A5; "=
><br><br>Hi Jim, <br><br>+Nishidha<br><br>Thank you for your inputs... <br>=
<br>Thanks and Regards<br>Sudarshan Jagadale <br>Power Open Source Solution=
s<br><br><br><img width=3D"16" height=3D"16" src=3D"cid:1=5F=5F=3DEABB0A87D=
FA1E50D8f9e8a93df938690918cEAB@" border=3D"0" alt=3D"Inactive hide details =
for Jim Apple ---08/17/2016 09:43:02 PM---&gt; I'm glad to tell you that we=
 are able to build and test Impal"><font color=3D"#424282">Jim Apple ---08/=
17/2016 09:43:02 PM---&gt; I'm glad to tell you that we are able to build a=
nd test Impala on Ubuntu &gt; linux ppc64le with the</font><br><br><font si=
ze=3D"2" color=3D"#5F5F5F">From:        </font><font size=3D"2">Jim Apple &=
lt;jbapple@cloudera.com&gt;</font><br><font size=3D"2" color=3D"#5F5F5F">To=
:        </font><font size=3D"2">&quot;dev@impala&quot; &lt;dev@impala.incu=
bator.apache.org&gt;</font><br><font size=3D"2" color=3D"#5F5F5F">Cc:      =
  </font><font size=3D"2">Silvius Rus &lt;srus@cloudera.com&gt;, Sudarshan =
Jagadale/Austin/Contr/IBM@IBMUS, Manish Patil/Austin/Contr/IBM@IBMUS, Valen=
cia Serrao/Austin/Contr/IBM@IBMUS</font><br><font size=3D"2" color=3D"#5F5F=
5F">Date:        </font><font size=3D"2">08/17/2016 09:43 PM</font><br><fon=
t size=3D"2" color=3D"#5F5F5F">Subject:        </font><font size=3D"2">Re: =
Contributions to Cloudera Impala</font><br><hr width=3D"100%" size=3D"2" al=
ign=3D"left" noshade style=3D"color:#8091A5; "><br><br><br><tt>&gt; I'm gla=
d to tell you that we are able to build and test Impala on Ubuntu<br>&gt; l=
inux ppc64le with the great support from the Cloudera Community.<br><br>Exc=
ellent!<br><br>&gt; Our next action is to upstream all our changes to Cloud=
era Impala.<br><br>Great!<br><br>Cloudera has donated Impala to the Apache =
Software Foundation (aka<br>&quot;ASF&quot;). Cloudera now contributes to t=
he project, and the project is<br>managed by the Impala community.<br><br>&=
gt; With<br>&gt; this, our plan is to start building latest Impala on Power=
8 as we'd been<br>&gt; porting quite an old version (code from cdh5-trunk b=
ranch till 23rd March,<br>&gt; 2016). Since then, I know there have been ma=
ny many changes happened which<br>&gt; are yet to be ported, specially kudu=
 stuff.<br><br>Yes, there have been many changes. One is that Impala is now=
 hosted on<br>ASF-owned git. Please see<br></tt><tt><a href=3D"https://cwik=
i.apache.org/confluence/display/IMPALA/How+to+switch+to+Apache-hosted+git">=
https://cwiki.apache.org/confluence/display/IMPALA/How+to+switch+to+Apache-=
hosted+git</a></tt><tt><br></tt><tt><font color=3D"#0000FF">[Nishidha] I've=
 read a few pages from this Confluence. Indeed, very useful to start with.<=
/font></tt><br><tt><br>&gt; &nbsp; &nbsp;We know we need CLA to be signed t=
o start contributions. We have already<br>&gt; &nbsp; &nbsp;initiated the p=
rocess and hoping to get it done soon.<br><br>I think the right thing to do=
 here is use the Apache CLAs. See<br><br></tt><tt><a href=3D"https://www.ap=
ache.org/licenses/cla-corporate.txt">https://www.apache.org/licenses/cla-co=
rporate.txt</a></tt><tt><br><br>and<br><br></tt><tt><a href=3D"https://www.=
apache.org/licenses/icla.txt">https://www.apache.org/licenses/icla.txt</a><=
/tt><tt><br></tt><tt><font color=3D"#0000FF">[Nishidha] We'll start with th=
is.</font></tt><br><tt><br>&gt; &nbsp; &nbsp; By the time we get CLA signed=
, we would start porting the changes done<br>&gt; &nbsp; &nbsp;in last 5 mo=
nths. So, I wanted to know which tag/branch should we take<br>&gt; &nbsp; &=
nbsp;up for this.<br><br>This is a question we could all discuss together, =
and it might end up<br>being a decision made by the Project Management Comm=
ittee (PMC).<br><br>This is a big question about how Apache Impala will evo=
lve. Our bylaws state:<br><br>&quot;Significant, pervasive features may be =
developed in a speculative<br>branch of the repository. The PMC may grant c=
ommit rights on the<br>branch to its consistent contributors for the durati=
on of the<br>initiative. Branch committers are responsible for shepherding =
their<br>feature into an active release and do not cast binding votes or ve=
toes<br>in the project.&quot;<br><br>So perhaps this should happen on a sep=
arate branch?<br><br>One question the community should also consider, IMHO,=
 is whether the<br>community will have sufficient resources to maintain a w=
orking ppc64le<br>codebase indefinitely into the future.<br></tt><tt><font =
color=3D"#0000FF">[Nishidha]</font></tt><tt>&nbsp;</tt><tt><font color=3D"#=
0000FF">We found two new source code URLs as one mentioned in Confluence </=
font></tt><font size=3D"2" face=3D"Lucida Console"><a href=3D"https://git-w=
ip-us.apache.org/repos/asf/incubator-impala.git">https://git-wip-us.apache.=
org/repos/asf/incubator-impala.git</a></font><tt>&nbsp;and another to be </=
tt><a href=3D"https://github.com/apache/incubator-impala"><tt>https://githu=
b.com/apache/incubator-impala</tt></a><tt>.</tt><br><tt><font color=3D"#000=
0FF">Commits wise both look same, though former one says &quot;wip&quot; in=
 the URL. Please suggest the URL to be forked and worked upon. We didn't kn=
ow if we could directly work on a separate branch on apache's Impala. We th=
ought of forking first into our repo and then working on it.<br></font></tt=
><tt><br>&gt; Working on cdh5-trunk will put us into an unending loop of<br=
>&gt; &nbsp; &nbsp;porting as it is being modified everyday. We are thinkin=
g to create a<br>&gt; &nbsp; &nbsp;branch from cdh5.8.0-release tag and sta=
rt working on it. Please suggest<br>&gt; &nbsp; &nbsp;us the best way to do=
 this.<br><br>Since Impala is now developed on Apache infrastructure, we ha=
ve<br>switched branching schemas. Our main branch is now &quot;master&quot;=
. We do not<br>have any release branches yet.<br></tt><tt><font color=3D"#0=
000FF">[Nishidha] Okay. So, after CLA, can we work directly on Apache's Imp=
ala or we'll need to fork it into our repository and create a new branch fr=
om master, and then generate PRs/Gerrit code reviews from it?</font></tt><b=
r><tt><br>&gt; &nbsp; &nbsp;Verifying all the changes on x86 platforms ours=
elf here will also be<br>&gt; &nbsp; &nbsp;time consuming and add potential=
 delays in upstreaming. So, we were<br>&gt; &nbsp; &nbsp;thinking if we can=
 get a job on Cloudera's Continuous integration server<br>&gt; &nbsp; &nbsp=
;which would simply fetch our branch and verify it on all the supported<br>=
&gt; &nbsp; &nbsp;platforms and do all the required checks. I'm not sure if=
 this is<br>&gt; &nbsp; &nbsp;feasible but just a thought. Any other sugges=
tions &nbsp;to foster this<br>&gt; &nbsp; &nbsp;activity would be appreciat=
ed.<br><br>We are working on making a publicly-available CI setup, but we a=
ren't done yet.<br><br>Do you have a CI setup and x86-64 machines that your=
 CI workers can run on?<br></tt><tt><font color=3D"#0000FF">[Nishidha] Whic=
h CI do you have or working on? We can setup Jenkins here and can get x86-6=
4 machines too. What is the expected timeline for your CI to be publicly av=
ailable?</font></tt><br><tt><br>&gt; &nbsp; &nbsp;For every Pull Request, w=
hat are the basic sanity tests required to be<br>&gt; &nbsp; &nbsp;ensured?=
 Do you test all BE, FE, End-to-End tests, Custom cluster tests?<br><br>Pat=
ches are sent to gerrit for review. Before they are merged, all<br>tests mu=
st pass in &quot;core&quot; (but not &quot;exhaustive&quot;) mode.<br></tt>=
<tt><font color=3D"#0000FF">[Nishidha] Sure.</font></tt><tt><br>=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D<br><br>If I were in your shoes, I might take the foll=
owing steps:<br><br>1. Start a discussion on dev@ about whether a new branc=
h is the right<br>way to develop.<br><br>2. Work out long-term maintenance =
plans and commitments and CI plans<br><br>3. Do the arduous work of rebasin=
g on a recent HEAD.<br></tt><tt><font color=3D"#001FE2">[Nishidha]</font></=
tt><tt><font color=3D"#0000FF">&nbsp;Thanks for this suggestion. Yes, defin=
itely, I would like to follow this. </font></tt><br><tt><font color=3D"#000=
0FF">1. Would you suggest me to start a new thread for branch discussion, a=
lthough I created a gist in </font></tt><a href=3D"https://gist.github.com/=
npanpaliya/bd58e554370455babc5c4f290e4b1723"><tt><font color=3D"#0000FF">gi=
thub.com</font></tt></a><tt><font color=3D"#0000FF">&nbsp;</font></tt><a hr=
ef=3D"https://gist.github.com/npanpaliya/bd58e554370455babc5c4f290e4b1723">=
<tt>https://gist.github.com/npanpaliya/bd58e554370455babc5c4f290e4b1723</tt=
></a><tt><font color=3D"#0000FF">&nbsp;</font></tt><tt><font color=3D"#0000=
FF">for the same? </font></tt><br><tt><font color=3D"#0000FF">2. For long-t=
erm maintenance plans, commitments and CI plans, I would start a separate t=
hread once we get clarity on above.</font></tt><br><tt><font color=3D"#0000=
FF">3. We would definitely need to rebase on a recent HEAD to submit change=
s upstream. (This would be again challenging as kudu, the newly added depen=
dency is even tougher to build on Power). We have also started looking at b=
uilding our native toolchain of Impala Pre-requisites for Power. &nbsp; </f=
ont></tt><tt><br></tt><br><BR>
</body></html>

--1__=EABB0A87DFA1E50D8f9e8a93df938690918cEABB0A87DFA1E50D--


--0__=EABB0A87DFA1E50D8f9e8a93df938690918cEABB0A87DFA1E50D--