hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sushanth Sowmyan <khorg...@gmail.com>
Subject Re: Apache Hive 0.13.1
Date Wed, 07 May 2014 01:41:52 GMT
Also, I wanted to throw in one more bit for those of you that are
interested in tinkering along :

http://people.apache.org/~khorgath/releases/0.13.1_RC0/relprep.pl
http://people.apache.org/~khorgath/releases/0.13.1_RC0/requested_jiras

This is the script and config file I'm using to generate this release.

It's very much a hack right now, and I hope to improve it to
streamline releases in the future, but how it can be used right now is
this way:

a) Put it in a hive git repo (and not have any changes that have not
been committed - this script will checkout a new branch and commit
things to that branch, so you want to make sure to have a clean repo)
b) Put the file requested_jiras in that dir as well.
c) Run the script from there.

It will check the differences between the branch being released
("branch-0.13" is hardcoded currently as a global), and looks at all
the commit logs in trunk that correspond to the jiras requested in the
requested_jiras file, sorts them in the order they were committed, and
then checks out a new branch called relprep-branch-0.13-<timestamp>,
and attempts to cherry-pick those commits in.

For some patches, this will not work, so there is an override
mechanism provided by entries in the requested_jiras file, as can be
observed in the file I mention above.

At the end of it, you'll have your 0.13.1 repo reproduction to test
against if you so desire.

Known Bugs :

a) I use system("....") or die "...";, which is faulty in that the
"die" code will never be reached. I need to fix this, but all the
system calls were working for me, and I'd much rather focus on the
release now, and improve this script later. This is a TODO
b) Some patches (those generated with --no-prefix) don't work with
older versions of git. You'll need a 1.8.x git for them, or you have
to generate git patches without --no-prefix.



On Tue, May 6, 2014 at 6:21 PM, Sushanth Sowmyan <khorgath@gmail.com> wrote:
> Hi Folks,
>
> After a run of the ptest framework across the 0.13.1 codebase, we have
> a couple of test failures that I'm trying to track down and resolve.
>
> If any of you are interested in looking at it on your own in the
> meanwhile, the conglomerate patch of all the patches I'm forward
> porting into 0.13.1 is over at
> http://people.apache.org/~khorgath/releases/0.13.1_RC0/0.13.1.gdiff.patch
>
> The current tests that are failing are as follows:
>
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tblproperties
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unset_table_view_property
> org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_unset_table_property
> org.apache.hive.service.cli.TestScratchDir.testLocalScratchDirs
>
> I'll update and follow up with patch devs as and when I find out the
> source for these errors.
>
> Thanks,
> -Sushanth
>
> On Mon, May 5, 2014 at 6:26 PM, Sushanth Sowmyan <khorgath@gmail.com> wrote:
>> Hi Folks,
>>
>> It's past 6pm PDT on May 5th 2014, so I'm beginning the process to
>> generate the 0.13.1 RC0.
>>
>> I've received backport patches for 0.13 for some of the jiras that did
>> not cleanly apply back from 0.14 to 0.13, and I'm thankful for them.
>> All requested patches are cleanly applying and compiling. Tests are
>> still running, and I expect them to run for a while more still - I
>> will make sure to wait for them to complete before actually cutting a
>> release candidate.
>>
>> The very first thing I'm supposed to do per the HowtoRelease wiki, is
>> to unlink any open non-blockers from the release, and then send out a
>> mail to the dev list confirming that there are no open non-blockers
>> for this release version. So, this is that mail:
>>
>>
>> The jira query string I used to verify this is as follows:
>>
>> project = HIVE AND priority not in (Critical, Major, Minor, Trivial)
>> AND fixVersion = 0.13.1 AND status = Open
>>
>> The good news is that the 0.13.1 branch name was only created a couple
>> of hours back, so no issues have been created on this, so there were
>> none to unlink. :)
>>
>>
>> On Fri, May 2, 2014 at 6:22 PM, Sushanth Sowmyan <khorgath@gmail.com> wrote:
>>> Hi Folks,
>>>
>>> It is now past 6pm PDT May 2nd 2014, so I have locked down the table
>>> of jiras for inclusion into 0.13.1. The jiras included are as listed
>>> in :
>>>
>>> https://cwiki.apache.org/confluence/display/Hive/Hive+0.13.1+Release+tracking
>>>
>>> The jiras being included are :
>>>
>>> HIVE-6919, HIVE-6921, HIVE-6957, HIVE-6945, HIVE-6862,
>>> HIVE-6883, HIVE-6961, HIVE-4576, HIVE-6952, HIVE-6828
>>> HIVE-6898, HIVE-6783, HIVE-6955, HIVE-6877, HIVE-6985
>>> HIVE-7001
>>>
>>> The inclusion requestors for the above are :
>>>
>>> thejas, ashutoshc, sershe, ekoifman, prasanth_j, hagleitn, szehon,
>>> rhbutani and amareshwari.
>>>
>>> If you are in this list, I might be pinging you in the next couple of
>>> days for help if the patch does not cleanly apply, or causes
>>> additional failures, and I appreciate any help you can tend.
>>>
>>> There is one jira on that list, HIVE-6826, which is in the list, and
>>> will still be included if it moves from Patch-available to Committed
>>> in 0.14 by Monday when I roll out the RC, which I have assurances from
>>> vikram that it will, given that it's waiting on a test result.
>>>
>>> If anyone has any further breaking jiras that they absolutely want in
>>> this release, please do still bring it up, I've created a new table
>>> for them in the release tracking wiki page. I do, however, raise the
>>> bar for inclusion by asking two other committers to also +1 such an
>>> inclusion request at this stage.
>>>
>>> Hopefully, we'll see an RC and a release pretty soon! :)
>>>
>>> Thanks all!
>>> -Sushanth
>>>
>>>
>>> On Thu, May 1, 2014 at 2:34 PM, Sushanth Sowmyan <khorgath@gmail.com> wrote:
>>>> Hi All,
>>>>
>>>> A quick reminder - per the timeline discussed so far, the list of
>>>> jiras for inclusion in 0.13.1 RC0 freezes tomorrow (May 2nd) evening
>>>> at 6pm PDT. Please make sure you ask for inclusion or edit the wiki
>>>> with any further requests so that we can get this release done in a
>>>> timely manner.
>>>>
>>>> Thanks,
>>>> -Sushanth
>>>>
>>>>
>>>> On Mon, Apr 28, 2014 at 1:39 PM, Sushanth Sowmyan <khorgath@gmail.com>
wrote:
>>>>> Good question - I think that if the number of patches is small ( <
~20
>>>>> which I expect because this is a stabilization release soon after
>>>>> 0.13.0), I'd prefer to commit them myself, so that I can retain order
>>>>> of patch application and make sure they all apply cleanly. I'm also
>>>>> actually using this as an opportunity towards testing out an
>>>>> semi-automated script that reads the requested jiras to do this - I
>>>>> figure it's useful to test something like that out on a smaller
>>>>> release that'll make further releases easier.
>>>>>
>>>>> On Mon, Apr 28, 2014 at 11:42 AM, Thejas Nair <thejas@hortonworks.com>
wrote:
>>>>>> Sushanth,
>>>>>> How do you want to go about committing the patches into 0.13 branch
?
>>>>>> Will you be able to commit the requested patches into 0.13.1 yourself,
or
>>>>>> do you want requestors to discuss it with you and then commit the
patches
>>>>>> themselves (if the requestor is a committer).
>>>>>>
>>>>>> Thanks,
>>>>>> Thejas
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Apr 26, 2014 at 2:19 PM, Sushanth Sowmyan <khorgath@gmail.com>wrote:
>>>>>>
>>>>>>> Added.
>>>>>>>
>>>>>>> If others have difficulty editing( I can't figure out how to
change editing
>>>>>>> privileges, but it seems to indicate that others can edit) the
page, I'll
>>>>>>> accept replies to this thread as well and can add it in.
>>>>>>>  On Apr 25, 2014 6:25 PM, "Sergey Shelukhin" <sergey@hortonworks.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > I don't have access to edit this page (or cannot figure
out the UI).
>>>>>>> > Username sershe.
>>>>>>> > Can you add
>>>>>>> > HIVE-6961 : Drop partitions treats partition columns as
strings (area -
>>>>>>> > metastore)
>>>>>>> >
>>>>>>> >
>>>>>>> > On Fri, Apr 25, 2014 at 4:20 PM, Sushanth Sowmyan <khorgath@gmail.com
>>>>>>> > >wrote:
>>>>>>> >
>>>>>>> > > I've created the following wiki link :
>>>>>>> > >
>>>>>>> > >
>>>>>>> >
>>>>>>> https://cwiki.apache.org/confluence/display/Hive/Hive+0.13.1+Release+tracking
>>>>>>> > >
>>>>>>> > > People should be able to request additional jiras by
adding it to the
>>>>>>> > > list. I think it might make sense to halt addition
of requests to the
>>>>>>> > > list 3 days before the RC is cut, so as to prevent
an endless-tail
>>>>>>> > > scenario, unless the bug in question is a breaking
severe issue,
>>>>>>> > > where, yes, after discussion, we can vote to add it
to the list. That
>>>>>>> > > also gives us time to run a full suite of tests on
a stable build
>>>>>>> > > before we cut the RC.
>>>>>>> > >
>>>>>>> > > I propose that the first RC (RC0) be built on Monday
May 5th at 6pm
>>>>>>> > > PDT, and the jira list on the wiki be closed to open/easy
additions at
>>>>>>> > > 6pm PDT on Friday May 2nd.
>>>>>>> > >
>>>>>>> > >
>>>>>>> > > On Fri, Apr 25, 2014 at 2:40 PM, Gunther Hagleitner
>>>>>>> > > <ghagleitner@hortonworks.com> wrote:
>>>>>>> > > > Sorry - HIVE-6824 isn't needed. Just the other
3. My bad.
>>>>>>> > > >
>>>>>>> > > > Thanks,
>>>>>>> > > > Gunther.
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > > On Fri, Apr 25, 2014 at 2:10 PM, Gunther Hagleitner
<
>>>>>>> > > > ghagleitner@hortonworks.com> wrote:
>>>>>>> > > >
>>>>>>> > > >> I'd like to request to include these Tez fixes:
>>>>>>> > > >>
>>>>>>> > > >> HIVE-6824, HIVE-6826, HIVE-6828, HIVE-6898
>>>>>>> > > >>
>>>>>>> > > >> Thanks,
>>>>>>> > > >> Gunther.
>>>>>>> > > >>
>>>>>>> > > >>
>>>>>>> > > >> On Fri, Apr 25, 2014 at 11:59 AM, Sushanth
Sowmyan <
>>>>>>> > khorgath@gmail.com
>>>>>>> > > >wrote:
>>>>>>> > > >>
>>>>>>> > > >>> True, I was counting two weeks from today,
but 0.13 has already
>>>>>>> been
>>>>>>> > > >>> out for a week. I'm amenable to having
an RC1 out on May 5th. If
>>>>>>> any
>>>>>>> > > >>> further issues appear that block, then
we can deal with them in an
>>>>>>> > > >>> RC2/etc modification to that.
>>>>>>> > > >>>
>>>>>>> > > >>> On Fri, Apr 25, 2014 at 11:45 AM, Thejas
Nair <
>>>>>>> > thejas@hortonworks.com>
>>>>>>> > > >>> wrote:
>>>>>>> > > >>> > On Fri, Apr 25, 2014 at 11:33 AM,
Sushanth Sowmyan <
>>>>>>> > > khorgath@gmail.com
>>>>>>> > > >>> >wrote:
>>>>>>> > > >>> >
>>>>>>> > > >>> >>
>>>>>>> > > >>> >> I think it's important to get
a bugfix/stabilization release
>>>>>>> > > >>> >> reasonably quickly, but it's
also important to give people a
>>>>>>> > little
>>>>>>> > > >>> >> time to try out 0.13, discover/report
bugs and fix them. So I
>>>>>>> > think
>>>>>>> > > >>> >> about two weeks is a good point?
And instead of releasing an RC
>>>>>>> > on a
>>>>>>> > > >>> >> friday, I'm thinking of pushing
it out to Monday - does 12th May
>>>>>>> > > sound
>>>>>>> > > >>> >> good to everyone?
>>>>>>> > > >>> >>
>>>>>>> > > >>> >
>>>>>>> > > >>> > I think we can aim for an earlier
date. Most of these issues seem
>>>>>>> > to
>>>>>>> > > be
>>>>>>> > > >>> > already committed to trunk or have
patches available. So the
>>>>>>> > > remaining
>>>>>>> > > >>> ones
>>>>>>> > > >>> > also might get committed to trunk
by early next week. How about
>>>>>>> > > shooting
>>>>>>> > > >>> > for May 5th (Monday) ?
>>>>>>> > > >>> > By then 0.13 would also have been
out for 2 weeks. If we have any
>>>>>>> > new
>>>>>>> > > >>> > critical bug reported that needs
a fix, we can hold off on the RC
>>>>>>> > for
>>>>>>> > > >>> few
>>>>>>> > > >>> > days.
>>>>>>> > > >>> > What do you think ?
>>>>>>> > > >>> >
>>>>>>> > > >>> > Thanks,
>>>>>>> > > >>> > Thejas
>>>>>>> > > >>> >
>>>>>>> > > >>> > --
>>>>>>> > > >>> > CONFIDENTIALITY NOTICE
>>>>>>> > > >>> > NOTICE: This message is intended
for the use of the individual or
>>>>>>> > > >>> entity to
>>>>>>> > > >>> > which it is addressed and may contain
information that is
>>>>>>> > > confidential,
>>>>>>> > > >>> > privileged and exempt from disclosure
under applicable law. If
>>>>>>> the
>>>>>>> > > >>> reader
>>>>>>> > > >>> > of this message is not the intended
recipient, you are hereby
>>>>>>> > > notified
>>>>>>> > > >>> that
>>>>>>> > > >>> > any printing, copying, dissemination,
distribution, disclosure or
>>>>>>> > > >>> > forwarding of this communication
is strictly prohibited. If you
>>>>>>> > have
>>>>>>> > > >>> > received this communication in error,
please contact the sender
>>>>>>> > > >>> immediately
>>>>>>> > > >>> > and delete it from your system. Thank
You.
>>>>>>> > > >>>
>>>>>>> > > >>
>>>>>>> > > >>
>>>>>>> > > >
>>>>>>> > > > --
>>>>>>> > > > CONFIDENTIALITY NOTICE
>>>>>>> > > > NOTICE: This message is intended for the use of
the individual or
>>>>>>> > entity
>>>>>>> > > to
>>>>>>> > > > which it is addressed and may contain information
that is
>>>>>>> confidential,
>>>>>>> > > > privileged and exempt from disclosure under applicable
law. If the
>>>>>>> > reader
>>>>>>> > > > of this message is not the intended recipient,
you are hereby
>>>>>>> notified
>>>>>>> > > that
>>>>>>> > > > any printing, copying, dissemination, distribution,
disclosure or
>>>>>>> > > > forwarding of this communication is strictly prohibited.
If you have
>>>>>>> > > > received this communication in error, please contact
the sender
>>>>>>> > > immediately
>>>>>>> > > > and delete it from your system. Thank You.
>>>>>>> > >
>>>>>>> >
>>>>>>> > --
>>>>>>> > CONFIDENTIALITY NOTICE
>>>>>>> > NOTICE: This message is intended for the use of the individual
or entity
>>>>>>> to
>>>>>>> > which it is addressed and may contain information that is
confidential,
>>>>>>> > privileged and exempt from disclosure under applicable law.
If the reader
>>>>>>> > of this message is not the intended recipient, you are hereby
notified
>>>>>>> that
>>>>>>> > any printing, copying, dissemination, distribution, disclosure
or
>>>>>>> > forwarding of this communication is strictly prohibited.
If you have
>>>>>>> > received this communication in error, please contact the
sender
>>>>>>> immediately
>>>>>>> > and delete it from your system. Thank You.
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
entity to
>>>>>> which it is addressed and may contain information that is confidential,
>>>>>> privileged and exempt from disclosure under applicable law. If the
reader
>>>>>> of this message is not the intended recipient, you are hereby notified
that
>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>> received this communication in error, please contact the sender immediately
>>>>>> and delete it from your system. Thank You.

Mime
View raw message