hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Wang <andrew.w...@cloudera.com>
Subject Re: [Continued] [Release thread] 2.8.0 release activities
Date Fri, 20 Jan 2017 18:48:47 GMT
You can check the error message by clicking on it, a bunch like this:

Missing Signature:
'/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.8.0/hadoop-mapreduce-client-jobclient-2.8.0-tests.jar.asc'
does not exist for 'hadoop-mapreduce-client-jobclient-2.8.0-tests.jar'.

Did you maven deploy all the signature files?

On Fri, Jan 20, 2017 at 2:04 AM, Junping Du <jdu@hortonworks.com> wrote:

> Hi,
>     I have successfully built the release bit on branch-2.8.0 by
> following: https://wiki.apache.org/hadoop/HowToRelease step by step.
> However, when try to close the "Staging Repositories" at Nexus page (
> https://repository.apache.org/#stagingRepositories), I found our
> repository - orgapachehadoop-1051 cannot be closed due to some signature
> validation failed and one Apache rule failed. I never met this problem
> before in previous releasing 2.6.3 and 2.6.4. Is this related to our recent
> changes on release process/tools (docker based)? Any ideas or thoughts on
> how to fix the problem?
>
> Thanks,
>
> Junping
>
> ________________________________________
> From: Junping Du <jdu@hortonworks.com>
> Sent: Thursday, January 19, 2017 6:46 PM
> To: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Cc: Varun Vasudev
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> According to Varun's offline email, the security fixes has landed on
> branch-2, 2.8 and 2.8.0 branch.
> I was kicking off a new RC build (RC1), and will publish it for vote soon.
> In the mean time, please mark fix version as 2.8.1 for any new commits
> landed on branch-2.8, and don't commit anything to branch-2.8.0 at this
> moment. Thanks!
>
> Cheers,
>
> Junping
>
> ________________________________________
> From: Junping Du <jdu@hortonworks.com>
> Sent: Wednesday, January 18, 2017 3:26 PM
> To: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Cc: Varun Vasudev
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> Hi folks,
>      In the passed one or two weeks, we found some new blockers get coming
> on branch-2.8, like: YARN-6068 (log aggregation get stuck when NM restart
> with work preserving enabled) and YARN-6072 (RM unable to start in secure
> mode). Both of them are fixed now (YARN-6072 is fixed by Ajith and I fixed
> YARN-6068), and I was starting the RC build process since early this week.
> As we have significant build tools/process change (docker based) since 2.8
> comparing with 2.6/2.7, it takes me a while to get familiar with it and
> finally get a successful build on 2.8.0-RC0 last night.
>      I already push RC0 tag into public which is prerequisite step before
> RC voting. However,  in the mean while, I was pinged by Varun Vasudev that
> there are a known vulnerability issues on container_executor get identified
> and discussed in hadoop security email threads - looks like YARN-5704 fixed
> part of it, but left part - the privilege escalation via /proc/self/environ
> is not fixed yet. So most likely, I have to withdraw our 2.8.0 RC0 although
> I haven't announced it public for vote yet. I will wait this issue get
> fixed to prepare a new release candidate. As RC0 tag cannot be reverted
> after push into apache, our next release candidate will start from RC1.
>     As I mentioned in early email, 2.8.0 is a very big release (2300+
> commits since 2.7.3) and I am glad that we are almost there. Thanks
> everyone for being patient and contributing our release work. Please let me
> know if you have more comments or suggestions.
>
> Thanks,
>
> Junping
> ________________________________________
> From: Junping Du <jdu@hortonworks.com>
> Sent: Wednesday, January 04, 2017 1:41 AM
> To: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> Hi all Hadoopers,
>      I just commit YARN-3866 which is the last blocker for branch-2.8, so
> since tomorrow I will kick off process to prepare the first RC for our
> 2.8.0 release. This release will include at least 2374 commits that never
> landed before in previous 2.x releases. Please check
> https://s.apache.org/RW5k for details. There are also 352 fixes marked in
> 2.7.1, 2.7.2 and 2.7.3 but not marked with 2.8 (https://s.apache.org/RKti)
> that I need to double check all are landed in branch-2.8 and fix the
> versions before kicking off the first RC.
>      Our progress is slightly behind my estimation weeks ago. However,
> considering we just go through holidays and Jenkins cleanup issues linger
> on for several projects (YARN, etc.), our achievement here is still great!
> Thanks everyone for keep calm and carry on the help for the release. Kudos
> to Wangda, Jian, Akira, Jason, Sangjin, Karthik and all for pushing hard on
> blockers during this time. Also, special thanks to Vinod and Andrew for
> sharing knowledge and practice (include scripts for auto version check) for
> releasing effort.
>      Will try to prepare RC0 of 2.8 release for vote within this week.
> Stay tuned!
>
> Thanks,
>
> Junping
>
> ________________________________________
> From: Junping Du <jdu@hortonworks.com>
> Sent: Wednesday, December 07, 2016 11:31 AM
> To: Akira Ajisaka; common-dev@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> Thanks Akira for reporting this. Actually, HADOOP-2.8-JACC worked well for
> several runs before last weekend, but it get failed for latest several
> runs, probably affected by recently Jenkins down.
> However, from my recent manual kick off, it seems to be good again:
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/9/. I
> will keep an eye for today's nightly run to see if it back to normal.
>
>
> Thanks,
>
> Junping
>
> ________________________________________
> From: Akira Ajisaka <ajisakaa@apache.org>
> Sent: Wednesday, December 07, 2016 12:12 AM
> To: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: Re: [Continued] [Release thread] 2.8.0 release activities
>
> Thanks Junping and Andrew!
>
> HADOOP-2.8-JACC is not working well, so I manually kicked a job to
> compare 2.8 with 2.7.3.
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/
> Hadoop-trunk-JACC/24/artifact/target/compat-check/report.html
>
> Regards,
> Akira
>
> On 2016/12/02 2:08, Junping Du wrote:
> > Thanks Andrew! That's also a nice suggestion. I already create a similar
> job: https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-2.8-JACC/
> for 2.8 and kick off several runs manually. Will monitor incompatible
> status from there.
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> >
> > ________________________________
> > From: Andrew Wang <andrew.wang@cloudera.com>
> > Sent: Wednesday, November 30, 2016 4:18 PM
> > To: Junping Du
> > Cc: Sangjin Lee; common-dev@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org;
> yarn-dev@hadoop.apache.org; Vinod Vavilapalli; Jian He; Wangda Tan;
> sjlee@twitter.com
> > Subject: Re: [Continued] [Release thread] 2.8.0 release activities
> >
> > I recommend giving the JACC report another look. I set up a
> parameterized jenkins job which you can trigger manually for branch-2.8:
> >
> > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-JACC/
> >
> > On Wed, Nov 30, 2016 at 4:06 PM, Junping Du <jdu@hortonworks.com<mailto:
> jdu@hortonworks.com>> wrote:
> > Hi Sangjin and all,
> >
> >      That sounds good. If anyone have priority fix to backport , please
> nominate it by following this email thread or ping me on JIRA directly. And
> I agree that we should keep in mind that our bar for 2.8 release should be
> high now as we want to move quickly on this release. Non-critical fixes can
> wait for next one.
> >
> >       Any other comments and suggestions?
> >
> >
> > Thanks,
> >
> >
> > Junping
> >
> >
> > ________________________________
> > From: sjlee0@gmail.com<mailto:sjlee0@gmail.com> <sjlee0@gmail.com
> <mailto:sjlee0@gmail.com>> on behalf of Sangjin Lee <sjlee@apache.org
> <mailto:sjlee@apache.org>>
> > Sent: Wednesday, November 30, 2016 3:24 PM
> > To: Junping Du
> > Cc: common-dev@hadoop.apache.org<mailto:common-dev@hadoop.apache.org>;
> hdfs-dev@hadoop.apache.org<mailto:hdfs-dev@hadoop.apache.org>;
> mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@hadoop.apache.org>;
> yarn-dev@hadoop.apache.org<mailto:yarn-dev@hadoop.apache.org>; Vinod
> Vavilapalli; Jian He; Wangda Tan; sjlee@twitter.com<mailto:sjlee
> @twitter.com>
> > Subject: Re: [Continued] [Release thread] 2.8.0 release activities
> >
> > Thanks for picking this up Junping!
> >
> > I heard on email threads and offline discussions that there may be
> interests in porting some fixes from branch-2 to branch-2.8. I still feel
> that the bar should be pretty high to do that, but it might be good to have
> folks suggest some critical fixes that need to be released sooner. Thoughts?
> >
> > On Tue, Nov 29, 2016 at 8:50 PM, Junping Du <jdu@hortonworks.com<mailto:
> jdu@hortonworks.com><mailto:jdu@hortonworks.com<mailto:jdu@hortonworks.com>>>
> wrote:
> > Hi folks,
> >      Per asked by Vinod offline, I would like to continue the effort to
> push 2.8 release out in next several weeks. The plan is as following:
> >      - We only have 5 blockers and 3 critical issues (
> https://s.apache.org/6kwx) so far, and most of them are in good progress
> now. I put them all on apache ciwiki ( https://cwiki.apache.org/
> confluence/display/HADOOP/Hadoop+2.8+Release) and will do daily monitor
> and update on these issues in next couple of weeks.
> >      - When all blocker/critical issues are gone (target EOD is Dec.
> 10), I will push out all major/minor issues that is open. So if you are
> working on some major issues and target for release in 2.8, please keep
> track of our release status and your work progress.
> >      - I will create RC immediately once our 2.8 target issues are clean
> (especially for blocker and critical issues) for voting process.
> >
> >      I hope we can make it within year 2016, best before Xmas holiday.
> And I need help from all of you. :)
> >
> >
> > Thanks,
> >
> > Junping
> >
> > ________________________________________
> > From: Vinod Kumar Vavilapalli <vinodkv@apache.org<mailto:vin
> odkv@apache.org><mailto:vinodkv@apache.org<mailto:vinodkv@apache.org>>>
> > Sent: Monday, July 25, 2016 2:57 PM
> > To: Jian He
> > Cc: mapreduce-dev@hadoop.apache.org<mailto:mapreduce-dev@
> hadoop.apache.org><mailto:mapreduce-dev@hadoop.apache.org<mailto:
> mapreduce-dev@hadoop.apache.org>>; yarn-dev@hadoop.apache.org<mailto:
> yarn-dev@hadoop.apache.org><mailto:yarn-dev@hadoop.apache.org<mailto:
> yarn-dev@hadoop.apache.org>>; hdfs-dev@hadoop.apache.org<mailto:
> hdfs-dev@hadoop.apache.org><mailto:hdfs-dev@hadoop.apache.org<mailto:
> hdfs-dev@hadoop.apache.org>>; common-dev@hadoop.apache.org<mailto:
> common-dev@hadoop.apache.org><mailto:common-dev@hadoop.apache.org<mailto:
> common-dev@hadoop.apache.org>>
> > Subject: Re: [Release thread] 2.8.0 release activities
> >
> > Thanks for all that great help moving this release forward, Jian,
> Sangjin, Wangda et al! Appreciate it.
> >
> > The License / Notice fix is finally in. And I pushed out a 2.7.3 RC0
> last week.
> >
> > I only see 9 blocker / critical fixes pending for 2.8.0 -
> https://issues.apache.org/jira/issues/?filter=12334985. Let’s do this!
> >
> > Thanks
> > +Vinod
> >
> >> On May 11, 2016, at 6:04 PM, Jian He <jhe@hortonworks.com<mailto:jh
> e@hortonworks.com><mailto:jhe@hortonworks.com<mailto:jhe@hortonworks.com>>>
> wrote:
> >>
> >> For MapReduce/YARN, I closed a few staled ones. Only 4 jiras needs
> attention for 2.8
> >>
> >> MAPREDUCE-6288
> >> YARN-1815
> >> YARN-4685
> >> YARN-4844
> >>
> >> The rest are either improvements or long-standing issues and does not
> qualify release blocker, IMO.
> >> I think we’ll try to get these 4 jiras in asap. The rest will be on
> best effort, resolve as much as possible and move them out if not resolved
> in time.
> >>
> >> Jian
> >>
> >> On May 11, 2016, at 5:37 PM, Wangda Tan <wheeleast@gmail.com<mailto:wh
> eeleast@gmail.com><mailto:wheeleast@gmail.com<mailto:wheeleast@gmail.com
> >><mailto:wheeleast@gmail.com<mailto:wheeleast@gmail.com><mailto:
> wheeleast@gmail.com<mailto:wheeleast@gmail.com>>>> wrote:
> >>
> >> Sounds good to me :).
> >>
> >> Jian and I have looked at all existing 2.8.0 blockers and criticals
> today.
> >> To me more than half of MR/YARN blockers/criticals of 2.8 should be
> moved
> >> out. Left comments on these JIRAs asked original owners, plan to update
> >> target version of these JIRAs early next week.
> >>
> >> Will keep this thread updated.
> >>
> >> Thanks,
> >> Wangda
> >>
> >>
> >> On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee <sjlee@apache.org<mailto:
> sjlee@apache.org><mailto:sjlee@apache.org<mailto:sjlee@apache.org
> >><mailto:sjlee@apache.org<mailto:sjlee@apache.org><mailto:sjlee@
> apache.org<mailto:sjlee@apache.org>>>> wrote:
> >>
> >> How about this? I'll review the HADOOP/HDFS bugs in that list to come up
> >> with true blockers for 2.8.0 or JIRAs that are close to being ready.
> I'll
> >> report the list here. Then folks can chime in if you agree
> >>
> >> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?
> >>
> >> Thanks,
> >> Sangjin
> >>
> >> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan <wheeleast@gmail.com
> <mailto:wheeleast@gmail.com><mailto:wheeleast@gmail.com<mailto:wheelea
> st@gmail.com>><mailto:wheeleast@gmail.com<mailto:wheeleast@gmail.com
> ><mailto:wheeleast@gmail.com<mailto:wheeleast@gmail.com>>>> wrote:
> >>
> >> +1, we should close such staled JIRAs to avoid doing unnecessary checks
> >> for
> >> every releases.
> >>
> >> I'm working on reviewing YARN/MR critical/blocker patches currently, it
> >> gonna very helpful if someone else can help with reviewing Common/HDFS
> >> JIRAs.
> >>
> >> Thanks,
> >> Wangda
> >>
> >>
> >> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee <sjlee@apache.org<mailto:
> sjlee@apache.org><mailto:sjlee@apache.org<mailto:sjlee@apache.org
> >><mailto:sjlee@apache.org<mailto:sjlee@apache.org><mailto:sjlee@
> apache.org<mailto:sjlee@apache.org>>>> wrote:
> >>
> >> Where do we stand in terms of closing out blocker/critical issues for
> >> 2.8.0? I still see 50 open JIRAs in Vinod's list:
> >> https://issues.apache.org/jira/issues/?filter=12334985
> >>
> >> But I see a lot of JIRAs with no patches or very stale patches. It
> >> would be
> >> a good exercise to come up with the list of JIRAs that we need to block
> >> 2.8.0 for and focus our attention on closing them out. Thoughts?
> >>
> >> Thanks,
> >> Sangjin
> >>
> >> On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran <stevel@hortonworks.com
> <mailto:stevel@hortonworks.com><mailto:stevel@hortonworks.com<mailto:
> stevel@hortonworks.com>><mailto:stevel@hortonworks.com<mailto:
> stevel@hortonworks.com><mailto:stevel@hortonworks.com<mailto:stevel@
> hortonworks.com>>>
> >>
> >> wrote:
> >>
> >>
> >> On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
> >> vinodkv@apache.org<mailto:vinodkv@apache.org><mailto:vinodkv@apache.org
> <mailto:vinodkv@apache.org>><mailto:vinodkv@apache.org<mailto:vinodkv@
> apache.org><mailto:vinodkv@apache.org<mailto:vinodkv@apache.org>>>>
> >> wrote:
> >>
> >> We are not converging - there’s still 58 more. I need help from the
> >> community in addressing / review 2.8.0 blockers. If folks can start
> >> with
> >> reviewing Patch available tickets, that’ll be great.
> >>
> >>
> >>
> >>
> >> I'm still doing the s3a stuff, other people testing and reviewing this
> >> stuff welcome.
> >>
> >> in particular, I could do with others playing with this patch of mine,
> >> which adds counters and things into S3a, based on the azure
> >> instrumentation
> >>
> >> https://issues.apache.org/jira/browse/HADOOP-13028
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org<mailto:
> yarn-dev-unsubscribe@hadoop.apache.org><mailto:yarn-dev-unsubscribe@
> hadoop.apache.org<mailto:yarn-dev-unsubscribe@hadoop.apache.org>>
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org<mailto:
> yarn-dev-help@hadoop.apache.org><mailto:yarn-dev-help@hadoop.apache.org<
> mailto:yarn-dev-help@hadoop.apache.org>>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> <mailto:mapreduce-dev-unsubscribe@hadoop.apache.org><mailto:mapreduce-
> dev-unsubscribe@hadoop.apache.org<mailto:mapreduce-dev-
> unsubscribe@hadoop.apache.org>>
> > For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
> <mailto:mapreduce-dev-help@hadoop.apache.org><mailto:mapreduce-dev-help@
> hadoop.apache.org<mailto:mapreduce-dev-help@hadoop.apache.org>>
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message