hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sangjin Lee <sj...@apache.org>
Subject Re: [VOTE] Release Apache Hadoop 3.0.0 RC0
Date Tue, 21 Nov 2017 03:51:40 GMT
I checked the client jars that are supposed to contain shaded dependencies,
and they don't look quite right:

$ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-api-3.0.0.jar
-rw-r--r--  0 andrew andrew    44531 Nov 14 11:53
hadoop-3.0.0/share/hadoop/client/hadoop-client-api-3.0.0.jar
$ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-runtime-3.0.0.jar
-rw-r--r--  0 andrew andrew    45533 Nov 14 11:53
hadoop-3.0.0/share/hadoop/client/hadoop-client-runtime-3.0.0.jar
$ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-minicluster-3.0.0.jar
-rw-r--r--  0 andrew andrew    47015 Nov 14 11:53
hadoop-3.0.0/share/hadoop/client/hadoop-client-minicluster-3.0.0.jar

When I look at what's inside those jar, they only seem to include
pom-related files with no class files. Am I missing something?

When I build from the source with -Pdist, I do get much bigger jars:
total 113760
-rw-r--r--  1 sangjinlee  120039211  17055399 Nov 20 17:17
hadoop-client-api-3.0.0.jar
-rw-r--r--  1 sangjinlee  120039211  20451447 Nov 20 17:19
hadoop-client-minicluster-3.0.0.jar
-rw-r--r--  1 sangjinlee  120039211  20730866 Nov 20 17:18
hadoop-client-runtime-3.0.0.jar

Sangjin

On Mon, Nov 20, 2017 at 5:52 PM, Sangjin Lee <sjlee@apache.org> wrote:

>
>
> On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
>
>> Thanks for all the push, Andrew!
>>
>> Looking at the RC. Went through my usual check-list. Here's my summary.
>> Will cast my final vote after comparing and validating my findings with
>> others.
>>
>> Verification
>>
>>  - [Check] Successful recompilation from source tar-ball
>>  - [Check] Signature verification
>>  - [Check] Generating dist tarballs from source tar-ball
>>  - [Check] Testing
>>     -- Start NN, DN, RM, NM, JHS, Timeline Service
>>     -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
>> grep, pi
>>     -- Tested CLIs to print nodes, apps etc and also navigated UIs
>>
>> Issues found during testing
>>
>> Major
>>  - The previously supported way of being able to use different tar-balls
>> for different sub-modules is completely broken - common and HDFS tar.gz are
>> completely empty.
>>  - Cannot enable new UI in YARN because it is under a non-default
>> compilation flag. It should be on by default.
>>  - One decommissioned node in YARN ResourceManager UI always appears to
>> start with, even when there are no NodeManagers that are started yet:  Info
>> <IP>:-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not
>> in the CLI node -list
>>
>> Minor
>>  - resourcemanager-metrics.out is going into current directory instead of
>> log directory
>>  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
>> even work. Not just deprecated in favor of timelineserver as was advertised.
>>  - Spurious warnings on CLI
>>             17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml
>> not found
>>             17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find
>> 'resource-types.xml'.
>>
>> Side notes
>>
>>  - When did we stop putting CHANGES files into the source artifacts?
>>  - Even after "mvn install"ing once, shading is repeated again and again
>> for every new 'mvn install' even though there are no source changes - we
>> should see how this can be avoided.
>>  - Compatibility notes
>>     -- NM's env list is curtailed unlike in 2.x (For e.g,
>> HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior)
>>     -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar
>> into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
>>
>
> Sleep has always been in the jobclient test jar as long as I can remember,
> so it's not new for 3.0.
>
>
>>
>> Thanks
>> +Vinod
>>
>> > On Nov 14, 2017, at 1:34 PM, Andrew Wang <andrew.wang@cloudera.com>
>> wrote:
>> >
>> > Hi folks,
>> >
>> > Thanks as always to the many, many contributors who helped with this
>> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>> > available here:
>> >
>> > http://people.apache.org/~wang/3.0.0-RC0/
>> >
>> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>> >
>> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>> > additions include the merge of YARN resource types, API-based
>> configuration
>> > of the CapacityScheduler, and HDFS router-based federation.
>> >
>> > I've done my traditional testing with a pseudo cluster and a Pi job. My
>> +1
>> > to start.
>> >
>> > Best,
>> > Andrew
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message