hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sangjin Lee <sj...@apache.org>
Subject Re: [VOTE] Release Apache Hadoop 3.0.0 RC0
Date Tue, 21 Nov 2017 05:59:40 GMT
On Mon, Nov 20, 2017 at 9:46 PM, Andrew Wang <andrew.wang@cloudera.com>
wrote:

> Thanks for the spot Sangjin. I think this bug introduced in create-release
> by HADOOP-14835. The multi-pass maven build generates these dummy client
> jars during the site build since skipShade is specified.
>
> This might be enough to cancel the RC. Thoughts?
>

IMO yes. This was one of the key features mentioned in the 3.0 release
notes. I appreciate your effort for the release Andrew!


>
> Best,
> Andrew
>
> On Mon, Nov 20, 2017 at 7:51 PM, Sangjin Lee <sjlee@apache.org> wrote:
>
>> I checked the client jars that are supposed to contain shaded
>> dependencies, and they don't look quite right:
>>
>> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-api-3.0.0.jar
>> -rw-r--r--  0 andrew andrew    44531 Nov 14 11:53
>> hadoop-3.0.0/share/hadoop/client/hadoop-client-api-3.0.0.jar
>> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-runtime-3.0.0.jar
>> -rw-r--r--  0 andrew andrew    45533 Nov 14 11:53
>> hadoop-3.0.0/share/hadoop/client/hadoop-client-runtime-3.0.0.jar
>> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-minicluster-3.0.
>> 0.jar
>> -rw-r--r--  0 andrew andrew    47015 Nov 14 11:53
>> hadoop-3.0.0/share/hadoop/client/hadoop-client-minicluster-3.0.0.jar
>>
>> When I look at what's inside those jar, they only seem to include
>> pom-related files with no class files. Am I missing something?
>>
>> When I build from the source with -Pdist, I do get much bigger jars:
>> total 113760
>> -rw-r--r--  1 sangjinlee  120039211  17055399 Nov 20 17:17
>> hadoop-client-api-3.0.0.jar
>> -rw-r--r--  1 sangjinlee  120039211  20451447 Nov 20 17:19
>> hadoop-client-minicluster-3.0.0.jar
>> -rw-r--r--  1 sangjinlee  120039211  20730866 Nov 20 17:18
>> hadoop-client-runtime-3.0.0.jar
>>
>> Sangjin
>>
>> On Mon, Nov 20, 2017 at 5:52 PM, Sangjin Lee <sjlee@apache.org> wrote:
>>
>>>
>>>
>>> On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli <
>>> vinodkv@apache.org> wrote:
>>>
>>>> Thanks for all the push, Andrew!
>>>>
>>>> Looking at the RC. Went through my usual check-list. Here's my summary.
>>>> Will cast my final vote after comparing and validating my findings with
>>>> others.
>>>>
>>>> Verification
>>>>
>>>>  - [Check] Successful recompilation from source tar-ball
>>>>  - [Check] Signature verification
>>>>  - [Check] Generating dist tarballs from source tar-ball
>>>>  - [Check] Testing
>>>>     -- Start NN, DN, RM, NM, JHS, Timeline Service
>>>>     -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
>>>> grep, pi
>>>>     -- Tested CLIs to print nodes, apps etc and also navigated UIs
>>>>
>>>> Issues found during testing
>>>>
>>>> Major
>>>>  - The previously supported way of being able to use different
>>>> tar-balls for different sub-modules is completely broken - common and HDFS
>>>> tar.gz are completely empty.
>>>>  - Cannot enable new UI in YARN because it is under a non-default
>>>> compilation flag. It should be on by default.
>>>>  - One decommissioned node in YARN ResourceManager UI always appears to
>>>> start with, even when there are no NodeManagers that are started yet:  Info
>>>> <IP>:-1, DECOMMISSIONED, null rack. It shows up only in the UI though,
not
>>>> in the CLI node -list
>>>>
>>>> Minor
>>>>  - resourcemanager-metrics.out is going into current directory instead
>>>> of log directory
>>>>  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
>>>> even work. Not just deprecated in favor of timelineserver as was advertised.
>>>>  - Spurious warnings on CLI
>>>>             17/11/20 17:07:34 INFO conf.Configuration:
>>>> resource-types.xml not found
>>>>             17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to
>>>> find 'resource-types.xml'.
>>>>
>>>> Side notes
>>>>
>>>>  - When did we stop putting CHANGES files into the source artifacts?
>>>>  - Even after "mvn install"ing once, shading is repeated again and
>>>> again for every new 'mvn install' even though there are no source changes
-
>>>> we should see how this can be avoided.
>>>>  - Compatibility notes
>>>>     -- NM's env list is curtailed unlike in 2.x (For e.g,
>>>> HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior)
>>>>     -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar
>>>> into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
>>>>
>>>
>>> Sleep has always been in the jobclient test jar as long as I can
>>> remember, so it's not new for 3.0.
>>>
>>>
>>>>
>>>> Thanks
>>>> +Vinod
>>>>
>>>> > On Nov 14, 2017, at 1:34 PM, Andrew Wang <andrew.wang@cloudera.com>
>>>> wrote:
>>>> >
>>>> > Hi folks,
>>>> >
>>>> > Thanks as always to the many, many contributors who helped with this
>>>> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>>>> > available here:
>>>> >
>>>> > http://people.apache.org/~wang/3.0.0-RC0/
>>>> >
>>>> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>>>> >
>>>> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>>>> > additions include the merge of YARN resource types, API-based
>>>> configuration
>>>> > of the CapacityScheduler, and HDFS router-based federation.
>>>> >
>>>> > I've done my traditional testing with a pseudo cluster and a Pi job.
>>>> My +1
>>>> > to start.
>>>> >
>>>> > Best,
>>>> > Andrew
>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message