Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
From: zhenke luo <zhenkeluousa@gmail.com>
Content-Type: multipart/alternative; boundary="Apple-Mail=_27C95F31-B44C-4B92-B308-6257805073B6"
Message-Id: <B6A1DC83-12B6-4192-A52C-E0BBE5DFFB5C@gmail.com>
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
Subject: Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
Date: Tue, 21 Mar 2017 16:56:39 -0700
References: <1489481154988.12454@hortonworks.com> <1489742296302.98222@hortonworks.com> <CAPC70Q8QfJ=7NPYEwSnS_ebQwYmisoF6e61P_0U4AvQtUTZ5=Q@mail.gmail.com> <1490029375875.97974@hortonworks.com> <CAPC70Q9fWRWMLLOJ+re+0G-i7B=KwzTvcQsKH1Vb0TETn6gn3w@mail.gmail.com> <CAOJC_diCCymyvS3fMtAtpLwk1vAbAqgQctcWYxG8a_aV1WBHGw@mail.gmail.com> <1885380663.577462.1490128948557@mail.yahoo.com> <2066798409.124076.1490134650365@mail.yahoo.com> <DABB4692-16DA-474D-8DB1-52D9FE58DCE5@amazon.com>
To: common-dev@hadoop.apache.org
In-Reply-To: <DABB4692-16DA-474D-8DB1-52D9FE58DCE5@amazon.com>
archived-at: Wed, 22 Mar 2017 00:03:08 -0000

--Apple-Mail=_27C95F31-B44C-4B92-B308-6257805073B6
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hello:

I found an issue with HADOOP-10027 that :

SnappyCompressor can=E2=80=99t load native lib properly.

Test TestHadoopnative is failing  in hadoop-apache2 version 0.10=20

stacktrace:
Caused by: java.lang.RuntimeException: native snappy library not =
available: SnappyCompressor has not been loaded.
	at =
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCode=
c.java:72)
	at =
org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.=
java:195)
	at =
com.facebook.presto.hadoop.HadoopNative.loadAllCodecs(HadoopNative.java:71=
)
	at =
com.facebook.presto.hadoop.HadoopNative.requireHadoopNative(HadoopNative.j=
ava:52)
	... 24 more

After look into the code found that in SnappyCompressor.java=20
static {
    if (NativeCodeLoader.isNativeCodeLoaded() &&
        NativeCodeLoader.buildSupportsSnappy()) {
      try {
        initIDs();
        nativeSnappyLoaded =3D true;
      } catch (Throwable t) {
        LOG.error("failed to load SnappyCompressor", t);
      }
    }
  }
the native method initIDs will complain: no such field clazz.


> On Mar 21, 2017, at 4:32 PM, Poepping, Thomas <poeppt@amazon.com> =
wrote:
>=20
>=20
>=20
> On 3/21/17, 3:17 PM, "Kuhu Shukla" <kshukla@yahoo-inc.com.INVALID> =
wrote:
>=20
>=20
>    +1 (non-binding)
>=20
>    - Verified signatures.
>    - Downloaded and built from source tar.gz.
>    - Deployed a pseudo-distributed cluster on Mac Sierra.
>    - Ran example Sleep job successfully.
>    - Deployed latest Apache Tez 0.9 and ran sample Tez =
orderedwordcount successfully.
>=20
>    Thank you Junping and everyone else who worked on getting this =
release out.
>=20
>    Warm Regards,
>    Kuhu
>    On Tuesday, March 21, 2017, 3:42:46 PM CDT, Eric Badger =
<ebadger@yahoo-inc.com.INVALID> wrote:
>    +1 (non-binding)
>=20
>    - Verified checksums and signatures of all files
>    - Built from source on MacOS Sierra via JDK 1.8.0 u65
>    - Deployed single-node cluster
>    - Successfully ran a few sample jobs
>=20
>    Thanks,
>=20
>    Eric
>=20
>    On Tuesday, March 21, 2017 2:56 PM, John Zhuge =
<john.zhuge@gmail.com> wrote:
>=20
>=20
>=20
>    +1. Thanks for the great effort, Junping!
>=20
>=20
>      - Verified checksums and signatures of the tarballs
>      - Built source code with Java 1.8.0_66-b17 on Mac OS X 10.12.3
>      - Built source and native code with Java 1.8.0_111 on Centos =
7.2.1511
>      - Cloud connectors:
>          - s3a: integration tests, basic fs commands
>          - adl: live unit tests, basic fs commands. See notes below.
>      - Deployed a pseudo cluster, passed the following sanity tests in
>      both insecure and SSL mode:
>          - HDFS: basic dfs, distcp, ACL commands
>          - KMS and HttpFS: basic tests
>          - MapReduce wordcount
>          - balancer start/stop
>=20
>=20
>    Needs the following JIRAs to pass all ADL tests:
>=20
>      - HADOOP-14205. No FileSystem for scheme: adl. Contributed by =
John Zhuge.
>      - HDFS-11132. Allow AccessControlException in contract tests when
>      getFileStatus on subdirectory of existing files. Contributed by =
Vishwajeet
>      Dusane
>      - HADOOP-13928. =
TestAdlFileContextMainOperationsLive.testGetFileContext1
>      runtime error. (John Zhuge via lei)
>=20
>=20
>    On Mon, Mar 20, 2017 at 10:31 AM, John Zhuge <jzhuge@cloudera.com> =
wrote:
>=20
>> Yes, it only affects ADL. There is a workaround of adding these 2
>> properties to core-site.xml:
>>=20
>> <property>
>>   <name>fs.adl.impl</name>
>>   <value>org.apache.hadoop.fs.adl.AdlFileSystem</value>
>> </property>
>>=20
>> <property>
>>   <name>fs.AbstractFileSystem.adl.impl</name>
>>   <value>org.apache.hadoop.fs.adl.Adl</value>
>> </property>
>>=20
>> I have the initial patch ready but hitting these live unit test =
failures:
>>=20
>> Failed tests:
>>=20
>> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
>> testListStatus:257
>> expected:<1> but was:<10>
>>=20
>> Tests in error:
>>=20
>> =
TestAdlFileContextMainOperationsLive>FileContextMainOperationsBaseTest.
>> testMkdirsFailsForSubdirectoryOfExistingFile:254
>> =C2=BB AccessControl
>>=20
>> TestAdlFileSystemContractLive.runTest:60->FileSystemContractBaseTest.
>> testMkdirsFailsForSubdirectoryOfExistingFile:190
>> =C2=BB AccessControl
>>=20
>>=20
>> Stay tuned...
>>=20
>> John Zhuge
>> Software Engineer, Cloudera
>>=20
>> On Mon, Mar 20, 2017 at 10:02 AM, Junping Du <jdu@hortonworks.com> =
wrote:
>>=20
>>> Thank you for reporting the issue, John! Does this issue only affect =
ADL
>>> (Azure Data Lake) which is a new feature for 2.8 rather than other
>> existing
>>> FS? If so, I think we can leave the fix to 2.8.1 to fix given this =
is
>> not a
>>> regression and just a new feature get broken.=E2=80=8B
>>>=20
>>>=20
>>> Thanks,
>>>=20
>>>=20
>>> Junping
>>> ------------------------------
>>> *From:* John Zhuge <jzhuge@cloudera.com>
>>> *Sent:* Monday, March 20, 2017 9:07 AM
>>> *To:* Junping Du
>>> *Cc:* common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
>>> yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>>> *Subject:* Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)
>>>=20
>>> Discovered https://issues.apache.org/jira/browse/HADOOP-14205 "No
>>> FileSystem for scheme: adl".
>>>=20
>>> The issue were caused by backporting HADOOP-13037 to branch-2 and
>> earlier.
>>> HADOOP-12666 should not be backported, but some changes are needed:
>>> property fs.adl.impl in core-default.xml and =
hadoop-tools-dist/pom.xml.
>>>=20
>>> I am working on a patch.
>>>=20
>>>=20
>>> John Zhuge
>>> Software Engineer, Cloudera
>>>=20
>>> On Fri, Mar 17, 2017 at 2:18 AM, Junping Du <jdu@hortonworks.com> =
wrote:
>>>=20
>>>> Hi all,
>>>>     With fix of HDFS-11431 get in, I've created a new release =
candidate
>>>> (RC3) for Apache Hadoop 2.8.0.
>>>>=20
>>>>     This is the next minor release to follow up 2.7.0 which has =
been
>>>> released for more than 1 year. It comprises 2,900+ fixes, =
improvements,
>> and
>>>> new features. Most of these commits are released for the first time =
in
>>>> branch-2.
>>>>=20
>>>>     More information about the 2.8.0 release plan can be found =
here:
>>>> =
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>>>>=20
>>>>     New RC is available at: http://home.apache.org/~junpin
>>>> g_du/hadoop-2.8.0-RC3
>>>>=20
>>>>     The RC tag in git is: release-2.8.0-RC3, and the latest commit =
id
>>>> is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009
>>>>=20
>>>>     The maven artifacts are available via repository.apache.org at:
>>>> =
https://repository.apache.org/content/repositories/orgapachehadoop-1057
>>>>=20
>>>>     Please try the release and vote; the vote will run for the =
usual 5
>>>> days, ending on 03/22/2017 PDT time.
>=20
>>>>=20
>>>> Thanks,
>>>>=20
>>>> Junping
>>>>=20
>>>=20
>>>=20
>>=20
>=20
>=20
>=20
>    --=20
>    John
>=20
>=20
>    =
---------------------------------------------------------------------
>    To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>    For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>=20
>=20


--Apple-Mail=_27C95F31-B44C-4B92-B308-6257805073B6--