hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoyu Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-1525) Mapreduce failure when using Hadoop 2.7.5
Date Thu, 30 May 2019 07:15:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-1525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851590#comment-16851590

Xiaoyu Yao commented on HDDS-1525:

Summary of the changes:
 # Remove the libs\ prefix for ozone classes in legacy jar POM. Otherwise, NM service loader
won't be able to find the OzoneTokenIdentifier class registered. cc: [~elek], can you think
of other solution for this? 

2019-05-30 05:36:00,204 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing
with tokens:
2019-05-30 05:36:00,325 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error
starting MRAppMaster
java.util.ServiceConfigurationError: org.apache.hadoop.security.token.TokenIdentifier: Provider
org.apache.hadoop.ozone.security.OzoneTokenIdentifier not found
at java.util.ServiceLoader.fail(ServiceLoader.java:239)
at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
2. Add the org.apache.hadoop.crypto.key.KeyProvider and org.apache.hadoop.io.Text to the
filtered class loader. 

3. Add BasicOzFs that instantiate BasicOzoneFileSystem() and use that for legacy hadoop
deployment instead of OzFs.

4. RpcClient should not implement the Hadoop3 KeyProviderTokenIssuer. 



> Mapreduce failure when using Hadoop 2.7.5
> -----------------------------------------
>                 Key: HDDS-1525
>                 URL: https://issues.apache.org/jira/browse/HDDS-1525
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>          Components: Ozone Filesystem
>    Affects Versions: 0.4.0
>            Reporter: Sammi Chen
>            Priority: Major
>         Attachments: HDDS-1525.poc.patch, teragen.log
> Integrate Ozone(0.4 branch) with Hadoop 2.7.5, "hdfs dfs -ls /" can pass, while teragen
> When add  -verbose:class to java options, it shows that class KeyProvider is loaded twice
by different classloader while it is only loaded once when execute  "hdfs dfs -ls /" 
> All jars under share/ozone/lib are added into hadoop classpath except ozone file system
current lib jar.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message