hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vishwajeet Dusane (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15965) Upgrade to ADLS SDK which has major performance improvement for ingress/egress
Date Wed, 05 Dec 2018 12:37:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710014#comment-16710014
] 

Vishwajeet Dusane commented on HADOOP-15965:
--------------------------------------------

Thanks [~stevel@apache.org].
{quote} * make sure it works when wildfly isn't on the CP, and doesn't bother warning people
at info/warn{quote} * Wildfly dependency is shaded with Adls Java SDK hence no dependency package
is needed. Verified that even if native libraries are failed to load, Java SDK uses JSE instead.
However, not recommended since would hit hard on performance.
 * For info/warn - You might be referring to HADOOP-15851? In that case, Java SDK 2.3.2 does
not incorporate this change. Java SDK 2.3.3 would have the change, timeline yet to be decided.
Would it be fine to bypass this check for this Jira since not a functional issue? We have
to unblock priority customer hence request to relax on this logline.

 
{quote} * can you make sure the SDK checks content tyoe of Auth & raises meaningful error
if != json{quote} * This issue is addressed in the [Java SDK 2.3.2 version|[https://github.com/Azure/azure-data-lake-store-java/blob/dafdcdef608f13e951b0336a44307a7b5f1e894b/src/main/java/com/microsoft/azure/datalake/store/oauth2/AzureADAuthenticator.java#L266].]

{quote} * review all transitive dependencies: is it going to force in new versions of anything?{quote}
* No other dependency version changes.

{quote} * and of course, usual "test everything process" applies{quote}
 
{code:java}
[INFO] Tests run: 470, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 134.406 s - in org.apache.hadoop.fs.adl.live.TestAdlSupportedCharsetInPath
[INFO] Running org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.372 s - in org.apache.hadoop.fs.adl.live.TestMetadata
[INFO] Running org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.952 s - in org.apache.hadoop.fs.adl.TestAzureADTokenProvider
[INFO] Running org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.1 s - in org.apache.hadoop.fs.adl.TestRelativePathFormation
[INFO] Running org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.154 s - in org.apache.hadoop.fs.adl.TestValidateConfiguration
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 869, Failures: 0, Errors: 0, Skipped: 4
{code}

> Upgrade to ADLS SDK which has major performance improvement for ingress/egress
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-15965
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15965
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/adl
>            Reporter: Vishwajeet Dusane
>            Priority: Major
>
> Upgrade ADLS SDK to version 2.3.2 which has major improvements
>  # Add special handling for 404 errors when requesting tokens from MSI
>  # Fix liststatus response parsing when filestatus object contains array in one field.
>  # Use wildfly openssl native binding with Java. This is a workaround to [https://bugs.openjdk.java.net/browse/JDK-8046943]issue.
2X performance boost over HTTPS. Similar to HADOOP-15669



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message