hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15544) ABFS: validate packing, transient classpath, hadoop fs CLI
Date Tue, 24 Jul 2018 01:30:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16553664#comment-16553664
] 

Steve Loughran commented on HADOOP-15544:
-----------------------------------------

Also worth exploring is my cloudstore/storediag module, which I keep to debug this stuff:
[https://github.com/steveloughran/cloudstore/releases]

{code}
bin/hadoop -jar cloudstore-2.8.jar abfs://storeuri/path
{code}

That attempts to load the abfs, look for (and log) classes/dependencies it knows of, then
does some basic IO. It's intended to be for remote debugging of support calls, so can be expanded
to do stuff about hostname and proxy lookup, once the details are known

> ABFS: validate packing, transient classpath, hadoop fs CLI
> ----------------------------------------------------------
>
>                 Key: HADOOP-15544
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15544
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 3.2
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: dependencies.txt
>
>
> Validate the packaging and dependencies of ABFS
> * hadoop-cloud-storage artifact to export everything needed
> * {{hadoop fs -ls abfs://path}} to work in ASF distributions
> * check transient CP (e.g spark)
> Spark master;s hadoop-cloud module depends on hadoop-cloud-storage if you build with
the hadoop-3.1 profile, so it should automatically get in there. Just need to check that it
picks it up too



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message