hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-672) Spark shell throws OzoneFileSystem not found
Date Fri, 19 Oct 2018 16:12:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657006#comment-16657006

Anu Engineer commented on HDDS-672:

[~elek] The issue is as follows:
 # Ozone FS expects to find a configuration address called, *Ozone.om.address*.
 # Spark, Hive, Yarn all these applications are familiar with HDCFs and they will pass Hadoop
Configuration to the file system.
 # Hadoop Config will *not* have this key defined since we define *ozone.om.address* in ozone-site.xml. 
 # One way to fix is this to add the ozone.om.address to hdfs--site.xml or core–site.xml.
 # Another way to fix this is to understand that the older applications are not aware of ozone,
and hence take the configuration and re-initialize configuration to include ozone values also.
That is what this patch does.
 # Why this patch is not correct-- This does something that is surprising to the user, the
user passes a config and we are second-guessing the user, Which IMHO is the wrong thing to
 # We are making this modification so that applications like spark can work against ozone
without any modifications.

Please let me know if this makes sense.


> Spark shell throws OzoneFileSystem not found
> --------------------------------------------
>                 Key: HDDS-672
>                 URL: https://issues.apache.org/jira/browse/HDDS-672
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>    Affects Versions: 0.2.1
>            Reporter: Namit Maheshwari
>            Assignee: Anu Engineer
>            Priority: Major
>              Labels: app-compat
>         Attachments: HDDS-672.001.patch, HDDS-672.002.patch
> Spark shell throws OzoneFileSystem not found, if the ozone jars are not specified in
the --jars options

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message