camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Grzegorz Grzybek (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CAMEL-7249) Upgrade Camel-HDFS component to be compatible with Apache Hadoop 2.x
Date Tue, 11 Mar 2014 12:28:44 GMT

    [ https://issues.apache.org/jira/browse/CAMEL-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13930278#comment-13930278
] 

Grzegorz Grzybek commented on CAMEL-7249:
-----------------------------------------

The problem is this hadoop code:
{code:java}
  private static void loadFileSystems() {
    synchronized (FileSystem.class) {
      if (!FILE_SYSTEMS_LOADED) {
        ServiceLoader<FileSystem> serviceLoader = ServiceLoader.load(FileSystem.class);
        for (FileSystem fs : serviceLoader) {
          SERVICE_FILE_SYSTEMS.put(fs.getScheme(), fs.getClass());
        }
        FILE_SYSTEMS_LOADED = true;
      }
    }
  }
...
  public static Class<? extends FileSystem> getFileSystemClass(String scheme,
      Configuration conf) throws IOException {
    if (!FILE_SYSTEMS_LOADED) {
      loadFileSystems();
    }
    Class<? extends FileSystem> clazz = null;
    if (conf != null) {
      clazz = (Class<? extends FileSystem>) conf.getClass("fs." + scheme + ".impl",
null);
    }
    if (clazz == null) {
      clazz = SERVICE_FILE_SYSTEMS.get(scheme);
    }
    if (clazz == null) {
      throw new IOException("No FileSystem for scheme: " + scheme);
    }
    return clazz;
  }
{code}
which uses either {{java.util.ServiceLoader}} (doesn't work in OSGi) or {{"fs.<scheme>.impl"}}
configuration properties.

OK. If I create routes manually, I can do the following:
{code:java}
org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
conf.setClass("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class, FileSystem.class);
conf.setClass("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class, FileSystem.class);

FileSystem.get("file:///", conf)
{code}

which caches the mapping {{file: -> org.apache.hadoop.fs.LocalFileSystem.class}} in current
classloader (static field {{org.apache.hadoop.fs.FileSystem.CACHE}}).

But it's not that easy with Blueprint based routes...

OK. I'll work more on ways to configure {{HdfsComponent}}.

> Upgrade Camel-HDFS component to be compatible with Apache Hadoop 2.x
> --------------------------------------------------------------------
>
>                 Key: CAMEL-7249
>                 URL: https://issues.apache.org/jira/browse/CAMEL-7249
>             Project: Camel
>          Issue Type: New Feature
>            Reporter: Grzegorz Grzybek
>            Assignee: Grzegorz Grzybek
>
> Currently camel-hdfs depends on hadoop-core:1.2.1. Make (working name) {{camel-hdfs2}}
component using newest (hadoop 2.3.0) libraries.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message