manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Getting class not found exception with DistributedFileSystem (New Connector for MICO)
Date Tue, 08 Mar 2016 09:30:00 GMT
Hi Chalitha,

This seems to be going through a Hadoop class loader of some kind, which
could well not be written in the standard way and would thus be unable to
locate the class in question.

Best practice for class loaders is to use the current class's classloader
which loading the desired class, but I suspect that Hadoop has its own way
of doing this, because it's Hadoop, and because Hadoop is pretty much its
own world.

If you look at the HDFS connector, it uses another path to get into the
file system and doesn't go in through the main door.

Thanks,
Karl



On Tue, Mar 8, 2016 at 12:13 AM, Chalitha Perera <cperera@zaizi.com> wrote:

> Hi All,
>
> We are writing a new transformation connector[1] to utilize MICO
> platform[2]. Platform back-end uses hadoop for storage.
> When trying to get the outputstream within connector which is a
> HdfsDataOutputStream I am getting class not found exception with
> DistributedFileSystem. ManifoldCF already includes hadoop-hdfs jar in
> connectors-lib which should contain this class.
>
> This error is generated only when used within manifoldcf. We have tried the
> platform utilization code from a separate jar and it worked without any
> errors.
>
> Any help for this matter is greatly appreciated.
>
> Here is the error I am getting,
>
> Error tossed: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.hdfs.DistributedFileSystem not found
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.hdfs.DistributedFileSystem not found
>     at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2074)
>     at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2578)
>     at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>     at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>     at
>
> eu.mico.platform.storage.impl.StorageServiceHDFS.getOutputStream(StorageServiceHDFS.java:62)
>     at
>
> eu.mico.platform.persistence.impl.MarmottaContent.getOutputStream(MarmottaContent.java:266)
>     at
>
> org.apache.manifoldcf.agents.transformation.mico.MicoExtractor.addOrReplaceDocumentWithException(MicoExtractor.java:182)
>     at
>
> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3221)
>     at
>
> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3072)
>     at
>
> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineObjectWithVersions.addOrReplaceDocumentWithException(IncrementalIngester.java:2706)
>     at
>
> org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:756)
>     at
>
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1583)
>     at
>
> org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1548)
>     at
>
> org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:404)
>     at
>
> org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399)
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.hdfs.DistributedFileSystem not found
>     at
>
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980)
>     at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2072)
>
>
> Thanks,
> Chalitha
>
>
> [1] https://github.com/zaizi/sensefy-mico-connector
> [2] https://bitbucket.org/mico-project/platform
>
> --
>
> ------------------------------
> This message should be regarded as confidential. If you have received this
> email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard copy
> by an authorised signatory.
>
> Zaizi Ltd is registered in England and Wales with the registration number
> 6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
> London W6 7AN.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message