manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Questing regarding Tika text extraction and elasticsearch
Date Sun, 15 May 2016 16:11:30 GMT
Hi Silvio,

This sounds like a problem with the way the Elastic Search connector is
forming JSON.  The spec is silent on control characters:

http://rfc7159.net/rfc7159#rfc.section.8.1

... so we just embed those in strings.  But it sounds like ElasticSearch's
JSON parser is not so happy with them.

If we can find an encoding that satisfies everyone, we can change the code
to do what is needed.  Maybe "\0" for null, etc?

Karl


On Sun, May 15, 2016 at 10:21 AM, <silvio.r.meier@quantentunnel.de> wrote:

> Hi Apache ManifoldCF user list
>
> I’m experimenting with Apache ManifoldCF 2.3 which I use to index the
> network Windows shares of our company. I’m using Elasticsearch 1.7.4,
> Apache ManifoldCF 2.3 with MS Active Directory as authority source.
> I defined a job with the following connection configuration comprising the
> following chain of transformations (order in the list indicates the order
> of the transformations):
>
> 1.    Repository connection (MS Network Share)
> 2.    Allowed documents
> 3.    Tika extractor
> 4.    Metadata adjuster
> 5.    Elasticsearch
>
> I do this because I don’t want to store the original document inside the
> elasticsearch index but only the extracted text of the document. This works
> so far. However, there are numerous documents which cause an exception of
> the following kind when being  analyzed and sent to the indexer by Apache
> ManifoldCF. Note that the exceptions happens in the Elastic search analyzer:
>
> [2016-03-16 22:22:43,884][DEBUG][action.index             ] [Tefral the
> Surveyor] [shareindex][2], node[O2bWpnsKS8iAE7hwGEOpuA], [P], s[STARTED]:
> Failed to execute [index {[sharein
> dex][attachment][file://///du-evs-01/AppDevData%24/0Repository/temp/indexingtestcorpus/M%C3%A4useTastaturen%202.3.16%20-%20Kopie.pdf],
> source[{"access_permission:extract_for_access
> ibility" : "true","dcterms:created" :
> "2016-03-02T13:03:47Z","access_permission:can_modify" :
> "true","access_permission:modify_annotations" : "true","Creation-Date" :
> "2016-03-02T1
> 3:03:47Z","fileLastModified" :
> "2016-03-02T13:03:37.433Z","access_permission:fill_in_form" :
> "true","created" : "Wed Mar 02 14:03:47 CET 2016","stream_size" :
> "52067","dc:format" :
>  "application\/pdf; version=1.4","access_permission:can_print" :
> "true","stream_name" : "MäuseTastaturen 2.3.16 -
> Kopie.pdf","xmp:CreatorTool" : "Canon iR-ADV C5250  PDF","resourc
> eName" : "MäuseTastaturen 2.3.16 - Kopie.pdf","fileCreatedOn" :
> "2016-03-16T21:22:24.085Z","access_permission:assemble_document" :
> "true","meta:creation-date" : "2016-03-02T13:03:
> 47Z","lastModified" : "Wed Mar 02 14:03:37 CET 2016","pdf:PDFVersion" :
> "1.4","X-Parsed-By" : "org.apache.tika.parser.DefaultParser","shareName" :
> "AppDevData$","access_permission:
> can_print_degraded" : "true","xmpTPg:NPages" : "1","createdOn" : "Wed Mar
> 16 22:22:24 CET 2016","pdf:encrypted" :
> "false","access_permission:extract_content" : "true","producer" :
> "Adobe PSL 1.2e for Canon ","attributes" : "32","Content-Type" :
> "applica-tion\/pdf","allow_token_document" :
> ["LDAPConn:S-1-5-21-1751174259-1996115066-1435642685-16152","LDAPConn:S
> -1-5-21-1751174259-1996115066-1435642685-16153","LDAPConn:S-1-5-21-1751174259-1996115066-1435642685-7894"],"deny_token_document"
> : "LDAPConn:DEAD_AUTHORITY","allow_token_share" : "
> __nosecurity__","deny_token_share" : "__nosecurity__","allow_token_parent"
> : "__nosecurity__","deny_token_parent" : "__nosecurity__","content" : ""}]}]
> org.elasticsearch.index.mapper.MapperParsingException: failed to parse
> [_source]
>         at
> org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
>         at
> org.elasticsearch.index.mapper.internal.SourceFieldMapper.preParse(SourceFieldMapper.java:240)
>         at
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:540)
>         at
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
>         at
> org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:492)
>         at
> org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:192)
>         at
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
>         at
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
>         at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.elasticsearch.ElasticsearchParseException: Failed to parse
> content to map
>         at
> org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:130)
>         at
> org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:81)
>         at
> org.elasticsearch.index.mapper.internal.SourceFieldMapper.parseCreateField(SourceFieldMapper.java:274)
>         at
> org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
>         ... 11 more
> Caused by: org.elasticsearch.common.jackson.core.JsonParseException:
> Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using
> backslash to be included in string va
> lue
>  at [Source: [B@5b774e8b; line: 1, column: 1145]
>         at
> org.elasticsearch.common.jackson.core.JsonParser._constructError(JsonParser.java:1487)
>         at
> org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518)
>         at
> org.elasticsearch.common.jackson.core.base.ParserMinimalBase._throwUnquotedSpace(ParserMinimalBase.java:482)
>         at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2357)
>         at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2287)
>         at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:286)
>         at
> org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:86)
>         at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:293)
>         at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:275)
>         at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.readOrderedMap(AbstractXContentParser.java:258)
>         at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.mapOrdered(AbstractXContentParser.java:213)
>         at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.mapOrderedAndClose(AbstractXContentParser.java:228)
>         at
> org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:125)
>         ... 14 more
>
> This happens for documents of different types/extension, such as pdfs as
> well as xlsx, etc. It seems that Tika sometimes does not remove special
> characters as the null character 0x0000. The presence of the special
> characters causes Elasticsearch to omit the indexing of the document. Thus
> the document is not indexed at all, as  special characters need to be
> escaped when handed over as a JSON request. Is there a way to work around
> the problem with the existing functionality of Apache ManifoldCF?
>
> Regards
> Silvio
>
>

Mime
View raw message