lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anatharaman, Srinatha (Contractor)" <Srinatha_Ananthara...@comcast.com>
Subject Flume SolrSink No Error but not Indexing
Date Tue, 07 Feb 2017 21:06:33 GMT
Hi,


I am indexing text document using Flume,
I do not see any error or warning message but data is not getting ingested to Solr
Log level for both Solr and Flume is set to TRACE, ALL

Flume version : 1.5.2.2.3
Solr Version : 5.5
Config files are as below
Flume Config :
agent.sources = SpoolDirSrc
agent.channels = FileChannel
agent.sinks = SolrSink

# Configure Source
agent.sources.SpoolDirSrc.channels = fileChannel
agent.sources.SpoolDirSrc.type = spooldir
agent.sources.SpoolDirSrc.spoolDir = /home/flume/source_emails
agent.sources.SpoolDirSrc.basenameHeader = true
agent.sources.SpoolDirSrc.fileHeader = true
#agent.sources.src1.fileSuffix = .COMPLETED
agent.sources.SpoolDirSrc.deserializer = org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder
# Use a channel that buffers events in memory
agent.channels.FileChannel.type = file
agent.channels.FileChannel.capacity = 10000
#agent.channels.FileChannel.transactionCapacity = 10000
# Configure Solr Sink
agent.sinks.SolrSink.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
agent.sinks.SolrSink.morphlineFile = /etc/flume/conf/morphline.conf
agent.sinks.SolrSink.batchsize = 1000
agent.sinks.SolrSink.batchDurationMillis = 2500
agent.sinks.SolrSink.channel = fileChannel
agent.sinks.SolrSink.morphlineId = morphline1
agent.sources.SpoolDirSrc.channels = FileChannel
agent.sinks.SolrSink.channel = FileChannel

Morphline Config
solrLocator: {
collection : gsearch
#zkHost : "127.0.0.1:9983"
zkHost : "codesolr-as-r3p:21810,codesolr-as-r3p:21811,codesolr-as-r3p:21812"
}
morphlines :
[
  {
    id : morphline1
    importCommands : ["org.kitesdk.**", "org.apache.solr.**"]
    commands :
    [
      { detectMimeType { includeDefaultMimeTypes : true } }
      {
        solrCell {
          solrLocator : ${solrLocator}
          captureAttr : true
          lowernames : true
          capture : [_attachment_body, _attachment_mimetype, basename, content, content_encoding,
content_type, file, meta]
          parsers : [ { parser : org.apache.tika.parser.txt.TXTParser } ]
         }
      }
      { generateUUID { field : id } }
      { sanitizeUnknownSolrFields { solrLocator : ${solrLocator} } }
      { logDebug { format : "output record: {}", args : ["@{}"] } }
      { loadSolr: { solrLocator : ${solrLocator} } }
    ]
  }
]

Please help me what could be the issue
Regards,
~Sri


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message