lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Potter <tim.pot...@lucidworks.com>
Subject RE: monitoring solr logs
Date Mon, 30 Dec 2013 16:53:36 GMT
We're (LucidWorks) are actively developing on logstash4solr so if you have issues, let us know.
So far, so good for me but I upgraded to logstash 1.3.2 even though the logstash4solr version
includes 1.2.2 you can use the newer one. I'm not quite in production with my logstash4solr
<- rabbit-mq <- log4j <- Solr solution yet though ;-)

Yeah, 50GB is too much logging for only 150K docs. Maybe start by filtering by log level (WARN
and more severe). If a server crashes, you're likely to see some errors in the logstash side
but sometimes you may have to SSH to the specific box and look at the local log (so definitely
append all messages to the local Solr log too), I'm using something like the following for
local logging:

log4j.rootLogger=INFO, file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.MaxFileSize=50MB
log4j.appender.file.MaxBackupIndex=10
log4j.appender.file.File=logs/solr.log
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{3} %x - %m%n


Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com

________________________________________
From: adfel70 <adfel70@gmail.com>
Sent: Monday, December 30, 2013 9:34 AM
To: solr-user@lucene.apache.org
Subject: RE: monitoring solr logs

Actually I was considering using logstash4solr, but it didn't seem mature
enough.
does it work fine? any known bugs?

are you collecting the logs in the same solr cluster you use for the
production systems?
if so, what will you do if for some reason solr is down and you would like
to analyze the logs to see what happend?

btw, i started a new solr cluster with 7 shards, replicationfactor=3 and run
indexing job of 400K docs,
it got stuck on 150K because I used Socketappender directly to write to
logstash and logstash disk got full.

that's why I moved to using AsyncAppender, and I plan on moving to using
rabbit.
but this is also why I wanted to filter some of the logs. indexing 150K docs
prodcued 50GB of logs.
this seemed too much.




Tim Potter wrote
> I'm using logstash4solr (http://logstash4solr.org) for something similar
> ...
>
> I setup my Solr to use Log4J by passing the following on the command-line
> when starting Solr:
> -Dlog4j.configuration=file:///$SCRIPT_DIR/log4j.properties
>
> Then I use a custom Log4J appender that writes to RabbitMQ:
>
> https://github.com/plant42/rabbitmq-log4j-appender
>
> You can then configure a RabbitMQ input for logstash -
> http://logstash.net/docs/1.3.2/inputs/rabbitmq
>
> This decouples the log writes from log indexing in logstash4solr, which
> scales better for active Solr installations.
>
> Btw ... I just log everything from Solr using this approach but you can
> use standard Log4J configuration settings to limit which classes / log
> levels to send to the RabbitMQ appender.
>
> Cheers,
>
> Timothy Potter
> Sr. Software Engineer, LucidWorks
> www.lucidworks.com
>
> ________________________________________
> From: adfel70 &lt;

> adfel70@

> &gt;
> Sent: Monday, December 30, 2013 8:15 AM
> To:

> solr-user@.apache

> Subject: monitoring solr logs
>
> hi
> i'm trying to figure out which solr and zookeeper logs i should monitor
> and
> collect.
> All the logs will be written to a file but I want to collect some of them
> with logstash in order to be able to analyze them efficiently.
> any inputs on logs of which classes i should collect?
>
> thanks.
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721.html
> Sent from the Solr - User mailing list archive at Nabble.com.





--
View this message in context: http://lucene.472066.n3.nabble.com/monitoring-solr-logs-tp4108721p4108737.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message