lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fred Smith <thefredsm...@gmail.com>
Subject Can Solr with the StatsComponent analyze 20+ million files?
Date Mon, 08 Aug 2011 20:59:38 GMT
Hi,
Currently we are in the process of figuring out how to deal with
millions of CSV files containing weather data(20+ million files). Each
file is about 500 bytes in size.
We want to calculate statistics on fields read from the file. For
example, the standard deviation of wind speed across all 20+ million files.
Processing speed isn't an important issue. The analysis routine can run
for days, if needed.

The StatsComponent(http://wiki.apache.org/solr/StatsComponent) for Solr
appears to be able to calculate the statistics we are interested in.

Will the StatsComponent in Solr do what we need with minimal configuration?
Can the StatsComponent only be used on a subset of the data? For
example, only look at data from certain months?
Are there other free programs out there that can parse and analyze 20+
million files?

We are still very new to Solr and really appreciate all your help.
Thanks,
Fred

Mime
View raw message