lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anton Potehin" <an...@orbita1.ru>
Subject search problem
Date Tue, 28 Feb 2006 07:59:41 GMT
I have a problem. 

There is an index, which contains about 6,000,000 records (15,000,000
will be soon) the size is 4GB. Index is optimized and consists of only
one segment. This index stores the products. Each product has brand,
price and about 10 more additional fields. I want to not just find
something in the index also I want to get the lists of all brands and
price. The list of brands is needed for displaying all of the products
and the quantity of products of this brand for certain search request.

 

Here is the fragment of the code for results obtaining:

 

            ...the fragment of code for search is passed by, because it
is not so important... 

int count = hits.length(); 

            TreeMap brands = new TreeMap();

            float[] priceArr = new float[count];

 

            ArrayList results = new ArrayList();

            for (int i = 0; i < count; i++) {

                try {

                    Document doc = hits.doc(i);

                    String brand =
doc.getField("manufacturer").stringValue();

                    int number = 1;

                    try {

                        number = Integer.parseInt((String)
brands.get(brand));

                        number++;

                    } catch (Exception ex) {

                    }

                    brands.put(brand, number + "");

                    float price =
PriceUtils.priceFromStringToFloat(doc.getField("price").stringValue());

                    float maxPriceCurrent =
PriceUtils.priceFromStringToFloat(doc.getField("max_price").stringValue(
));

                    float minPriceCurrent =
PriceUtils.priceFromStringToFloat(doc.getField("min_price").stringValue(
));

 

                    priceArr[i] = price;

 

//start----------------------Brand Filter

                    if (!brandFilter.equals("") &&
!brandFilter.toLowerCase().equals(brand.toLowerCase())) {

                        continue;

                    }

//end------------------------Brand Filter

 

//start----------------------Price Filter

                    if (minPrice != -1 && price < minPrice) {

                        continue;

                    }

                    if (maxPrice != -1 && price > maxPrice) {

                        continue;

                    }

//start----------------------Price Filter

                    if (startAt <= numberOfResults && results.size() <
maxNumberResults) {

                        String name =
StringUtils.stringLimitation(MAX_LENGTH_OF_NAME,
doc.getField("name").stringValue());

                        String description =
StringUtils.stringLimitation(MAX_LENGTH_OF_DESCRIPTION,
doc.getField("description").stringValue());

                        String idProduct =
doc.getField("id_product").stringValue();

                        Element item =
documentFactory.createElement("item");

 

//                        ...the reading of needed fields from document
is omitted... 

 

                        results.add((threadNumber - 1), item);

                    }

                    numberOfResults++;

                } catch (IOException e) {

                    System.err.println(e.toString());

                }

            }

 

As it is seen, to feel brands and priceArr I have to check all of
results from hits. The search may give up to 100,000 results, the
checking takes too much time and memory. Do anybody have some ideas how
to speed up this process?

 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message