lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martijn van Groningen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3099) Grouping module should allow subclasses to set the group key per document
Date Sat, 28 May 2011 21:54:47 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040681#comment-13040681
] 

Martijn van Groningen commented on LUCENE-3099:
-----------------------------------------------

Also some initial performance indications. This data was created with the runner provided
in the patch. A ran it on a index of 30M documents that has 7502 unique groups.
{code}
===== First pass collectors execution =====
number   |collector                               |query                   |time (ms)   |Current
mem usage (MB)  
1        |FirstPassGroupingCollector              |country:es              |888         |170
                    
2        |ResearchFirstPassGroupingCollector      |country:es              |710         |170
                    
3        |FirstPassGroupingCollector              |country:es country:eg   |1448        |170
                     
4        |ResearchFirstPassGroupingCollector      |country:es country:eg   |1350        |170
                     
5        |FirstPassGroupingCollector              |country:it              |2           |170
                   
6        |ResearchFirstPassGroupingCollector      |country:it              |3           |170
                    
7        |FirstPassGroupingCollector              |country:gr              |45          |170
                     
8        |ResearchFirstPassGroupingCollector      |country:gr              |43          |170
                   
9        |FirstPassGroupingCollector              |country:it              |3           |170
                   
10       |ResearchFirstPassGroupingCollector      |country:it              |3           |170
                   
11       |FirstPassGroupingCollector              |country:it              |2           |170
                   
12       |ResearchFirstPassGroupingCollector      |country:it              |3           |170
                   
13       |FirstPassGroupingCollector              |country:eg              |67          |170
                   
14       |ResearchFirstPassGroupingCollector      |country:eg              |91          |171
                    
15       |FirstPassGroupingCollector              |country:es country:eg   |1387        |171
                   
16       |ResearchFirstPassGroupingCollector      |country:es country:eg   |1351        |171
                    
17       |FirstPassGroupingCollector              |country:e*              |2264        |173
                   
18       |ResearchFirstPassGroupingCollector      |country:e*              |1036        |176
                    
19       |FirstPassGroupingCollector              |country:it              |2           |176
                    
20       |ResearchFirstPassGroupingCollector      |country:it              |3           |176
                     
21       |FirstPassGroupingCollector              |*:*                     |526         |176
                    
22       |ResearchFirstPassGroupingCollector      |*:*                     |410         |176
                    
23       |FirstPassGroupingCollector              |country:gr              |50          |177
                     
24       |ResearchFirstPassGroupingCollector      |country:gr              |45          |177
                     
25       |FirstPassGroupingCollector              |country:es country:eg   |1397        |177
                    
26       |ResearchFirstPassGroupingCollector      |country:es country:eg   |1362        |177
                    
27       |FirstPassGroupingCollector              |*:*                     |544         |177
                     
28       |ResearchFirstPassGroupingCollector      |*:*                     |411         |177
                   
29       |FirstPassGroupingCollector              |country:tr              |54          |177
                    
30       |ResearchFirstPassGroupingCollector      |country:tr              |52          |177
                     
31       |FirstPassGroupingCollector              |*:*                     |531         |177
                   
32       |ResearchFirstPassGroupingCollector      |*:*                     |413         |177
                    
33       |FirstPassGroupingCollector              |*:*                     |534         |177
                    
34       |ResearchFirstPassGroupingCollector      |*:*                     |416         |177
                     
35       |FirstPassGroupingCollector              |country:eg              |67          |177
                   
36       |ResearchFirstPassGroupingCollector      |country:eg              |64          |177
                   
37       |FirstPassGroupingCollector              |country:gr              |45          |177
                    
38       |ResearchFirstPassGroupingCollector      |country:gr              |47          |177
                     
39       |FirstPassGroupingCollector              |country:e*              |1109        |181
                    
40       |ResearchFirstPassGroupingCollector      |country:e*              |965         |185
                     
41       |FirstPassGroupingCollector              |country:us              |56          |185
                     
42       |ResearchFirstPassGroupingCollector      |country:us              |62          |185
                     
43       |FirstPassGroupingCollector              |country:gr              |44          |185
                     
44       |ResearchFirstPassGroupingCollector      |country:gr              |42          |185
                     
45       |FirstPassGroupingCollector              |country:es              |872         |185
                     
46       |ResearchFirstPassGroupingCollector      |country:es              |723         |185
                    
47       |FirstPassGroupingCollector              |country:es              |804         |185
                     
48       |ResearchFirstPassGroupingCollector      |country:es              |717         |185
                     
49       |FirstPassGroupingCollector              |country:es              |747         |185
                     
50       |ResearchFirstPassGroupingCollector      |country:es              |723         |185
                     



===== All groups collectors execution =====
number   |collector                     |query                   |time (ms)   |count     
 |Current mem usage (MB)
1        |AllGroupsCollector            |country:us              |54          |11        
 |185         
2        |ResearchAllGroupsCollector    |country:us              |53          |11        
 |169         
3        |AllGroupsCollector            |country:it              |2           |9         
 |169         
4        |ResearchAllGroupsCollector    |country:it              |2           |9         
 |169         
5        |AllGroupsCollector            |country:tr              |64          |1700      
 |170         
6        |ResearchAllGroupsCollector    |country:tr              |67          |1700      
 |170         
7        |AllGroupsCollector            |*:*                     |454         |7502      
 |170         
8        |ResearchAllGroupsCollector    |*:*                     |471         |7502      
 |170         
9        |AllGroupsCollector            |country:tr              |51          |1700      
 |171         
10       |ResearchAllGroupsCollector    |country:tr              |61          |1700      
 |171         
11       |AllGroupsCollector            |country:es              |717         |3012      
 |171         
12       |ResearchAllGroupsCollector    |country:es              |755         |3012      
 |171         
13       |AllGroupsCollector            |country:es              |691         |3012      
 |172         
14       |ResearchAllGroupsCollector    |country:es              |765         |3012      
 |172         
15       |AllGroupsCollector            |*:*                     |411         |7502      
 |172         
16       |ResearchAllGroupsCollector    |*:*                     |462         |7502      
 |173         
17       |AllGroupsCollector            |country:es              |685         |3012      
 |173         
18       |ResearchAllGroupsCollector    |country:es              |743         |3012      
 |173         
19       |AllGroupsCollector            |country:eg              |57          |646       
 |173         
20       |ResearchAllGroupsCollector    |country:eg              |63          |646       
 |173         
21       |AllGroupsCollector            |country:tr              |51          |1700      
 |173         
22       |ResearchAllGroupsCollector    |country:tr              |54          |1700      
 |174         
23       |AllGroupsCollector            |country:it              |2           |9         
 |174         
24       |ResearchAllGroupsCollector    |country:it              |3           |9         
 |174         
25       |AllGroupsCollector            |*:*                     |433         |7502      
 |174         
26       |ResearchAllGroupsCollector    |*:*                     |709         |7502      
 |175         
27       |AllGroupsCollector            |country:tr              |50          |1700      
 |175         
28       |ResearchAllGroupsCollector    |country:tr              |59          |1700      
 |175         
29       |AllGroupsCollector            |country:gr              |44          |2061      
 |175         
30       |ResearchAllGroupsCollector    |country:gr              |43          |2061      
 |175         
31       |AllGroupsCollector            |country:us              |17          |11        
 |176         
32       |ResearchAllGroupsCollector    |country:us              |27          |11        
 |176         
33       |AllGroupsCollector            |country:tr              |55          |1700      
 |176         
34       |ResearchAllGroupsCollector    |country:tr              |55          |1700      
 |176         
35       |AllGroupsCollector            |country:e*              |941         |3658      
 |180         
36       |ResearchAllGroupsCollector    |country:e*              |1087        |3658      
 |184         
37       |AllGroupsCollector            |country:es              |706         |3012      
 |184         
38       |ResearchAllGroupsCollector    |country:es              |747         |3012      
 |184         
39       |AllGroupsCollector            |country:eg              |57          |646       
 |185         
40       |ResearchAllGroupsCollector    |country:eg              |61          |646       
 |185         
41       |AllGroupsCollector            |country:es country:eg   |1507        |3658      
 |170         
42       |ResearchAllGroupsCollector    |country:es country:eg   |1512        |3658      
 |171         
43       |AllGroupsCollector            |country:gr              |38          |2061      
 |171         
44       |ResearchAllGroupsCollector    |country:gr              |41          |2061      
 |172         
45       |AllGroupsCollector            |country:us              |17          |11        
 |172         
46       |ResearchAllGroupsCollector    |country:us              |19          |11        
 |172         
47       |AllGroupsCollector            |country:es              |708         |3012      
 |172         
48       |ResearchAllGroupsCollector    |country:es              |742         |3012      
 |172         
49       |AllGroupsCollector            |country:us              |17          |11        
 |172         
50       |ResearchAllGroupsCollector    |country:us              |19          |11        
 |173         
{code}

> Grouping module should allow subclasses to set the group key per document
> -------------------------------------------------------------------------
>
>                 Key: LUCENE-3099
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3099
>             Project: Lucene - Java
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>             Fix For: 3.2, 4.0
>
>         Attachments: LUCENE-3099.patch
>
>
> The new grouping module can only group by a single-valued indexed field.
> But, if we make the 'getGroupKey' a method that a subclass could override, then I think
we could refactor Solr over to the module, because it could do function queries and normal
queries via subclass (I think).
> This also makes the impl more extensible to apps that might have their own interesting
group values per document.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message