lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joel Bernstein (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-9905) Add NullStream to isolate the performance of the ExportWriter
Date Thu, 29 Dec 2016 15:30:58 GMT

    [ https://issues.apache.org/jira/browse/SOLR-9905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15785517#comment-15785517
] 

Joel Bernstein edited comment on SOLR-9905 at 12/29/16 3:30 PM:
----------------------------------------------------------------

The NullStream is a utility function to test the raw performance of the ExportWriter. This
is a nice utility to have to diagnose bottlenecks in streaming MapReduce operations. The NullStream
will allow developers to test the performance of the shuffling (Sorting, Partitioning, Exporting)
in isolation from the reduce operation (Rollup, Join, Group, etc..). 

The NullStream simply iterates it's internal stream and eats the tuples. It returns a single
Tuple from each worker with the number of Tuples processed. The idea is to iterate the stream
without additional overhead so the performance of the underlying stream can be isolated.

{code}
parallel(collection2, workers=7, sort="count desc", 
      null(search(collection1, 
                   q=*:*, 
                   fl="id", 
                   sort="id desc", 
                   qt="/export", 
                   wt="javabin", 
                   partitionKeys=id)))
{code}



was (Author: joel.bernstein):
The NullStream is a utility function to test the raw performance of the ExportWriter. This
is a nice utility to have to diagnose bottlenecks in streaming MapReduce operations. The NullStream
will allow developers to test the performance of the shuffling (Sorting, Partitioning, Exporting)
in isolation from the reduce operation (Rollup, Join, Group, etc..). 

{code}
parallel(collection2, workers=7, sort="count desc", 
      null(search(collection1, 
                   q=*:*, 
                   fl="id", 
                   sort="id desc", 
                   qt="/export", 
                   wt="javabin", 
                   partitionKeys=id)))
{code}


> Add NullStream to isolate the performance of the ExportWriter
> -------------------------------------------------------------
>
>                 Key: SOLR-9905
>                 URL: https://issues.apache.org/jira/browse/SOLR-9905
>             Project: Solr
>          Issue Type: New Feature
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Joel Bernstein
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message