lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: replication handler - compression
Date Fri, 31 Oct 2008 05:36:03 GMT
man gzip:

       -# --fast --best
              Regulate the speed of compression using the specified digit #, where -1 or --fast
indicates the  fastest  compres-
              sion  method (less compression) and -9 or --best indicates the slowest compression
method (best compression).  The
              default compression level is -6 (that is, biased towards high compression at
expense of speed).

 
So it could be better than the factor of 2, but also take longer. :)

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----
> From: Walter Underwood <wunderwood@netflix.com>
> To: solr-user@lucene.apache.org
> Sent: Thursday, October 30, 2008 11:52:47 AM
> Subject: Re: replication handler - compression
> 
> About a factor of 2 on a small, optimized index. Gzipping took 20 seconds,
> so it isn't free.
> 
> $ cd index-copy
> $ du -sk
> 134336  .
> $ gzip *
> $ du -sk
> 62084   .
> 
> wunder
> 
> On 10/30/08 8:20 AM, "Otis Gospodnetic" wrote:
> 
> > Yeah.  I'm just not sure how much benefit in terms of data transfer this will
> > save.  Has anyone tested this to see if this is even worth it?
> > 
> > 
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> > 
> > 
> > 
> > ----- Original Message ----
> >> From: Erik Hatcher 
> >> To: solr-user@lucene.apache.org
> >> Sent: Thursday, October 30, 2008 9:54:28 AM
> >> Subject: Re: replication handler - compression
> >> 
> >> +1 - the GzipServletFilter is the way to go.
> >> 
> >> Regarding request handlers reading HTTP headers, yeah,... this will improve,
> >> for 
> >> sure.
> >> 
> >>     Erik
> >> 
> >> On Oct 30, 2008, at 12:18 AM, Chris Hostetter wrote:
> >> 
> >>> 
> >>> : You are partially right. Instead of the HTTP header , we use a request
> >>> : parameter. (RequestHandlers cannot read HTP headers). If the param is
> >>> 
> >>> hmmm, i'm with walter: we shouldn't invent new mechanisms for
> >>> clients to request compression over HTTP from servers.
> >>> 
> >>> replicatoin is both special enough and important enough that if we had to
> >>> add special support to make that information available to the handler on
> >>> the master we could.
> >>> 
> >>> but frankly i don't think that's neccessary: the logic to turn on
> >>> compression if the client requests it using "Accept-Encoding: gzip" is
> >>> generic enough that there is no reason for it to be in a handler.  we
> >>> could easily put it in the SolrDispatchFilter, or even in a new
> >>> ServletFilte (i'm guessing iv'e seen about 74 different implementations
of
> >>> a GzipServletFilter in the wild that could be used as is.
> >>> 
> >>> then we'd have double wins: compression for replication, and compression
> >>> of all responses generated by Solr if hte client requests it.
> >>> 
> >>> -Hoss
> > 


Mime
View raw message