lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gregg Donovan <gregg...@gmail.com>
Subject Compression for solrbin?
Date Fri, 13 Nov 2015 21:30:31 GMT
We've had success with LZ4 compression in a custom ShardHandler to reduce
network overhead, getting ~25% compression with low CPU impact. LZ4 or
Snappy seem like reasonable choices[1] for maximizing compression +
transfer + decompression times in the data center.

Would it make sense to integrate compression into javabin itself? For the
ShardHandler and transaction log javabin usage it seems to make sense. We
could flip on gzip in Jetty for HTTP, but GZIP may add more CPU than is
desirable and wouldn't help with the transaction log.

If we did, i t seems incrementing the javabin version[2] and
compressing/decompressing inside of JavaBinCodec#marshal[3] and
JavaBinCodec#unmarshal[4] would allow us to retain backwards compatibility
with older clients or existing files.

Thoughts?

--Gregg

[1] http://cyan4973.github.io/lz4/#tab-2
[2]
https://github.com/apache/lucene-solr/blob/trunk/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L83
[3]
https://github.com/apache/lucene-solr/blob/trunk/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L112:L120
[4]
https://github.com/apache/lucene-solr/blob/trunk/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L129:L137

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message