lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shalin Shekhar Mangar (JIRA)" <j...@apache.org>
Subject [jira] Updated: (SOLR-829) replication Compression
Date Fri, 21 Nov 2008 07:54:44 GMT

     [ https://issues.apache.org/jira/browse/SOLR-829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shalin Shekhar Mangar updated SOLR-829:
---------------------------------------

    Attachment: solr-829.patch

Changes:
# Fixes a possible connection leak in FileFetcher#getStream method.
# A single HttpClient is created with MultiThreadedHttpConnectionManager in SnapPuller and
is re-used for every operation
# Idle connections are closed in SnapPull#destroy method
# Release connection and stream closing is not done in a separate thread anymore
# ReplicationHandler does a snappull command in a new thread so that an API call for this
command is not kept waiting until the operation completes. The admin jsp which used to make
a call to this method in another thread is also changed to remove the thread creation.

I'll commit this in a day or two if there are no problems.

> replication Compression
> -----------------------
>
>                 Key: SOLR-829
>                 URL: https://issues.apache.org/jira/browse/SOLR-829
>             Project: Solr
>          Issue Type: Improvement
>          Components: replication (java)
>            Reporter: Simon Collins
>            Assignee: Shalin Shekhar Mangar
>         Attachments: email discussion.txt, solr-829.patch, solr-829.patch, solr-829.patch,
solr-829.patch
>
>
> From a discussion on the mailing list solr-user, it would be useful to have an option
to compress the files sent between servers for replication purposes.
> Files sent across between indexes can be compressed by a large margin allowing for easier
replication between sites.
> ...Noted by Noble Paul 
> we will use a gzip on both ends of the pipe . On the slave side you can say <str name="zip">true<str>
as an extra option to compress and send data from server 
> Other thoughts on issue: 
> Do keep in mind that compression is a CPU intensive process so it is a trade off between
CPU utilization and network bandwidth.  I have see cases where compressing the data before
a network transfer ended up being slower than without compression because the cost of compression
and un-compression was more than the gain in network transfer.
> Why invent something when compression is standard in HTTP? --wunder

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message