lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eungsop Yoo (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SOLR-9236) AutoAddReplicas feature with one replica loses some documents not committed during failover
Date Fri, 08 Jul 2016 00:06:11 GMT

     [ https://issues.apache.org/jira/browse/SOLR-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eungsop Yoo updated SOLR-9236:
------------------------------
    Description: 
I need to index huge amount of logs, so I decide to use AutoAddReplica feature with only one
replica.
When using AutoAddReplicas with one replica, some benefits are expected.
- no redundant data files for replicas
-- saving disk usage
- best indexing performance 

I expected that Solr fails over just like HBase.
The feature worked almost as it was expected, except for some missing documents during failover.
I found two reasons for the missing.

1. The leader replica does not replay any transaction logs. But when there is only one replica,
it should be the leader.
So I made the leader replica replay the transaction logs when using AutoAddReplicas with on
replica.

But the above fix did not resolve the problem.

2. As failover occurred, the transaction log directory had a deeper directory depth. Just
like this, tlog/tlog/tlog/...
The transaction log could not be replayed, because the transaction log directory was changed
during failover. 
So I made the transaction log directory not changed during failover.

After these fixes, AutoAddReplicas with one replica fails over well without losing any documents.


  was:
I need to index huge amount of logs, so I decide to use AutoAddReplica feature with only one
replica.
When using AutoAddReplicas with one replica, some benefits are expected.
- no redundant data files for replicas
-- saving disk usage
- best indexing performance 

I expected that Solr fails over just like HBase.
The feature worked almost as it was expected, except for some missing documents during failover.
I found two regions for the missing.

1. The leader replica does not replay any transaction logs. But when there is only one replica,
it should be the leader.
So I made the leader replica replay the transaction logs when using AutoAddReplicas with on
replica.

But the above fix did not resolve the problem.

2. As failover occurred, the transaction log directory had a deeper directory depth. Just
like this, tlog/tlog/tlog/...
The transaction log could not be replayed, because the transaction log directory was changed
during failover. 
So I made the transaction log directory not changed during failover.

After these fixes, AutoAddReplicas with one replica fails over well without losing any documents.



> AutoAddReplicas feature with one replica loses some documents not committed during failover
> -------------------------------------------------------------------------------------------
>
>                 Key: SOLR-9236
>                 URL: https://issues.apache.org/jira/browse/SOLR-9236
>             Project: Solr
>          Issue Type: Bug
>          Components: hdfs, SolrCloud
>            Reporter: Eungsop Yoo
>            Assignee: Mark Miller
>            Priority: Minor
>         Attachments: SOLR-9236.patch, SOLR-9236.patch
>
>
> I need to index huge amount of logs, so I decide to use AutoAddReplica feature with only
one replica.
> When using AutoAddReplicas with one replica, some benefits are expected.
> - no redundant data files for replicas
> -- saving disk usage
> - best indexing performance 
> I expected that Solr fails over just like HBase.
> The feature worked almost as it was expected, except for some missing documents during
failover.
> I found two reasons for the missing.
> 1. The leader replica does not replay any transaction logs. But when there is only one
replica, it should be the leader.
> So I made the leader replica replay the transaction logs when using AutoAddReplicas with
on replica.
> But the above fix did not resolve the problem.
> 2. As failover occurred, the transaction log directory had a deeper directory depth.
Just like this, tlog/tlog/tlog/...
> The transaction log could not be replayed, because the transaction log directory was
changed during failover. 
> So I made the transaction log directory not changed during failover.
> After these fixes, AutoAddReplicas with one replica fails over well without losing any
documents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message