accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-2990) BatchWriter never recovers from failure(s)
Date Wed, 16 Jul 2014 16:08:07 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063654#comment-14063654
] 

Josh Elser commented on ACCUMULO-2990:
--------------------------------------

It sounds like we would need to introduce some sort of flush ID/counter semantics on the client
side. Thread 1 and 2 would both have some local ID less than the ID generated when those 11
mutations were flushed. We would then know that both of those threads need to receive the
exception while any new thread that uses the BatchWriter after that flush in step 3 would
have a new ID and would know that the exception should not be passed to the new thread.

Is there an easier approach that the above? I can think of a couple of odd cases already that
would be tricky.

> BatchWriter never recovers from failure(s)
> ------------------------------------------
>
>                 Key: ACCUMULO-2990
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2990
>             Project: Accumulo
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 1.5.1, 1.6.0
>            Reporter: Josh Elser
>            Priority: Critical
>             Fix For: 1.5.2, 1.6.1, 1.7.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> In trying to understand what's happening in ACCUMULO-2964, I noticed that I had similar
exceptions from two different threads. One of the threads starting working after the unexplained
thrift exceptions from a tserver restart, and the other continued to repeatedly fail for the
lifetime of the test.
> I repeatedly saw this exception: 
> {noformat}
> 2014-07-11 04:14:41,591 [replication.WorkMaker] WARN : Failed to write work mutations
for replication, will retry
> org.apache.accumulo.core.client.MutationsRejectedException: # constraint violations :
0  security codes: {accumulo.metadata(ID:!0)=[DEFAULT_SECURITY_ERROR]}  # server errors 0
# exceptions 0
>         at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:537)
>         at org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addMutation(TabletServerBatchWriter.java:249)
>         at org.apache.accumulo.core.client.impl.BatchWriterImpl.addMutation(BatchWriterImpl.java:45)
>         at org.apache.accumulo.master.replication.WorkMaker.addWorkRecord(WorkMaker.java:184)
>         at org.apache.accumulo.master.replication.WorkMaker.run(WorkMaker.java:124)
>         at org.apache.accumulo.master.replication.ReplicationDriver.run(ReplicationDriver.java:91)
> {noformat}
> The part that struck me as odd was that the BatchWriter wasn't against the metadata table,
but the replication table.
> I looked into the TabletServerBatchWriter. It appears that once the client sees a MutationsRejectedException,
that BatchWriter becomes useless as the internal member {{somethingFailed}} is never reset
back to {{false}} after the failure is reported. Same goes for {{serverSideErrors}}, {{unknownErrors}},
{{lastUnknownErrors}}, too.
> If this is the case, this is a bug because the BatchWriter should be resilient in this
regard and not force the client to create a new Instance. If that's infeasible to do, we should
add exceptions to the BatchWriter that fail fast when a BatchWriter is used that will report
repeatedly report the same failure over and over again.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message