accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis <de...@camfex.cz>
Subject Re: Values go to a wrong table during recovery.
Date Fri, 20 Feb 2015 22:45:08 GMT
It is always used from the single thread

On 2/20/15, Denis <denis@camfex.cz> wrote:
>> So just to confirm, you have seen this issue appear while walogs are
>> enabled for all tables, yes?
>
> Yes.
>
>> Any chance you can share your code that writes across several
>> BatchWriters?
>
> The writer is:
>
> // way faster than Connector.createMultiTableBatchWriter
> class MyMultiTableBatchWriter(maxMemory: Long, maxLatency: Long,
> maxWriteThreads: Int)(implicit val conn: Connector) {
>   private[this] val writers = new
> scala.collection.mutable.HashMap[String, BatchWriter]
>   def getBatchWriter(tableName: String): BatchWriter = synchronized {
>     writers.getOrElseUpdate(tableName,
> conn.createBatchWriter(tableName, maxMemory, maxLatency,
> maxWriteThreads))
>   }
>   def flush() = writers.foreach(_._2.flush())
>   def close() = writers.foreach(_._2.close())
> }
>
> But the full code which creates/uses the indexes is more than 1Mb of
> scala code...
>
> On 2/20/15, Sean Busbey <busbey@cloudera.com> wrote:
>> On Fri, Feb 20, 2015 at 2:33 PM, Denis <denis@camfex.cz> wrote:
>>
>>> > Did you have walogs laying around when you upgraded?
>>>
>>> In 1.4-cluster (and first time in 1.6-cluster) I had walogs enabled
>>> for data tables and disabled for index tables.
>>>
>>> There was a bug in 1.4, if a tablet had empty walog there were some
>>> startup issues (tablet remains offline or something like this), and it
>>> happened often with index tables (hmm, the same tables I have this
>>> problem).
>>>
>>> So, in 1.4-cluster I disabled walog and ran full reindex periodically.
>>> After running 1.6-cluster some time I enabled walogs for all tables as
>>> the new cluster have less reliable hardware, which reboots from time
>>> to time.
>>>
>>>
>> So just to confirm, you have seen this issue appear while walogs are
>> enabled for all tables, yes?
>>
>> Any chance you can share your code that writes across several
>> BatchWriters?
>>
>> --
>> Sean
>>
>

Mime
View raw message