accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christopher Tubbs (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-4749) Need a bulk loading test equivalent to continuous ingest
Date Wed, 03 Jan 2018 20:23:00 GMT


Christopher Tubbs commented on ACCUMULO-4749:

[~jkrdev], the design ideas I discussed with you were mostly a "getting started" brainstorm.
Those ideas originated with a discussion I had with [~kturner].

One of the problems I can see with the design as written up above is that the ingest processes
can finish out of order if there are multiples running. That could cause problems with verification.
To resolve that, Keith had suggested that ingestors could be responsible for a predetermined
subset of the generators only (if generatorNumber mod ingestorCount == ingestorNumber, then
ingestorNumber handles generatorNumber files). That way, the data from a single generator
could not arrive into Accumulo out of order, because a single ingestor was responsible for
data from any given generator. But, we can still have multiple ingestors, for a realistic
test case.

> Need a bulk loading test equivalent to continuous ingest
> --------------------------------------------------------
>                 Key: ACCUMULO-4749
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: test
>            Reporter: Ivan Bella
>            Assignee: Jared R
>              Labels: pull-request-available
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
> There are some known cases at least in past versions where bulk loading may fail leaving
the ~blip in place but no transaction left to handle it.  This will result in directories
of files being left around that are not loaded.  We should create a continuous ingest variant
that uses bulk loading instead.  Then if this is run with agitation, the continuous ingest
verification can find data that has been essentially orphaned.

This message was sent by Atlassian JIRA

View raw message