accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser" <josh.el...@gmail.com>
Subject Re: Review Request 19790: ACCUMULO-378 Design document
Date Mon, 31 Mar 2014 00:10:41 GMT


> On March 28, 2014, 6:53 p.m., kturner wrote:
> > docs/src/main/resources/design/ACCUMULO-378-design.mdtext, line 119
> > <https://reviews.apache.org/r/19790/diff/1/?file=539855#file539855line119>
> >
> >     Seems like accumulo should have a public API for querying what needs to be replicated,
notifying it when something has been replicated, and methods for importing replicated data.
 I am thinking of something different than a plugin, more like the import/export table API.
 How the replication happens is up the user.  We could provide a default implementation that
does replication as you mentioned.  Some users may want to occassionally replicate large batches
using map reduce.  Others may want to continually replicate files using distributed queueing
solutions.

My initial thoughts were to provide something at a public api layer due to the likely desire
to integrate WALs as a part of said API. Opening up an API might prove difficult to implement
well -- we would have to design something that scales out to adequately support the ingest
rates Accumulo will support.

Not saying I'm against it, but it would be difficult to get right. Hooking into it would also
likely be difficult to implement.


> On March 28, 2014, 6:53 p.m., kturner wrote:
> > docs/src/main/resources/design/ACCUMULO-378-design.mdtext, line 139
> > <https://reviews.apache.org/r/19790/diff/1/?file=539855#file539855line139>
> >
> >     A walog or bulk imported file could be referenced by multiple tablets.  I am
wondering if it would be better to move this info out of the tablet and do something like
~del markers in the metadata table.  Like a ~repl_hdfs://foo/a.rf row in the metadata table.
 This row could store replication status.  If the ~repl row exist, then file would not be
deleted.  The ~repl marker could not be removed until the file is replicated and there are
no more refs in the tablet metadata (is this sufficient to prevent addint a repl marker for
something that already replicated).  Could possibly update repl markers using conditional
mutations, since multiple tablets and the master may mutate it.

Yeah, this ties into what Mike had asked about. Having it in a completely separate table would
be best from a "screwing up other things" perspective. I don't have any example of why these
markers would need to be in the same row as the tablets. I need to read some of that code
again.


> On March 28, 2014, 6:53 p.m., kturner wrote:
> > docs/src/main/resources/design/ACCUMULO-378-design.mdtext, line 157
> > <https://reviews.apache.org/r/19790/diff/1/?file=539855#file539855line157>
> >
> >     Are you thinking a FATE operation per file?  FATE uses zookeeper, and zookeeper
keeps everything in memory.

Not sure. Using FATE when appropriate is mostly what I was thinking of right now - I don't
have explicit examples of where we would want to use FATE. The obvious place is that we don't
want multiple hosts sending the same data more than once, but we also want to make sure we
re-send data that failed to send the first time around.

Some more thought is needed here, I believe.


> On March 28, 2014, 6:53 p.m., kturner wrote:
> > docs/src/main/resources/design/ACCUMULO-378-design.mdtext, line 184
> > <https://reviews.apache.org/r/19790/diff/1/?file=539855#file539855line184>
> >
> >     How will the locality group config changing be handled?

Given the current replication configuration, the updated locality group configuration would
use that replication configuration to determine which new data should be replicated (and to
where). Splitting out the data from the WAL into aforementioned replication records may prove
to be difficult, at which point, I may drop down to only supporting table-wide replication
rules.

Supporting locality group replication might require changes in how WALs work before this really
becomes feasible.


> On March 28, 2014, 6:53 p.m., kturner wrote:
> > docs/src/main/resources/design/ACCUMULO-378-design.mdtext, line 254
> > <https://reviews.apache.org/r/19790/diff/1/?file=539855#file539855line254>
> >
> >     Not gonna work.  It may be worthwhile to consider having the ConditionalWriter
detect unsupported replication configurations and throw an exception.

Haha, I figured this to be the case. Throwing an exception is what I had planned.


> On March 28, 2014, 6:53 p.m., kturner wrote:
> > docs/src/main/resources/design/ACCUMULO-378-design.mdtext, line 261
> > <https://reviews.apache.org/r/19790/diff/1/?file=539855#file539855line261>
> >
> >     Tablets could support an atomic operation that marks all of its current files
as needing replication and appropriately handle new data coming in.  The master would go through
all tablets in a table calling this operation.  Tablets could write something to the metadata
table when the operation is successful.  This allows the master to know which tablets are
done.

That's a possibility. Like you mentioned earlier, depending on the amount of data to be replicated,
exporttable and distcp might be (wildly) more efficient. Worth it to try to do this now, or
leave pre-existing data replication as a follow-on?


- Josh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19790/#review38927
-----------------------------------------------------------


On March 28, 2014, 5:54 p.m., kturner wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19790/
> -----------------------------------------------------------
> 
> (Updated March 28, 2014, 5:54 p.m.)
> 
> 
> Review request for accumulo.
> 
> 
> Bugs: ACCUMULO-378
>     https://issues.apache.org/jira/browse/ACCUMULO-378
> 
> 
> Repository: accumulo
> 
> 
> Description
> -------
> 
> ACCUMULO-378 Design document.  Posting for review here, not meant for commit.  Final
version of document should be posted on issue.
> 
> 
> Diffs
> -----
> 
>   docs/src/main/resources/design/ACCUMULO-378-design.mdtext PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/19790/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> kturner
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message