accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bob.Thor...@l-3com.com
Subject RE: Time delay question
Date Wed, 08 Aug 2012 19:08:44 GMT
So the only tunable elements in the BatchWriter are memory size and
timeout?  Otherwise the data is available for the next scanner.  Thanks
for the help!

 

From: John Vines [mailto:vines@apache.org] 
Sent: Wednesday, August 08, 2012 14:06
To: user@accumulo.apache.org
Subject: Re: Time delay question

 

Once BatchWriter's return from flush() or when they automatically do a
flush in the background (when their buffers get sufficiently large), the
data is availble for scanners. At that point the data is at least in the
in memory map, as well as the walog.

As the in memory maps for the tablets for the table fill, they will
write to disk. And when there are more files of given sizes than
desired, major compaction will occur which will compact files to a file.

So when you scan, data is merged from the in memory map and the files.

John

On Wed, Aug 8, 2012 at 2:09 PM, Keith Turner <keith@deenlo.com> wrote:

Once the batch writer flushes, the data is available for read by the
batch scanner.   On the server side data is read from memory and
files.


On Wed, Aug 8, 2012 at 1:53 PM,  <Bob.Thorman@l-3com.com> wrote:
> I have a set of actors that are doing batchWriters and another set
doing
> batchScanner(s) and I'm getting questions from my users about the time
> delay between them.  So my question is, what are all the tunable
> parameters between writes and reads?
>
> Do I have the following facts correct?
>
> BatchWriters have a time value for flushing to memory cache
> Minor compaction has its time frequency for flushing to WALogs
> Major compaction has its time frequency for flushing to disc
>
> So where all does a batchScanner get its data from?
>
> Any/all help is appreciated.
>
>
> Bob Thorman
> Engineering Fellow
> L-3 Communications, ComCept
> 1700 Science Place
> Rockwall, TX 75032
> (972) 772-7501 <tel:%28972%29%20772-7501>  work
> Bob.Thorman@ncct.af.smil.mil
> rdthorm@nsa.ic.gov
>
>

 


Mime
View raw message