Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3359ADCCF for ; Wed, 8 Aug 2012 19:11:48 +0000 (UTC) Received: (qmail 8419 invoked by uid 500); 8 Aug 2012 19:11:48 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 8389 invoked by uid 500); 8 Aug 2012 19:11:48 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 8380 invoked by uid 99); 8 Aug 2012 19:11:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Aug 2012 19:11:47 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of Bob.Thorman@l-3com.com designates 166.20.51.112 as permitted sender) Received: from [166.20.51.112] (HELO smtp2out.l-3com.com) (166.20.51.112) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Aug 2012 19:11:43 +0000 X-filenames: X-filesizes: None X-filetypes: X-IronPort-AV: E=Sophos;i="4.77,734,1336348800"; d="scan'208,217";a="128701840" From: Bob.Thorman@l-3com.com Received: from generichost.l-3com.com (HELO mx1.Comcept.L-3Com.com) ([128.170.184.25]) by smtp2out.l-3com.com with ESMTP; 08 Aug 2012 19:11:21 +0000 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CD7599.98097FCD" Subject: RE: Time delay question Date: Wed, 8 Aug 2012 14:08:44 -0500 Message-ID: <97EB0FF1279CC5428640A3FB61B10BD602D47FFE@mx1.Comcept.L-3Com.com> In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Time delay question Thread-Index: Ac11mN/c3yCP+1HyQ2acrlHIHByKQwAACkIg References: <97EB0FF1279CC5428640A3FB61B10BD602D47F9B@mx1.Comcept.L-3Com.com> To: X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. ------_=_NextPart_001_01CD7599.98097FCD Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable So the only tunable elements in the BatchWriter are memory size and timeout? Otherwise the data is available for the next scanner. Thanks for the help! =20 From: John Vines [mailto:vines@apache.org]=20 Sent: Wednesday, August 08, 2012 14:06 To: user@accumulo.apache.org Subject: Re: Time delay question =20 Once BatchWriter's return from flush() or when they automatically do a flush in the background (when their buffers get sufficiently large), the data is availble for scanners. At that point the data is at least in the in memory map, as well as the walog. As the in memory maps for the tablets for the table fill, they will write to disk. And when there are more files of given sizes than desired, major compaction will occur which will compact files to a file. So when you scan, data is merged from the in memory map and the files. John On Wed, Aug 8, 2012 at 2:09 PM, Keith Turner wrote: Once the batch writer flushes, the data is available for read by the batch scanner. On the server side data is read from memory and files. On Wed, Aug 8, 2012 at 1:53 PM, wrote: > I have a set of actors that are doing batchWriters and another set doing > batchScanner(s) and I'm getting questions from my users about the time > delay between them. So my question is, what are all the tunable > parameters between writes and reads? > > Do I have the following facts correct? > > BatchWriters have a time value for flushing to memory cache > Minor compaction has its time frequency for flushing to WALogs > Major compaction has its time frequency for flushing to disc > > So where all does a batchScanner get its data from? > > Any/all help is appreciated. > > > Bob Thorman > Engineering Fellow > L-3 Communications, ComCept > 1700 Science Place > Rockwall, TX 75032 > (972) 772-7501 work > Bob.Thorman@ncct.af.smil.mil > rdthorm@nsa.ic.gov > > =20 ------_=_NextPart_001_01CD7599.98097FCD Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

So the only tunable elements in the BatchWriter are memory size and = timeout?  Otherwise the data is available for the next = scanner.  Thanks for the help!

 

From:= = John Vines [mailto:vines@apache.org]
Sent: Wednesday, August = 08, 2012 14:06
To: user@accumulo.apache.org
Subject: = Re: Time delay question

 

Once BatchWriter's return from flush() or = when they automatically do a flush in the background (when their buffers = get sufficiently large), the data is availble for scanners. At that = point the data is at least in the in memory map, as well as the = walog.

As the in memory maps for the tablets for the table fill, = they will write to disk. And when there are more files of given sizes = than desired, major compaction will occur which will compact files to a = file.

So when you scan, data is merged from the in memory map and = the files.

John

On Wed, = Aug 8, 2012 at 2:09 PM, Keith Turner <keith@deenlo.com> wrote:

Once the batch writer flushes, the data is available = for read by the
batch scanner.   On the server side data is read = from memory and
files.


On Wed, Aug 8, 2012 at 1:53 PM,  <Bob.Thorman@l-3com.com> = wrote:
> I have a set of actors that are doing batchWriters and = another set doing
> batchScanner(s) and I'm getting questions from = my users about the time
> delay between them.  So my question = is, what are all the tunable
> parameters between writes and = reads?
>
> Do I have the following facts = correct?
>
> BatchWriters have a time value for flushing to = memory cache
> Minor compaction has its time frequency for = flushing to WALogs
> Major compaction has its time frequency for = flushing to disc
>
> So where all does a batchScanner get = its data from?
>
> Any/all help is = appreciated.
>
>
> Bob Thorman
> Engineering = Fellow
> L-3 Communications, ComCept
> 1700 Science = Place
> Rockwall, TX 75032
> (972) 772-7501 work
> Bob.Thorman@ncct.af.smil.mil=
> rdthorm@nsa.ic.gov
>
>=

 

------_=_NextPart_001_01CD7599.98097FCD--