Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6CEDA10638 for ; Thu, 26 Sep 2013 15:33:58 +0000 (UTC) Received: (qmail 84136 invoked by uid 500); 26 Sep 2013 15:33:56 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 84075 invoked by uid 500); 26 Sep 2013 15:33:55 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 84058 invoked by uid 99); 26 Sep 2013 15:33:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Sep 2013 15:33:54 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of billie.rinaldi@gmail.com designates 209.85.192.182 as permitted sender) Received: from [209.85.192.182] (HELO mail-pd0-f182.google.com) (209.85.192.182) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Sep 2013 15:33:50 +0000 Received: by mail-pd0-f182.google.com with SMTP id r10so1310919pdi.13 for ; Thu, 26 Sep 2013 08:33:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=hbuhFQhPfipO627A1CHohECsnDq7sMth3eVpsB4W5eU=; b=s8lGrQwTCZq5K/bcacKMetUBk3vUSHqpKnnpMF3XZ+uVPYxy5aQ5cQnkraIa4TcrNz adzrb7bkyClxMnEs9HDkGE7ZuoI2R8WYeyzcCjbOZ/eikH9O5xNXZe0R5rQ5Pl9NDXGn kYGtCTglwyq+YfQqXyshoWi4GM+9vTGd4d1Y+YMyxzbw/BKofYqzasTg1ggBBO8A/CJ3 t7ZWBJMspG0OA9IsbLD1eK+4o7FChCdajg7dXu97Rqoj1YMWRE7ZXLjtV6Wc3x3wCIai 4xUlueN2zlvj+8qcLNQr6jbVMuW5veAQ1vdSYoX8jN3AZ//pKzTRK2mcq7D7yNSLN448 Ctyg== MIME-Version: 1.0 X-Received: by 10.68.232.3 with SMTP id tk3mr1792895pbc.121.1380209610197; Thu, 26 Sep 2013 08:33:30 -0700 (PDT) Received: by 10.66.189.201 with HTTP; Thu, 26 Sep 2013 08:33:30 -0700 (PDT) In-Reply-To: References: Date: Thu, 26 Sep 2013 08:33:30 -0700 Message-ID: Subject: Re: WALOG Design From: Billie Rinaldi To: "user@accumulo.apache.org" Content-Type: multipart/alternative; boundary=047d7b33dbf28810b804e74b170e X-Virus-Checked: Checked by ClamAV on apache.org --047d7b33dbf28810b804e74b170e Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On Thu, Sep 26, 2013 at 8:04 AM, Slater, David M. wrote: > Awesome, thanks! > > For clarification, when the in memory map is full, if it contains data fo= r > multiple tablets, how does it prioritize which tablets to do minor > compactions for? The memory manager chooses a tablet to minor compact based on the tablet's memory usage and idle time. It chooses the tablet with the largest (size in bytes * 2^(minutes idle / 15)). If memory isn't full, but there are tablets meeting a per-table idleness threshold, it will select one of those to minor compact. It tries to initiate minor compactions before the memory is completely full in an attempt to avoid blocking writes during minor compactions. It uses a dynamic threshold, a percentage of the maximum memory, to decide when to initiate them. It adjusts the threshold based on how full the memory gets before a minor compaction is completed successfully. > Is there a separate in memory map for each tablet on a tablet server? Whe= n > the in memory map is full, will it do minor compactions until all of the > data currently in it is empty, or will it trigger a smaller number of min= or > compactions until it is at a more reasonable size? > > It is unclear to me if you can change the minimum size of rfiles written > by minor compactions. > > Also, how does the in memory map handle mutations for tablets that are > currently doing a minor compaction? > > Best, > David > > -----Original Message----- > From: Christopher [mailto:ctubbsii@apache.org] > Sent: Wednesday, September 25, 2013 4:49 PM > To: Accumulo User List > Subject: Re: WALOG Design > > Mutations are written to WALOGS when they are inserted into a TServer's > in-memory map. The TServer's in-memory map gets flushed to disk > periodically, but there's a risk that the TServer will die after the data > has been ingested, but before it is flushed to disk. The WALOGS, when > enabled, protect against this data loss, by first writing out incoming da= ta > to a WALOG. The WALOG is more efficient than creating RFiles, because it > does not contain sorted data or indexes. > It's just a playback file, so that in case of a failure, Mutations that > the client believed had been ingested, aren't lost. > > Putting the WALOG in memory defeats the purpose of the WALOG, but it can > be disabled (per-table), if you care more about performance than protecti= on > against data loss. Don't disable it for the !METADATA table, though... > > You can generate RFiles directly (perhaps using a M/R job), and bypass th= e > WALOG, and bulk import them into Accumulo. > > -- > Christopher L Tubbs II > http://gravatar.com/ctubbsii > > > On Wed, Sep 25, 2013 at 4:39 PM, Slater, David M. > wrote: > > First, thank you all for the responses on my BatchWriter question, as > > I was able to increase my ingestion rate by a large factor. I am now > > hitting disk i/o limits, which is forcing me to look at reducing file > > copying. My primary thoughts concerning this are reducing the hadoop > > replication factor as well as reducing the number of major compactions. > > > > However, from what I understand about write ahead logs (in 1.4), even > > if you remove all major compactions, all data will essentially be > > written to disk > > twice: once to the WALOG in the local directory (HDFS is 1.5), then > > from the WALOG to an RFile on HDFS. Is this understanding correct? > > > > I=92m trying to understand what the primary reasons are for having the > WALOG. > > > > Is there any way to write directly to an RFile from the In-Memory Map > > (or have the WALOG in memory)? > > > > > > > > Thanks, > > David > --047d7b33dbf28810b804e74b170e Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
On Thu, Sep 26, 2013 at 8:04 AM, Slater, David M. <= David.Slater@jhuapl.edu> wrote:
Awesome, thanks!

For clarification, when the in memory map is full, if it contains data for = multiple tablets, how does it prioritize which tablets to do minor compacti= ons for?

The memory manager chooses a table= t to minor compact based on the tablet's memory usage and idle time.=A0= It chooses the tablet with the largest (size in bytes * 2^(minutes idle / = 15)).=A0 If memory isn't full, but there are tablets meeting a per-tabl= e idleness threshold, it will select one of those to minor compact.=A0 It t= ries to initiate minor compactions before the memory is completely full in = an attempt to avoid blocking writes during minor compactions.=A0 It uses a = dynamic threshold, a percentage of the maximum memory, to decide when to in= itiate them.=A0 It adjusts the threshold based on how full the memory gets = before a minor compaction is completed successfully.
=A0
Is there a separate in m= emory map for each tablet on a tablet server? When the in memory map is ful= l, will it do minor compactions until all of the data currently in it is em= pty, or will it trigger a smaller number of minor compactions until it is a= t a more reasonable size?

It is unclear to me if you can change the minimum size of rfiles written by= minor compactions.

Also, how does the in memory map handle mutations for tablets that are curr= ently doing a minor compaction?

Best,
David

-----Original Message-----
From: Christopher [mailto:ctubbsii@a= pache.org]
Sent: Wednesday, September 25, 2013 4:49 PM
To: Accumulo User List
Subject: Re: WALOG Design

Mutations are written to WALOGS when they are inserted into a TServer's= in-memory map. The TServer's in-memory map gets flushed to disk period= ically, but there's a risk that the TServer will die after the data has= been ingested, but before it is flushed to disk. The WALOGS, when enabled,= protect against this data loss, by first writing out incoming data to a WA= LOG. The WALOG is more efficient than creating RFiles, because it does not = contain sorted data or indexes.
It's just a playback file, so that in case of a failure, Mutations that= the client believed had been ingested, aren't lost.

Putting the WALOG in memory defeats the purpose of the WALOG, but it can be= disabled (per-table), if you care more about performance than protection a= gainst data loss. Don't disable it for the !METADATA table, though...
You can generate RFiles directly (perhaps using a M/R job), and bypass the = WALOG, and bulk import them into Accumulo.

--
Christopher L Tubbs II
http://gravatar.= com/ctubbsii


On Wed, Sep 25, 2013 at 4:39 PM, Slater, David M.
<David.Slater@jhuapl.edu&= gt; wrote:
> First, thank you all for the responses on my BatchWriter question, as<= br> > I was able to increase my ingestion rate by a large factor. I am now > hitting disk i/o limits, which is forcing me to look at reducing file<= br> > copying. My primary thoughts concerning this are reducing the hadoop > replication factor as well as reducing the number of major compactions= .
>
> However, from what I understand about write ahead logs (in 1.4), even<= br> > if you remove all major compactions, all data will essentially be
> written to disk
> twice: once to the WALOG in the local directory (HDFS is 1.5), then > from the WALOG to an RFile on HDFS. Is this understanding correct?
>
> I=92m trying to understand what the primary reasons are for having the= WALOG.
>
> Is there any way to write directly to an RFile from the In-Memory Map<= br> > (or have the WALOG in memory)?
>
>
>
> Thanks,
> David

--047d7b33dbf28810b804e74b170e--