Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0F43C10587 for ; Thu, 26 Sep 2013 15:14:38 +0000 (UTC) Received: (qmail 27105 invoked by uid 500); 26 Sep 2013 15:14:37 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 27068 invoked by uid 500); 26 Sep 2013 15:14:36 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 27060 invoked by uid 99); 26 Sep 2013 15:14:35 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Sep 2013 15:14:35 +0000 Received: from localhost (HELO mail-lb0-f180.google.com) (127.0.0.1) (smtp-auth username vines, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Sep 2013 15:14:35 +0000 Received: by mail-lb0-f180.google.com with SMTP id q8so1185348lbi.39 for ; Thu, 26 Sep 2013 08:14:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=wh/HRN/1DAgY7FmYGz23UjXKkQosf0C9SYwbMteZds8=; b=nNYzVClFic13wERGUawjXNrwUAusermYoannNV7Pohb4Vp6QT+AJSnfcOrpG59e0ng 0+njmjBlDRY/CpiBei7YsY9Smof6/lSTb3XhMxjrxt49GfRhWSpnCfHiQphTeOBSXtqK pyE5uhfYR2L3DYhV/B0u+CRGAQ7RAey7agL44kHCxhz1Ach1nAoUNVfSyK5FU/ZJqNHB 7s3+cc+c+NAfDsyJ0yVR0PwHClncHw1yCGO3ocSku3Y+No0kJ+tFEmi+iJN9ROggo8LS d9Gpoy+MNHYxbAZz1vnIwBJZjIz6czsv+O8SbeRhCJHZ/9hPV0J2Mi3X/HFuIlkUkyms aAEQ== X-Received: by 10.112.29.17 with SMTP id f17mr1492777lbh.45.1380208473072; Thu, 26 Sep 2013 08:14:33 -0700 (PDT) MIME-Version: 1.0 Reply-To: vines@apache.org Received: by 10.114.230.231 with HTTP; Thu, 26 Sep 2013 08:13:52 -0700 (PDT) In-Reply-To: References: From: John Vines Date: Thu, 26 Sep 2013 11:13:52 -0400 Message-ID: Subject: Re: WALOG Design To: "user@accumulo.apache.org" Content-Type: multipart/alternative; boundary=001a1133c690c0eacc04e74ad3d8 --001a1133c690c0eacc04e74ad3d8 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable The in memory map gets divided to be used by all existing tablets on the tserver. It has a manager which I think is pluggable (at one point it was) which tries to be intelligent in allocating space to tablets which need/are using space vs. idle tablets. But when a tablet's portion of the in memory map is full, their entire portion of the map gets written at once. So you can sort of manage minor compact files size by manipulating the size of the in memory map, the number of tablets on a tserver, and this memory manager. If the tserver has space, I think it gives the tablet a new hunk for it to start to use to keep things streamlined. However, if things are really backed up and memory is unable to be provided, the mutations will be rejected and this error will propagate back to clients. On Thu, Sep 26, 2013 at 11:04 AM, Slater, David M. wrote: > Awesome, thanks! > > For clarification, when the in memory map is full, if it contains data fo= r > multiple tablets, how does it prioritize which tablets to do minor > compactions for? Is there a separate in memory map for each tablet on a > tablet server? When the in memory map is full, will it do minor compactio= ns > until all of the data currently in it is empty, or will it trigger a > smaller number of minor compactions until it is at a more reasonable size= ? > > It is unclear to me if you can change the minimum size of rfiles written > by minor compactions. > > Also, how does the in memory map handle mutations for tablets that are > currently doing a minor compaction? > > Best, > David > > -----Original Message----- > From: Christopher [mailto:ctubbsii@apache.org] > Sent: Wednesday, September 25, 2013 4:49 PM > To: Accumulo User List > Subject: Re: WALOG Design > > Mutations are written to WALOGS when they are inserted into a TServer's > in-memory map. The TServer's in-memory map gets flushed to disk > periodically, but there's a risk that the TServer will die after the data > has been ingested, but before it is flushed to disk. The WALOGS, when > enabled, protect against this data loss, by first writing out incoming da= ta > to a WALOG. The WALOG is more efficient than creating RFiles, because it > does not contain sorted data or indexes. > It's just a playback file, so that in case of a failure, Mutations that > the client believed had been ingested, aren't lost. > > Putting the WALOG in memory defeats the purpose of the WALOG, but it can > be disabled (per-table), if you care more about performance than protecti= on > against data loss. Don't disable it for the !METADATA table, though... > > You can generate RFiles directly (perhaps using a M/R job), and bypass th= e > WALOG, and bulk import them into Accumulo. > > -- > Christopher L Tubbs II > http://gravatar.com/ctubbsii > > > On Wed, Sep 25, 2013 at 4:39 PM, Slater, David M. > wrote: > > First, thank you all for the responses on my BatchWriter question, as > > I was able to increase my ingestion rate by a large factor. I am now > > hitting disk i/o limits, which is forcing me to look at reducing file > > copying. My primary thoughts concerning this are reducing the hadoop > > replication factor as well as reducing the number of major compactions. > > > > However, from what I understand about write ahead logs (in 1.4), even > > if you remove all major compactions, all data will essentially be > > written to disk > > twice: once to the WALOG in the local directory (HDFS is 1.5), then > > from the WALOG to an RFile on HDFS. Is this understanding correct? > > > > I=92m trying to understand what the primary reasons are for having the > WALOG. > > > > Is there any way to write directly to an RFile from the In-Memory Map > > (or have the WALOG in memory)? > > > > > > > > Thanks, > > David > --001a1133c690c0eacc04e74ad3d8 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
The in memory map gets divided to be used by all existing = tablets on the tserver. It has a manager which I think is pluggable (at one= point it was) which tries to be intelligent in allocating space to tablets= which need/are using space vs. idle tablets. But when a tablet's porti= on of the in memory map is full, their entire portion of the map gets writt= en at once. So you can sort of manage minor compact files size by manipulat= ing the size of the in memory map, the number of tablets on a tserver, and = this memory manager.

If the tserver has space, I think it gives the tablet a new = hunk for it to start to use to keep things streamlined. However, if things = are really backed up and memory is unable to be provided, the mutations wil= l be rejected and this error will propagate back to clients.


On Thu,= Sep 26, 2013 at 11:04 AM, Slater, David M. <David.Slater@jhuapl.e= du> wrote:
Awesome, thanks!

For clarification, when the in memory map is full, if it contains data for = multiple tablets, how does it prioritize which tablets to do minor compacti= ons for? Is there a separate in memory map for each tablet on a tablet serv= er? When the in memory map is full, will it do minor compactions until all = of the data currently in it is empty, or will it trigger a smaller number o= f minor compactions until it is at a more reasonable size?

It is unclear to me if you can change the minimum size of rfiles written by= minor compactions.

Also, how does the in memory map handle mutations for tablets that are curr= ently doing a minor compaction?

Best,
David

-----Original Message-----
From: Christopher [mailto:ctubbsii@a= pache.org]
Sent: Wednesday, September 25, 2013 4:49 PM
To: Accumulo User List
Subject: Re: WALOG Design

Mutations are written to WALOGS when they are inserted into a TServer's= in-memory map. The TServer's in-memory map gets flushed to disk period= ically, but there's a risk that the TServer will die after the data has= been ingested, but before it is flushed to disk. The WALOGS, when enabled,= protect against this data loss, by first writing out incoming data to a WA= LOG. The WALOG is more efficient than creating RFiles, because it does not = contain sorted data or indexes.
It's just a playback file, so that in case of a failure, Mutations that= the client believed had been ingested, aren't lost.

Putting the WALOG in memory defeats the purpose of the WALOG, but it can be= disabled (per-table), if you care more about performance than protection a= gainst data loss. Don't disable it for the !METADATA table, though...
You can generate RFiles directly (perhaps using a M/R job), and bypass the = WALOG, and bulk import them into Accumulo.

--
Christopher L Tubbs II
http://gravatar.= com/ctubbsii


On Wed, Sep 25, 2013 at 4:39 PM, Slater, David M.
<David.Slater@jhuapl.edu&= gt; wrote:
> First, thank you all for the responses on my BatchWriter question, as<= br> > I was able to increase my ingestion rate by a large factor. I am now > hitting disk i/o limits, which is forcing me to look at reducing file<= br> > copying. My primary thoughts concerning this are reducing the hadoop > replication factor as well as reducing the number of major compactions= .
>
> However, from what I understand about write ahead logs (in 1.4), even<= br> > if you remove all major compactions, all data will essentially be
> written to disk
> twice: once to the WALOG in the local directory (HDFS is 1.5), then > from the WALOG to an RFile on HDFS. Is this understanding correct?
>
> I=92m trying to understand what the primary reasons are for having the= WALOG.
>
> Is there any way to write directly to an RFile from the In-Memory Map<= br> > (or have the WALOG in memory)?
>
>
>
> Thanks,
> David

--001a1133c690c0eacc04e74ad3d8--