Return-Path: X-Original-To: apmail-accumulo-notifications-archive@minotaur.apache.org Delivered-To: apmail-accumulo-notifications-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AB2AD11A7D for ; Mon, 16 Jun 2014 21:30:11 +0000 (UTC) Received: (qmail 12901 invoked by uid 500); 16 Jun 2014 21:30:11 -0000 Delivered-To: apmail-accumulo-notifications-archive@accumulo.apache.org Received: (qmail 12871 invoked by uid 500); 16 Jun 2014 21:30:11 -0000 Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: jira@apache.org Delivered-To: mailing list notifications@accumulo.apache.org Received: (qmail 12799 invoked by uid 99); 16 Jun 2014 21:30:11 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Jun 2014 21:30:11 +0000 Date: Mon, 16 Jun 2014 21:30:11 +0000 (UTC) From: "ASF subversion and git services (JIRA)" To: notifications@accumulo.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (ACCUMULO-2905) Seeing short hold times and no MinC's MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/ACCUMULO-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032987#comment-14032987 ] ASF subversion and git services commented on ACCUMULO-2905: ----------------------------------------------------------- Commit 399364bca0e8ccc81dba6b13be7cabd0ba48f72d in accumulo's branch refs/heads/master from [~ecn] [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=399364b ] ACCUMULO-2905 start multiple MinCs, starting with the largest ACCUMULO-2905 updates based on [~elserj]'s review ACCUMULO-2905 improved custom map ACCUMULO-2905 more review feedback, formatted ACCUMULO-2905 schedule fewer idle compactions ACCUMULO-2905 improve javadocs based on rb comments > Seeing short hold times and no MinC's > ------------------------------------- > > Key: ACCUMULO-2905 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2905 > Project: Accumulo > Issue Type: Bug > Components: tserver > Affects Versions: 1.5.0, 1.5.1, 1.6.0 > Environment: continuous ingest test > Reporter: Eric Newton > Assignee: Eric Newton > Priority: Minor > Fix For: 1.5.2, 1.6.1, 1.7.0 > > Attachments: SlowToMinC.png > > > Started continuous ingest on a single node cluster to investigate oscillating ingest rates. Set the split size to 200M, and fired up 3 ingesters. I increase the major/minor compactors to 24/12 respectively. > The native map was set to 4G. > When the table got to 256 tablets, ingest was constantly showing hold times. MinC would run, but memory would constantly fill before new MinCs would start. Here's some output from the log: > {noformat} > $ tail -F ../../../logs/tserver_host.debug.log | egrep 'Up|Got|held|MinC \[' > 2014-06-13 14:51:56,553 [tablet.DatafileManager] TABLET_HIST: 2;5d104;5cf07 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003kc/F0000qtk.rf > 2014-06-13 14:51:56,695 [tablet.DatafileManager] TABLET_HIST: 2;11cb1;11ab MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000eww/F0000qtl.rf > 2014-06-13 14:51:56,861 [tablet.DatafileManager] TABLET_HIST: 2;0e4a3;0e2a3 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ery/F0000qtm.rf > 2014-06-13 14:51:56,927 [tablet.DatafileManager] TABLET_HIST: 2;44907;44504 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008q9/F0000qtn.rf > 2014-06-13 14:51:57,019 [tablet.DatafileManager] TABLET_HIST: 2;4a0fa;49cfd MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00001lg/F0000qto.rf > 2014-06-13 14:51:57,535 [tablet.DatafileManager] TABLET_HIST: 2;4c0ff;4bd03 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00000ut/F0000qtq.rf > 2014-06-13 14:51:58,443 [tablet.DatafileManager] TABLET_HIST: 2;770cf;76ed2f MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003lv/F0000qts.rf > 2014-06-13 14:51:58,448 [tserver.TabletServerResourceManager] DEBUG: Commits held for 2.18 secs > 2014-06-13 14:51:58,449 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 225 in 1.669s, at=[0 0 0.00 1] ft=1.668s(pt=1.667s lt=0.001s ct=0.000s) > 2014-06-13 14:51:58,790 [tablet.DatafileManager] TABLET_HIST: 2;4ad04b;4a902 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f05/F0000qtt.rf > 2014-06-13 14:51:59,113 [tablet.DatafileManager] TABLET_HIST: 2;23af3;238f4 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ei6/F0000qtw.rf > 2014-06-13 14:52:00,545 [tablet.DatafileManager] TABLET_HIST: 2;4d906e;4d508 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008s0/F0000qtx.rf > 2014-06-13 14:52:00,599 [tablet.DatafileManager] TABLET_HIST: 2;1bcb4f;1b8b6 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f9v/F0000qty.rf > 2014-06-13 14:52:00,640 [tablet.DatafileManager] TABLET_HIST: 2;1d8be;1d4c1 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008j7/F0000qtz.rf > 2014-06-13 14:52:00,648 [tablet.DatafileManager] TABLET_HIST: 2;40ce3;408de MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000etz/F0000qu0.rf > 2014-06-13 14:52:01,040 [tablet.DatafileManager] TABLET_HIST: 2;2c13;2bf32 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00000uc/F0000qu5.rf > 2014-06-13 14:52:01,939 [tablet.DatafileManager] TABLET_HIST: 2;5fedd;5fce3 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fir/F0000qu7.rf > 2014-06-13 14:52:02,063 [tablet.DatafileManager] TABLET_HIST: 2;26d0a;26907 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000eum/F0000qu8.rf > 2014-06-13 14:52:02,067 [tserver.TabletServerResourceManager] DEBUG: Commits held for 3.19 secs > 2014-06-13 14:52:02,069 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 288 in 2.617s, at=[0 0 0.00 1] ft=2.616s(pt=2.614s lt=0.001s ct=0.001s) > 2014-06-13 14:52:02,604 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:58940 279,538 in 9.656s, at=[0 1 0.00 899] ft=9.274s(pt=7.516s lt=1.099s ct=0.659s) > 2014-06-13 14:52:02,700 [tablet.DatafileManager] TABLET_HIST: 2;758e6;756e8 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008py/F0000qub.rf > 2014-06-13 14:52:02,765 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:58937 279,538 in 6.735s, at=[0 1 0.00 899] ft=6.446s(pt=4.446s lt=1.359s ct=0.641s) > 2014-06-13 14:52:02,821 [tablet.DatafileManager] TABLET_HIST: 2;53b18;53918 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000kk7/F0000quc.rf > 2014-06-13 14:52:03,503 [tablet.DatafileManager] TABLET_HIST: 2;116af;114af MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000jnw/F0000qud.rf > 2014-06-13 14:52:03,554 [tablet.DatafileManager] TABLET_HIST: 2;76ed2f;76cd7 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ju3/F0000que.rf > 2014-06-13 14:52:04,197 [tablet.DatafileManager] TABLET_HIST: 2;7da4e;7d851 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000igh/F0000quf.rf > 2014-06-13 14:52:04,603 [tablet.DatafileManager] TABLET_HIST: 2;38d03;388fe MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000evr/F0000quh.rf > 2014-06-13 14:52:05,119 [tablet.DatafileManager] TABLET_HIST: 2;156d1f;154d1e MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000jqx/F0000quj.rf > 2014-06-13 14:52:05,577 [tablet.DatafileManager] TABLET_HIST: 2;3fada;3f8db MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000pre/F0000qul.rf > 2014-06-13 14:52:05,582 [tserver.TabletServerResourceManager] DEBUG: Commits held for 3.29 secs > 2014-06-13 14:52:05,583 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 252 in 2.512s, at=[0 0 0.00 1] ft=2.511s(pt=2.510s lt=0.001s ct=0.000s) > 2014-06-13 14:52:06,206 [tablet.DatafileManager] TABLET_HIST: 2;4550e;4510b MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ex0/F0000qum.rf > 2014-06-13 14:52:06,360 [tablet.DatafileManager] TABLET_HIST: 2;7fe06;7fc0b MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000hyp/F0000qun.rf > 2014-06-13 14:52:06,642 [tablet.DatafileManager] TABLET_HIST: 2;238f4;236f4 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008i1/F0000quq.rf > 2014-06-13 14:52:08,626 [tablet.DatafileManager] TABLET_HIST: 2;5931a;59119 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fy8/F0000qus.rf > 2014-06-13 14:52:08,682 [tablet.DatafileManager] TABLET_HIST: 2;4a902;4a5 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008kd/F0000quu.rf > 2014-06-13 14:52:08,740 [tablet.DatafileManager] TABLET_HIST: 2;1dcbe;1d8be MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000etq/F0000quv.rf > 2014-06-13 14:52:08,748 [tablet.DatafileManager] TABLET_HIST: 2;23ef2;23cf2 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000ei9/F0000qut.rf > 2014-06-13 14:52:09,215 [tablet.DatafileManager] TABLET_HIST: 2;0fa97;0f899 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fgk/F0000quy.rf > 2014-06-13 14:52:09,623 [tablet.DatafileManager] TABLET_HIST: 2;0101f;00e1df MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003ka/F0000qv0.rf > 2014-06-13 14:52:09,626 [tserver.TabletServerResourceManager] DEBUG: Commits held for 3.81 secs > 2014-06-13 14:52:09,628 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 250 in 3.043s, at=[0 0 0.00 1] ft=3.042s(pt=3.041s lt=0.000s ct=0.001s) > 2014-06-13 14:52:10,256 [tablet.DatafileManager] TABLET_HIST: 2;29d21;2991f MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000f6f/F0000qv2.rf > 2014-06-13 14:52:10,388 [tablet.DatafileManager] TABLET_HIST: 2;6f6f3;6f4f4 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000pr5/F0000qv4.rf > 2014-06-13 14:52:10,879 [tablet.DatafileManager] TABLET_HIST: 2;16cc5;168c8 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fb5/F0000qv5.rf > 2014-06-13 14:52:11,379 [tablet.DatafileManager] TABLET_HIST: 2;4b104;4ad04b MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00003lg/F0000qv7.rf > 2014-06-13 14:52:11,879 [tablet.DatafileManager] TABLET_HIST: 2;6f8f;6f6f3 MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-00008mo/F0000qv9.rf > 2014-06-13 14:52:12,395 [tablet.DatafileManager] TABLET_HIST: 2;3f4e1;3f2e3e MinC [memory] -> hdfs://host:9000/accumulo/tables/2/t-0000fb1/F0000qva.rf > {noformat} > Perhaps the server should be more aggressive with MinC's to get better ingest performance? -- This message was sent by Atlassian JIRA (v6.2#6252)