accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-2905) Seeing short hold times and no MinC's
Date Mon, 16 Jun 2014 21:30:05 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032971#comment-14032971
] 

ASF subversion and git services commented on ACCUMULO-2905:
-----------------------------------------------------------

Commit a61a795191f5c35fe7e74cdabaad0c129e291fdb in accumulo's branch refs/heads/1.5.2-SNAPSHOT
from [~ecn]
[ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=a61a795 ]

ACCUMULO-2905 start multiple MinCs, starting with the largest

ACCUMULO-2905 updates based on [~elserj]'s review

ACCUMULO-2905 improved custom map

ACCUMULO-2905 more review feedback, formatted

ACCUMULO-2905 schedule fewer idle compactions

ACCUMULO-2905 improve javadocs based on rb comments

Conflicts:

	server/src/main/java/org/apache/accumulo/server/tabletserver/LargestFirstMemoryManager.java


> Seeing short hold times and no MinC's
> -------------------------------------
>
>                 Key: ACCUMULO-2905
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2905
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.5.0, 1.5.1, 1.6.0
>         Environment: continuous ingest test
>            Reporter: Eric Newton
>            Assignee: Eric Newton
>            Priority: Minor
>             Fix For: 1.5.2, 1.6.1, 1.7.0
>
>         Attachments: SlowToMinC.png
>
>
> Started continuous ingest on a single node cluster to investigate oscillating ingest
rates.  Set the split size to 200M, and fired up 3 ingesters.  I increase the major/minor
compactors to 24/12 respectively.
> The native map was set to 4G.
> When the table got to 256 tablets, ingest was constantly showing hold times.  MinC would
run, but memory would constantly fill before new MinCs would start.  Here's some output from
the log:
> {noformat}
> $ tail -F ../../../logs/tserver_host.debug.log  | egrep 'Up|Got|held|MinC \['
> 2014-06-13 14:51:56,553 [tablet.DatafileManager] TABLET_HIST: 2;5d104;5cf07 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00003kc/F0000qtk.rf
> 2014-06-13 14:51:56,695 [tablet.DatafileManager] TABLET_HIST: 2;11cb1;11ab MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000eww/F0000qtl.rf
> 2014-06-13 14:51:56,861 [tablet.DatafileManager] TABLET_HIST: 2;0e4a3;0e2a3 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000ery/F0000qtm.rf
> 2014-06-13 14:51:56,927 [tablet.DatafileManager] TABLET_HIST: 2;44907;44504 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00008q9/F0000qtn.rf
> 2014-06-13 14:51:57,019 [tablet.DatafileManager] TABLET_HIST: 2;4a0fa;49cfd MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00001lg/F0000qto.rf
> 2014-06-13 14:51:57,535 [tablet.DatafileManager] TABLET_HIST: 2;4c0ff;4bd03 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00000ut/F0000qtq.rf
> 2014-06-13 14:51:58,443 [tablet.DatafileManager] TABLET_HIST: 2;770cf;76ed2f MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00003lv/F0000qts.rf
> 2014-06-13 14:51:58,448 [tserver.TabletServerResourceManager] DEBUG: Commits held for
  2.18 secs
> 2014-06-13 14:51:58,449 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 225 in 1.669s,
at=[0 0 0.00 1] ft=1.668s(pt=1.667s lt=0.001s ct=0.000s)
> 2014-06-13 14:51:58,790 [tablet.DatafileManager] TABLET_HIST: 2;4ad04b;4a902 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000f05/F0000qtt.rf
> 2014-06-13 14:51:59,113 [tablet.DatafileManager] TABLET_HIST: 2;23af3;238f4 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000ei6/F0000qtw.rf
> 2014-06-13 14:52:00,545 [tablet.DatafileManager] TABLET_HIST: 2;4d906e;4d508 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00008s0/F0000qtx.rf
> 2014-06-13 14:52:00,599 [tablet.DatafileManager] TABLET_HIST: 2;1bcb4f;1b8b6 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000f9v/F0000qty.rf
> 2014-06-13 14:52:00,640 [tablet.DatafileManager] TABLET_HIST: 2;1d8be;1d4c1 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00008j7/F0000qtz.rf
> 2014-06-13 14:52:00,648 [tablet.DatafileManager] TABLET_HIST: 2;40ce3;408de MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000etz/F0000qu0.rf
> 2014-06-13 14:52:01,040 [tablet.DatafileManager] TABLET_HIST: 2;2c13;2bf32 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00000uc/F0000qu5.rf
> 2014-06-13 14:52:01,939 [tablet.DatafileManager] TABLET_HIST: 2;5fedd;5fce3 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000fir/F0000qu7.rf
> 2014-06-13 14:52:02,063 [tablet.DatafileManager] TABLET_HIST: 2;26d0a;26907 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000eum/F0000qu8.rf
> 2014-06-13 14:52:02,067 [tserver.TabletServerResourceManager] DEBUG: Commits held for
  3.19 secs
> 2014-06-13 14:52:02,069 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 288 in 2.617s,
at=[0 0 0.00 1] ft=2.616s(pt=2.614s lt=0.001s ct=0.001s)
> 2014-06-13 14:52:02,604 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:58940 279,538
in 9.656s, at=[0 1 0.00 899] ft=9.274s(pt=7.516s lt=1.099s ct=0.659s)
> 2014-06-13 14:52:02,700 [tablet.DatafileManager] TABLET_HIST: 2;758e6;756e8 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00008py/F0000qub.rf
> 2014-06-13 14:52:02,765 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:58937 279,538
in 6.735s, at=[0 1 0.00 899] ft=6.446s(pt=4.446s lt=1.359s ct=0.641s)
> 2014-06-13 14:52:02,821 [tablet.DatafileManager] TABLET_HIST: 2;53b18;53918 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000kk7/F0000quc.rf
> 2014-06-13 14:52:03,503 [tablet.DatafileManager] TABLET_HIST: 2;116af;114af MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000jnw/F0000qud.rf
> 2014-06-13 14:52:03,554 [tablet.DatafileManager] TABLET_HIST: 2;76ed2f;76cd7 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000ju3/F0000que.rf
> 2014-06-13 14:52:04,197 [tablet.DatafileManager] TABLET_HIST: 2;7da4e;7d851 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000igh/F0000quf.rf
> 2014-06-13 14:52:04,603 [tablet.DatafileManager] TABLET_HIST: 2;38d03;388fe MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000evr/F0000quh.rf
> 2014-06-13 14:52:05,119 [tablet.DatafileManager] TABLET_HIST: 2;156d1f;154d1e MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000jqx/F0000quj.rf
> 2014-06-13 14:52:05,577 [tablet.DatafileManager] TABLET_HIST: 2;3fada;3f8db MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000pre/F0000qul.rf
> 2014-06-13 14:52:05,582 [tserver.TabletServerResourceManager] DEBUG: Commits held for
  3.29 secs
> 2014-06-13 14:52:05,583 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 252 in 2.512s,
at=[0 0 0.00 1] ft=2.511s(pt=2.510s lt=0.001s ct=0.000s)
> 2014-06-13 14:52:06,206 [tablet.DatafileManager] TABLET_HIST: 2;4550e;4510b MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000ex0/F0000qum.rf
> 2014-06-13 14:52:06,360 [tablet.DatafileManager] TABLET_HIST: 2;7fe06;7fc0b MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000hyp/F0000qun.rf
> 2014-06-13 14:52:06,642 [tablet.DatafileManager] TABLET_HIST: 2;238f4;236f4 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00008i1/F0000quq.rf
> 2014-06-13 14:52:08,626 [tablet.DatafileManager] TABLET_HIST: 2;5931a;59119 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000fy8/F0000qus.rf
> 2014-06-13 14:52:08,682 [tablet.DatafileManager] TABLET_HIST: 2;4a902;4a5 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00008kd/F0000quu.rf
> 2014-06-13 14:52:08,740 [tablet.DatafileManager] TABLET_HIST: 2;1dcbe;1d8be MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000etq/F0000quv.rf
> 2014-06-13 14:52:08,748 [tablet.DatafileManager] TABLET_HIST: 2;23ef2;23cf2 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000ei9/F0000qut.rf
> 2014-06-13 14:52:09,215 [tablet.DatafileManager] TABLET_HIST: 2;0fa97;0f899 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000fgk/F0000quy.rf
> 2014-06-13 14:52:09,623 [tablet.DatafileManager] TABLET_HIST: 2;0101f;00e1df MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00003ka/F0000qv0.rf
> 2014-06-13 14:52:09,626 [tserver.TabletServerResourceManager] DEBUG: Commits held for
  3.81 secs
> 2014-06-13 14:52:09,628 [tserver.TabletServer] DEBUG: UpSess 127.0.0.1:33363 250 in 3.043s,
at=[0 0 0.00 1] ft=3.042s(pt=3.041s lt=0.000s ct=0.001s)
> 2014-06-13 14:52:10,256 [tablet.DatafileManager] TABLET_HIST: 2;29d21;2991f MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000f6f/F0000qv2.rf
> 2014-06-13 14:52:10,388 [tablet.DatafileManager] TABLET_HIST: 2;6f6f3;6f4f4 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000pr5/F0000qv4.rf
> 2014-06-13 14:52:10,879 [tablet.DatafileManager] TABLET_HIST: 2;16cc5;168c8 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000fb5/F0000qv5.rf
> 2014-06-13 14:52:11,379 [tablet.DatafileManager] TABLET_HIST: 2;4b104;4ad04b MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00003lg/F0000qv7.rf
> 2014-06-13 14:52:11,879 [tablet.DatafileManager] TABLET_HIST: 2;6f8f;6f6f3 MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-00008mo/F0000qv9.rf
> 2014-06-13 14:52:12,395 [tablet.DatafileManager] TABLET_HIST: 2;3f4e1;3f2e3e MinC [memory]
-> hdfs://host:9000/accumulo/tables/2/t-0000fb1/F0000qva.rf
> {noformat}
> Perhaps the server should be more aggressive with MinC's to get better ingest performance?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message