incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus Sorensen <shadow...@gmail.com>
Subject RE: [DISCUSS] Management Server Memory Requirements
Date Thu, 21 Feb 2013 06:18:16 GMT
Yeah, it will work for a little while until it starts swapping. Up until
this point most of the bugs filed have been about starting management
server or initial zone setup.
On Feb 20, 2013 11:09 PM, "Sudha Ponnaganti" <sudha.ponnaganti@citrix.com>
wrote:

> I just had a call with some of the QA Team members and starting yesterday,
> everyone seem to be running into this issue.  I haven't heard anyone
> complaining about it earlier, however only little testing has happened
> before.
>
>
> -----Original Message-----
> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> Sent: Wednesday, February 20, 2013 10:05 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [DISCUSS] Management Server Memory Requirements
>
> We began seeing this at the point of the 4.1 cut. Our xen devclouds
> stopped working until we increased the dom0 RAM to 1.5GB. Nobody else was
> complaining, and most people don't run the mgmt server in devcloud, so I
> was just waiting to see where the conversation about memory went on the
> MAVEN_OPTS e-mail thread.
>
> On Wed, Feb 20, 2013 at 10:26 PM, Sudha Ponnaganti <
> sudha.ponnaganti@citrix.com> wrote:
> > Parth / Sailaja,
> >
> > Can you update ticket with data points and see if this can be assigned
> to Alex to start investigation with Javelin merge unless this can be
> associated with a specific check-in.
> >
> > Talluri - would you be able to narrow down to the check-in or build that
> we started to see this??
> >
> > Thanks
> > /sudha
> >
> > -----Original Message-----
> > From: Parth Jagirdar [mailto:Parth.Jagirdar@citrix.com]
> > Sent: Wednesday, February 20, 2013 9:05 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Re: [DISCUSS] Management Server Memory Requirements
> >
> > Marcus,
> >
> >
> > I attempted login into UI while running the log.
> >
> > [root@localhost management]# vmstat 1
> > procs -----------memory---------- ---swap-- -----io---- --system--
> > -----cpu-----
> >  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
> id
> > wa st
> >  0  0 191132  70904  10340  16192   36   28    40    91    9    1  0  0
> 89
> > 10  0
> >  0  0 191132  70904  10340  16204    0    0     0     0   46   75  0  0
> > 100  0  0
> >  0  0 191132  70904  10356  16204    0    0     0    36   72  221  1  0
> 91
> >  8  0
> >  0  0 191132  70904  10372  16188    0    0     0    44   73  130  0  1
> 88
> > 11  0
> >  0  0 191132  70780  10420  16208    0    0     4   276   83  191  1  0
> 75
> > 24  0
> >  0  0 191132  70780  10452  16192    0    0     0   120  106  309  1  0
> 77
> > 22  0
> >  0  0 191132  70780  10468  16200    0    0     0    40   91  183  1  1
> 90
> >  8  0
> >  0  0 191132  70780  10468  16216    0    0     0     0   47  128  0  0
> > 100  0  0
> >  0  0 191132  70780  10484  16216    0    0     0    36   70  136  0  0
> 94
> >  6  0
> >  0  0 191132  70656  10500  16200    0    0     0    40   66  116  1  0
> 91
> >  8  0
> >  0  0 191132  70656  10500  16216    0    0     0     0   47   94  0  0
> > 100  0  0
> >  0  1 189504  66216  10400  17940 2192  100  3928   172  404  579  9  2
>  5
> > 84  0
> >  1  1 188772  60220  10552  21992 1000    0  5220    68  412  741  7  2
> 21
> > 69  0
> >  1  2 187352  49316   7832  30344 1660   32 10052    32  833 1015 28  3
>  0
> > 69  0
> >  0  4 188816  52420   1392  25716 1488 2872  3168  3240  663  870 19  3
>  0
> > 78  0
> >  1  1 187388  51808   1372  25040 2476 1104  3084  1260  675  813 15  3
>  0
> > 82  0
> >  0  1 187360  54040   1500  24980   32    0  1048     0  447  379  6  1
>  0
> > 93  0
> >  0  1 187360  53916   1516  25004    0    0   924    52  309  283  1  0
>  0
> > 99  0
> >  0  1 195476  64076   1272  20624    0 8116    32  8156  312  308  1  2
>  0
> > 97  0
> >  0  0 203084  71920   1264  19412    0 7608     0  7608  256  173  0  2
> 89
> >  9  0
> >  0  0 203076  71324   1376  20132   64    0   868    40  192  232  2  0
> 65
> > 33  0
> >  0  0 203076  71328   1392  20108    0    0     0    68   75  144  1  0
> 85
> > 14  0
> >  0  0 203076  71084   1392  20392    0    0   268     0   66  132  0  1
> 96
> >  3  0
> >  0  0 203076  71084   1408  20392    0    0     0    36   60  122  0  0
> 94
> >  6  0
> >  0  0 203076  71076   1424  20376    0    0     0    36   77  148  1  0
> 92
> >  7  0
> >  0  1 203072  70696   1472  20460   96    0   168   280  196 1080  7  1
> 66
> > 26  0
> >  0  0 202900  68704   1512  21236  656    0  1432   104  338  760 10  1
> 10
> > 79  0
> >  0  0 201804  65728   1540  21984 1184    0  1904    64  547 1117 26  2
> 40
> > 33  0
> >  0  2 161904 122500   1540  22640   68    0   776     0  407  477 23  2
> 57
> > 18  0
> >  1  0 161384 122132   1556  22748   36    0    92    60  970  200 92  0
>  0
> >  8  0
> >  0  1 160840 119512   1836  23228  676    0  1432    76  772  866 58  2
>  0
> > 40  0
> >  0  0 160776 119636   1836  23516  196    0   500     0  104  199  1  0
> 63
> > 36  0
> >  0  0 160776 119636   1852  23520    0    0     0    44   83  251  2  0
> 92
> >  6  0
> >  0  0 160776 119644   1868  23504    0    0     0    40   64  117  0  1
> 90
> >  9  0
> >  0  0 160776 119644   1868  23520    0    0     0     0   46   91  0  0
> > 100  0  0
> >  0  1 160764 119456   1888  23556   28    0    32   164   71  121  0  0
> 87
> > 13  0
> >  0  0 160764 119208   1952  23572    0    0     4   288  392 1083  4  1
> 66
> > 29  0
> >  0  0 160764 119192   1952  23596    0    0     0     0   42   69  0  0
> > 100  0  0
> >  0  0 160764 119192   1968  23596    0    0     0    40   60  127  1  0
> 92
> >  7  0
> >  0  0 160764 119192   1984  23584    0    0     4    36   71  135  0  1
> 91
> >  8  0
> >  0  0 160764 119192   1984  23600    0    0     0     0   46   89  0  0
> > 100  0  0
> >  0  0 160764 119192   2000  23600    0    0     0    36   59  121  1  0
> 92
> >  7  0
> >  0  0 160764 119192   2016  23584    0    0     0    36   82  196  0  0
> 93
> >  7  0
> >  0  0 160764 119192   2016  23600    0    0     0     0   38   69  0  0
> > 100  0  0
> >  0  0 160764 119192   2032  23600    0    0     0    36   63  130  1  0
> 91
> >  8  0
> >  0  0 160764 119192   2048  23584    0    0     0    36   67  132  0  0
> 94
> >  6  0
> >  0  0 160764 119192   2096  23584    0    0     0   272   89  193  0  0
> 76
> > 24  0
> >
> >
> > ...Parth
> >
> >
> > On 2/20/13 8:59 PM, "Marcus Sorensen" <shadowsor@gmail.com> wrote:
> >
> >>Well, it doesn't seem to be actively swapping at this point, but I
> >>think it's got active memory swapped out and being used as
> >>occasionally wait% goes up significantly. At any rate this system is
> >>severely memory limited.
> >>
> >>On Wed, Feb 20, 2013 at 9:52 PM, Parth Jagirdar
> >><Parth.Jagirdar@citrix.com> wrote:
> >>> Marcus,
> >>>
> >>> vmstat 1 output
> >>>
> >>>
> >>> [root@localhost management]# vmstat 1  procs
> >>>-----------memory---------- ---swap-- -----io---- --system--
> >>> -----cpu-----
> >>>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
> >>>id
> >>> wa st
> >>>  0  1 190820  72380  10904  15852   36   28    40    92    9    1  0  0
> >>>89
> >>> 10  0
> >>>  0  0 190820  72256  10932  15828    0    0     0    56   63  130  0  0
> >>>88
> >>> 12  0
> >>>  1  0 190820  72256  10932  15844    0    0     0     0   53  153  1  0
> >>>99
> >>>  0  0
> >>>  0  0 190820  72256  10948  15844    0    0     0    44   89  253  2  0
> >>>88
> >>> 10  0
> >>>  0  0 190820  72256  10964  15828    0    0     0    72   64  135  0  0
> >>>88
> >>> 12  0
> >>>  0  0 190820  72256  10964  15844    0    0     0     0   43   76  0  0
> >>> 100  0  0
> >>>  0  0 190820  72256  10980  15844    0    0     0    36   86  244  1  1
> >>>91
> >>>  7  0
> >>>  0  0 190820  72256  10996  15828    0    0     0    44   57  112  0  1
> >>>88
> >>> 11  0
> >>>  0  0 190820  72256  10996  15844    0    0     0     0   45   88  0  0
> >>> 100  0  0
> >>>  0  0 190820  72256  11012  15844    0    0     0    36  100  264  1  1
> >>>91
> >>>  7  0
> >>>  0  0 190820  72132  11044  15824    0    0     4    96  106  211  1  0
> >>>80
> >>> 19  0
> >>>  0  0 190820  72132  11092  15856    0    0     0   368   81  223  0  1
> >>>74
> >>> 25  0
> >>>  0  0 190820  72132  11108  15856    0    0     0    36   78  145  0  1
> >>>93
> >>>  6  0
> >>>  0  0 190820  72132  11124  15840    0    0     0    40   55  106  1  0
> >>>90
> >>>  9  0
> >>>  0  0 190820  72132  11124  15856    0    0     0     0   47   96  0  0
> >>> 100  0  0
> >>>  0  0 190820  72132  11140  15856    0    0     0    36   61  113  0  0
> >>>85
> >>> 15  0
> >>>  0  0 190820  72008  11156  15840    0    0     0    36   61  158  0  0
> >>>93
> >>>  7  0
> >>>  0  0 190820  72008  11156  15856    0    0     0     0   41   82  0  0
> >>> 100  0  0
> >>>  0  0 190820  72008  11172  15856    0    0     0    36   74  149  1  0
> >>>94
> >>>  5  0
> >>>  0  0 190820  72008  11188  15840    0    0     0    36   60  117  0  0
> >>>93
> >>>  7  0
> >>>  0  0 190820  72008  11188  15856    0    0     0     0   43   91  0  0
> >>> 100  0  0
> >>>  1  0 190820  72008  11252  15860    0    0     4   312  108  243  1  1
> >>>68
> >>> 30  0
> >>>  0  0 190820  72008  11268  15844    0    0     0    36   60  128  0  0
> >>>92
> >>>  8  0
> >>>  0  0 190820  72008  11268  15860    0    0     0     0   36   67  0  0
> >>> 100  0  0
> >>>  0  0 190820  71884  11284  15860    0    0     0   104   84  139  0  1
> >>>83
> >>> 16  0
> >>>  0  0 190820  71884  11300  15844    0    0     0    60   55  111  0  0
> >>>69
> >>> 31  0
> >>>  0  0 190820  71884  11300  15860    0    0     0     0   53  121  1  0
> >>>99
> >>>  0  0
> >>>  0  0 190820  71884  11316  15860    0    0     0    40   67  130  0  0
> >>>87
> >>> 13  0
> >>>  0  0 190820  71884  11332  15844    0    0     0    40   58  130  0  0
> >>>90
> >>> 10  0
> >>>  0  0 190820  71884  11332  15864    0    0     0     0   59  824  1  1
> >>>98
> >>>  0  0
> >>>  1  0 190820  71884  11348  15864    0    0     0    40  113  185  1  0
> >>>67
> >>> 32  0
> >>>  0  0 190820  71744  11412  15852    0    0     4   540  100  238  0  0
> >>>67
> >>> 33  0
> >>>  0  0 190820  71744  11412  15868    0    0     0     0   55  159  1  0
> >>>99
> >>>  0  0
> >>>  0  0 190820  71744  11428  15868    0    0     0    40   89  246  2  1
> >>>90
> >>>  7  0
> >>>  0  0 190820  71620  11444  15852    0    0     0    72   65  135  0  0
> >>>93
> >>>  7  0
> >>>  0  0 190820  71620  11444  15868    0    0     0     0   40   74  0  0
> >>> 100  0  0
> >>>  0  0 190820  71620  11460  15868    0    0     0    52   75  216  1  0
> >>>92
> >>>  7  0
> >>>  0  0 190820  71620  11476  15852    0    0     0    44   53  109  0  0
> >>>89
> >>> 11  0
> >>>  0  0 190820  71620  11476  15868    0    0     0     0   43   87  0  0
> >>> 100  0  0
> >>>  0  0 190820  71620  11496  15868    0    0     4    36   83  143  0  1
> >>>90
> >>>  9  0
> >>>  0  0 190820  71620  11512  15852    0    0     0    40   78  869  1  0
> >>>91
> >>>  8  0
> >>>  0  1 190820  71628  11524  15856    0    0     0   188   94  145  0  0
> >>>87
> >>> 13  0
> >>>  0  0 190820  71496  11576  15872    0    0     4   132   96  214  1  0
> >>>80
> >>> 19  0
> >>>  0  0 190820  71496  11592  15856    0    0     0    36   94  128  1  0
> >>>92
> >>>  7  0
> >>>  0  0 190820  71496  11592  15872    0    0     0     0  115  164  0  0
> >>> 100  0  0
> >>>  0  0 190820  71496  11608  15876    0    0     0    36  130  200  0  0
> >>>87
> >>> 13  0
> >>>  0  0 190820  71496  11624  15860    0    0     0    36  141  218  1  1
> >>>91
> >>>  7  0
> >>>  0  0 190820  71504  11624  15876    0    0     0     0  105  119  0  0
> >>> 100  0  0
> >>>  0  0 190820  71504  11640  15876    0    0     0    36  140  218  1  0
> >>>90
> >>>  9  0
> >>> procs -----------memory---------- ---swap-- -----io---- --system--
> >>> -----cpu-----
> >>>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
> >>>id
> >>> wa st
> >>>  0  0 190820  71504  11656  15860    0    0     0    36  131  169  1  0
> >>>92
> >>>  7  0
> >>>  0  0 190820  71504  11656  15876    0    0     0     0  115  146  0  0
> >>> 100  0  0
> >>>  0  0 190820  71380  11672  15876    0    0     0    36  128  173  0  1
> >>>91
> >>>  8  0
> >>>  0  0 190820  71380  11736  15860    0    0     0   308  146  279  1  0
> >>>69
> >>> 30  0
> >>>  0  0 190820  71380  11736  15876    0    0     0     0   59   82  0  0
> >>> 100  0  0
> >>>  0  0 190820  71380  11760  15876    0    0     4    64   90  174  1  1
> >>>86
> >>> 12  0
> >>>
> >>> ...Parth
> >>>
> >>>
> >>>
> >>>
> >>> On 2/20/13 8:46 PM, "Parth Jagirdar" <Parth.Jagirdar@citrix.com>
> wrote:
> >>>
> >>>>JAVA_OPTS="-Djava.awt.headless=true
> >>>>-Dcom.sun.management.jmxremote.port=45219
> >>>>-Dcom.sun.management.jmxremote.authenticate=false
> >>>>-Dcom.sun.management.jmxremote.ssl=false -Xmx512m -Xms512m
> >>>>-XX:+HeapDumpOnOutOfMemoryError
> >>>>-XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=256M"
> >>>>
> >>>>Which did not help.
> >>>>
> >>>>--------------
> >>>>
> >>>>[root@localhost management]# cat /proc/meminfo
> >>>>MemTotal:        1016656 kB
> >>>>MemFree:           68400 kB
> >>>>Buffers:            9108 kB
> >>>>Cached:            20984 kB
> >>>>SwapCached:        17492 kB
> >>>>Active:           424152 kB
> >>>>Inactive:         433152 kB
> >>>>Active(anon):     409812 kB
> >>>>Inactive(anon):   417412 kB
> >>>>Active(file):      14340 kB
> >>>>Inactive(file):    15740 kB
> >>>>Unevictable:           0 kB
> >>>>Mlocked:               0 kB
> >>>>SwapTotal:       2031608 kB
> >>>>SwapFree:        1840900 kB
> >>>>Dirty:                80 kB
> >>>>Writeback:             0 kB
> >>>>AnonPages:        815460 kB
> >>>>Mapped:            11408 kB
> >>>>Shmem:                 4 kB
> >>>>Slab:              60120 kB
> >>>>SReclaimable:      10368 kB
> >>>>SUnreclaim:        49752 kB
> >>>>KernelStack:        5216 kB
> >>>>PageTables:         6800 kB
> >>>>NFS_Unstable:          0 kB
> >>>>Bounce:                0 kB
> >>>>WritebackTmp:          0 kB
> >>>>CommitLimit:     2539936 kB
> >>>>Committed_AS:    1596896 kB
> >>>>VmallocTotal:   34359738367 kB
> >>>>VmallocUsed:        7724 kB
> >>>>VmallocChunk:   34359718200 kB
> >>>>HardwareCorrupted:     0 kB
> >>>>AnonHugePages:    503808 kB
> >>>>HugePages_Total:       0
> >>>>HugePages_Free:        0
> >>>>HugePages_Rsvd:        0
> >>>>HugePages_Surp:        0
> >>>>Hugepagesize:       2048 kB
> >>>>DirectMap4k:        6144 kB
> >>>>DirectMap2M:     1038336 kB
> >>>>[root@localhost management]#
> >>>>-----------------------------
> >>>>
> >>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >>>>
> >>>>
> >>>> 9809 cloud     20   0 2215m 785m 4672 S  0.7 79.1   1:59.40 java
> >>>>
> >>>>
> >>>> 1497 mysql     20   0  700m  15m 3188 S  0.3  1.5  23:04.58 mysqld
> >>>>
> >>>>
> >>>>    1 root      20   0 19348  300  296 S  0.0  0.0   0:00.73 init
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>On 2/20/13 8:26 PM, "Sailaja Mada" <sailaja.mada@citrix.com> wrote:
> >>>>
> >>>>>Hi,
> >>>>>
> >>>>>Cloudstack Java process statistics are given below when it stops
> >>>>>responding are given below :
> >>>>>
> >>>>>top - 09:52:03 up 4 days, 21:43,  2 users,  load average: 0.06,
> >>>>>0.05,
> >>>>>0.02
> >>>>>Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
> >>>>>Cpu(s):  1.7%us,  0.7%sy,  0.0%ni, 97.3%id,  0.3%wa,  0.0%hi,
> >>>>>0.0%si, 0.0%st
> >>>>>Mem:   1014860k total,   947632k used,    67228k free,     5868k
> >>>>>buffers
> >>>>>Swap:  2031608k total,   832320k used,  1199288k free,    26764k
> cached
> >>>>>
> >>>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >>>>>12559 cloud     20   0 3159m 744m 4440 S  2.3 75.1   6:38.39 java
> >>>>>
> >>>>>Thanks,
> >>>>>Sailaja.M
> >>>>>
> >>>>>-----Original Message-----
> >>>>>From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> >>>>>Sent: Thursday, February 21, 2013 9:35 AM
> >>>>>To: cloudstack-dev@incubator.apache.org
> >>>>>Subject: Re: [DISCUSS] Management Server Memory Requirements
> >>>>>
> >>>>>Yes, these are great data points, but so far nobody has responded
> >>>>>on that ticket with the information required to know if the
> >>>>>slowness is related to memory settings or swapping. That was just
a
> >>>>>hunch on my part from being a system admin.
> >>>>>
> >>>>>How much memory do these systems have that experience issues? What
> >>>>>does /proc/meminfo say during the issues? Does adjusting the
> >>>>>tomcat6.conf memory settings make a difference (see ticket
> >>>>>comments)? How much memory do the java processes list as resident
> >>>>>in top?
> >>>>>On Feb 20, 2013 8:53 PM, "Parth Jagirdar"
> >>>>><Parth.Jagirdar@citrix.com>
> >>>>>wrote:
> >>>>>
> >>>>>> +1 Performance degradation is dramatic and I too have observed
> >>>>>> +this
> >>>>>>issue.
> >>>>>>
> >>>>>> I have logged my comments into 1339.
> >>>>>>
> >>>>>>
> >>>>>> ŠParth
> >>>>>>
> >>>>>> On 2/20/13 7:34 PM, "Srikanteswararao Talluri"
> >>>>>> <srikanteswararao.talluri@citrix.com> wrote:
> >>>>>>
> >>>>>> >To add to what Marcus mentioned, Regarding bug CLOUDSTACK-1339
:
> >>>>>> >I have observed this issue within
> >>>>>> >5-10 min of starting management server and there has been
a lot
> >>>>>> >of API requests through automated tests. It is observed
that
> >>>>>> >Management server not only slows down but also goes down
after a
> while.
> >>>>>> >
> >>>>>> >~Talluri
> >>>>>> >
> >>>>>> >-----Original Message-----
> >>>>>> >From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> >>>>>> >Sent: Thursday, February 21, 2013 7:22
> >>>>>> >To: cloudstack-dev@incubator.apache.org
> >>>>>> >Subject: [DISCUSS] Management Server Memory Requirements
> >>>>>> >
> >>>>>> >When Javelin was merged, there was an email sent out stating
> >>>>>> >that devs should set their MAVEN_OPTS to use 2g of heap,
and
> >>>>>> >512M of permanent memory.  Subsequently, there have also
been
> >>>>>> >several
> >>>>>>e-mails
> >>>>>> >and issues where devs have echoed this recommendation, and
> >>>>>>presumably
> >>>>>> >it fixed issues. I've seen the MS run out of memory myself
and
> >>>>>> >applied those recommendations.
> >>>>>> >
> >>>>>> >Is this what we want to provide in the tomcat config for
a
> >>>>>> >package based install as well? It's effectively saying that
the
> >>>>>> >minimum requirements for the management server are something
> >>>>>> >like
> >>>>>> >3 or 4 GB (to be safe for other running tasks) of RAM, right?
> >>>>>> >
> >>>>>> >There is currently a bug filed that may or may not have
to do
> >>>>>> >with this, CLOUDSTACK-1339. Users report mgmt server slowness,
> >>>>>> >going unresponsive for minutes at a time, but the logs seem
to
> >>>>>> >show business as usual. User reports that java is taking
75% of
> >>>>>> >RAM, depending on what else is going on they may be swapping.
> >>>>>> >Settings in the code for an install are currently at 2g/512M,
> >>>>>> >I've been running this on a 4GB server for awhile now, java
is
> >>>>>> >at 900M, but I haven't been pounding it with requests or
anything.
> >>>>>> >
> >>>>>> >This bug might not have anything to do with the memory settings,
> >>>>>> >but I figured it would be good to nail down what our minimum
> >>>>>>requirements
> >>>>>> >are for 4.1
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message