deltacloud-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ronelle Landy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DTACLOUD-347) Deltacloud API leaking memory
Date Tue, 30 Oct 2012 17:00:14 GMT

    [ https://issues.apache.org/jira/browse/DTACLOUD-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487012#comment-13487012
] 

Ronelle Landy commented on DTACLOUD-347:
----------------------------------------

Tested deltacloud git version: 3c5a74ae228c53044ddbd6b280c3ec96921003c1 on Fedora 17, ruby
version 1.9.3p194.

I ran the following script (using mock provider):

**********
    #! /bin/bash
     
    for (( i=0; i<=10000; i++ ))
    do
     source <(curl -X GET --user "un:pw" "http://server:3003/api/images?format=xml";
     curl -X POST -F "image_id=img2" --user "un:pw" "http://server:3003/api/instances?format=xml";
     sleep 2;
     curl -X GET --user "un:pw" "http://server:3003/api/instances/inst3?format=xml";
    curl -X POST --user "un:pw" "http://server:3003/api/instances/inst3/stop?format=xml";
     sleep 2;
     curl -X DELETE --user "un:pw" "http://server:3003/api/instances/inst3?format=xml";
     sleep 2;);
    done 

**********

After running for a number of hours - overnight, the memory usage seems to have grown rapidly
... >> top showing ruby taking 47% to 50% of Memory.
According to tracker, this fix is already pushed. Below is the deltacloud server output after
running overnight - notice how long it takes to complete *some* GET/POST/DELETE operations
on mock:

    HTTP/1.1" 204 - 49.9845
    192.168.1.42 - - [30/Oct/2012 11:13:57] "GET /api/images?format=xml HTTP/1.1" 200 3718
75.8069
    192.168.1.42 - - [30/Oct/2012 11:13:58] "POST /api/instances?format=xml HTTP/1.1" 201
1318 0.0564
    192.168.1.42 - - [30/Oct/2012 11:16:26] "GET /api/instances/inst3?format=xml HTTP/1.1"
200 1318 144.8398
    192.168.1.42 - - [30/Oct/2012 11:18:12] "POST /api/instances/inst3/stop?format=xml HTTP/1.1"
202 1316 103.5509
    192.168.1.42 - - [30/Oct/2012 11:19:36] "DELETE /api/instances/inst3?format=xml HTTP/1.1"
204 - 82.4481
    192.168.1.42 - - [30/Oct/2012 11:22:10] "GET /api/images?format=xml HTTP/1.1" 200 3718
149.5234
    192.168.1.42 - - [30/Oct/2012 11:22:11] "POST /api/instances?format=xml HTTP/1.1" 201
1318 0.1001
    192.168.1.42 - - [30/Oct/2012 11:22:59] "GET /api/instances/inst3?format=xml HTTP/1.1"
200 1318 45.3005
    192.168.1.42 - - [30/Oct/2012 11:23:56] "POST /api/instances/inst3/stop?format=xml HTTP/1.1"
202 1316 56.4984
    192.168.1.42 - - [30/Oct/2012 11:25:15] "DELETE /api/instances/inst3?format=xml HTTP/1.1"
204 - 76.9550
    192.168.1.42 - - [30/Oct/2012 11:28:45] "POST /api/instances?format=xml HTTP/1.1" 201
1318 2.5896
    192.168.1.42 - - [30/Oct/2012 11:28:45] "GET /api/images?format=xml HTTP/1.1" 200 3718
195.9203
    192.168.1.42 - - [30/Oct/2012 11:29:57] "GET /api/instances/inst3?format=xml HTTP/1.1"
200 1318 69.5493
    192.168.1.42 - - [30/Oct/2012 11:30:00] "POST /api/instances/inst3/stop?format=xml HTTP/1.1"
202 1316 1.3402
    192.168.1.42 - - [30/Oct/2012 11:33:57] "DELETE /api/instances/inst3?format=xml HTTP/1.1"
204 - 235.0925
    192.168.1.42 - - [30/Oct/2012 11:39:59] "GET /api/images?format=xml HTTP/1.1" 200 3718
356.7110
    192.168.1.42 - - [30/Oct/2012 11:40:13] "POST /api/instances?format=xml HTTP/1.1" 201
1318 10.4527
    192.168.1.42 - - [30/Oct/2012 11:44:50] "GET /api/instances/inst3?format=xml HTTP/1.1"
200 1318 266.1652
    192.168.1.42 - - [30/Oct/2012 11:49:08] "POST /api/instances/inst3/stop?format=xml HTTP/1.1"
202 1316 253.1082
    192.168.1.42 - - [30/Oct/2012 11:51:43] "DELETE /api/instances/inst3?format=xml HTTP/1.1"
204 - 151.6960
    192.168.1.42 - - [30/Oct/2012 11:53:14] "GET /api/images?format=xml HTTP/1.1" 200 3718
86.9377
    192.168.1.42 - - [30/Oct/2012 11:53:17] "POST /api/instances?format=xml HTTP/1.1" 201
1318 0.3289 

Munin also shows growing system usage.

Going to reboot and try marios's GET images experiment above
                
> Deltacloud API leaking memory
> -----------------------------
>
>                 Key: DTACLOUD-347
>                 URL: https://issues.apache.org/jira/browse/DTACLOUD-347
>             Project: DeltaCloud
>          Issue Type: Bug
>          Components: Server
>         Environment: deltacloud-core, GIT adade32451815c8ee28e29be35636bd0f98473fc
>            Reporter: Michal Fojtik
>            Assignee: Michal Fojtik
>            Priority: Blocker
>              Labels: blocker-1.0.5
>
> Deltacloud API memory usage is slowly increasing after doing 100+ requests:
> Starting Deltacloud API :: mock :: http://localhost:3001/api
> >> Thin web server (v1.5.0 codename Knife)
> >> Debugging ON
> >> Maximum connections set to 1024
> >> Listening on localhost:3001, CTRL+C to stop
> 127.0.0.1 - - [17/Oct/2012 15:26:10] "GET /api/images HTTP/1.1" 200 4420 0.0308
> 127.0.0.1 - - [17/Oct/2012 15:26:17] "GET /api/images HTTP/1.1" 200 4420 0.0220
> 127.0.0.1 - - [17/Oct/2012 15:26:52] "GET /api/images HTTP/1.1" 200 4420 0.0219
> ... 1000 requests after:
> 127.0.0.1 - - [17/Oct/2012 15:27:53] "GET /api/images HTTP/1.1" 200 4420 0.1539
> 127.0.0.1 - - [17/Oct/2012 15:27:53] "GET /api/images HTTP/1.1" 200 4420 0.1554
> 127.0.0.1 - - [17/Oct/2012 15:27:53] "GET /api/images HTTP/1.1" 200 4420 0.1526
> As you can see the response time is slowly increasing and after few minutes it gets to
the point where the request take like 10 seconds to complete and all your free memory is gone
;-)
> This is easy to reproduce:
> 1. Start Deltacloud API ($ deltacloudd -i mock)
> 2. $ while true; do curl -H 'Accept: application/xml' --user 'mockuser:mockpassword'
http://localhost:3001/api/images > /dev/null; done
> 3. Keep it running for ~1minute and watch the console with Deltacloud API log.
> Thanks to Tomas Hrcka for reporting this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message