incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Kimber <mkim...@kana.com>
Subject RE: BigCouch - Replication failing with Cannot Allocate memory
Date Fri, 13 Apr 2012 16:54:10 GMT
Sorry forgot to say that I have already up'd it to N=3 and still get the same issue. 

I ran it again with the 6GB of RAM on each of the servers and ran vmstat and got the following:

r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 2067468  31816 302204    0    0     0     5 1820  360 63 32  5  0  0
 2  0      0 2457728  31816 302212    0    0     0     2 2188  322 70 25  4  0  0
 2  0      0 1936092  31816 302212    0    0     0     0 3020  200 73 24  3  0  0
 2  0      0 687428  31816 302212    0    0     0     1 1958  368 56 42  2  0  0
 2  0      0 2128192  31824 302212    0    0     0     2 2779  243 64 29  7  0  0
 1  0      0 1829848  31824 302216    0    0     0     0 1734  280 68 29  3  0  0
 1  0      0 1200300  31832 302216    0    0     0     8 1841  231 43 13 44  0  0
 2  0      0 1638752  31840 302208    0    0     0     5 2625  350 71 20  8  0  0
 3  0      0 1670856  31848 302216    0    0     0     3 2150  492 40 21 39  0  0
 2  0      0 1020848  31848 302216    0    0     0     0 2307  644 67 22 11  0  0
 1  0      0 271640  31848 302216    0    0     0     6 1995  280 54 42  4  0  0
 1  0      0 455408  31848 302216    0    0     0     1 1879  238 64 33  3  0  0
 2  0      0 1240616  25584 193044    0    0     0     2 2408  232 59 34  8  0  0
 2  0      0 611280  25592 193036    0    0     0     3 2286  246 72 25  2  0  0
 2  0      0 679548  25592 193044    0    0     0     2 3038  175 78 21  2  0  0
 2  0      0 786360  25600 193044    0    0     0     3 1679  269 74 23  3  0  0
 2  0      0 568632  25600 193044    0    0     0     0 2796  274 74 24  2  0  0
eheap_alloc: Cannot allocate 1824525600 bytes of memory (of type "heap").
 0  0      0 5749480  25600 193044    0    0     0     0 1389  160 33 15 52  0  0
 0  0      0 5749956  25608 193044    0    0     0    10 1007   82  0  0 100  0  0
 0  0      0 5749988  25616 193036    0    0     0     3 1016   85  0  0 100  0  0
 0  0      0 5750020  25616 193044    0    0     0     0  998   79  0  0 100  0  0
 0  0      0 5750168  25620 193040    0    0     0     1 1007   87  0  0 100  0  0
 0  0      0 5750308  25620 193044    0    0     0     0 1008   82  0  0 100  0  0

I really need to work out what each process is doing with respect to memory at the time of
failure. I had top running, but not on the node that failed this time, sods law :-)

Mike 

-----Original Message-----
From: Robert Newson [mailto:rnewson@apache.org] 
Sent: 13 April 2012 17:31
To: user@couchdb.apache.org
Subject: Re: BigCouch - Replication failing with Cannot Allocate memory

I should note that bigcouch is tested much more often with N=3.
Perhaps there's something about N=1 that exasperates the issue. For a
test, could you try with N=3?

B.

On 13 April 2012 16:24, Mike Kimber <mkimber@kana.com> wrote:
> "1. Try to replicate the database in another CouchDB."
>
> I have done this to a couchdb 1.2 database successfully. FYI The Source DB is a couchdb
1.1.1.
>
> I haven't done the other tests, but have tested replicating from the couchdb 1.2 database
to the bigcouch install and got the same issue.
>
> Mike
>
>
>
> -----Original Message-----
> From: CGS [mailto:cgsmcmlxxv@gmail.com]
> Sent: 13 April 2012 15:01
> To: user@couchdb.apache.org
> Subject: Re: BigCouch - Replication failing with Cannot Allocate memory
>
> If you say so, Robert, I won't argue with you on that. I meant no offense,
> so, please, accept my apologies if I crossed the line. It's all your's from
> now on.
>
> Mike, please, ignore my suggestion. Sorry for interfering.
>
> Good luck!
>
> CGS
>
>
>
>
> On Fri, Apr 13, 2012 at 3:19 PM, Robert Newson <rnewson@apache.org> wrote:
>
>> I think you should point out that "My idea behind these tests is that
>> it may be that your database may be
>> corrupted (or seen as corrupted by BigCouch at the second test) and what
>> you get is just garbage at a certain document. " is based on no
>> evidence. Nor, if it were true, would it necessarily explain the
>> observed behavior either.
>>
>> It would be useful if we could all stick to asserting only things we
>> know to be true or have reasonable grounds to believe are true.
>> Unfounded speculation, though offered sincerely, is not helpful on a
>> mailing list intended to provide assistance.
>>
>> Thanks,
>> B.
>>
>> On 13 April 2012 13:55, CGS <cgsmcmlxxv@gmail.com> wrote:
>> > Hi Mike,
>> >
>> > I haven't used BigCouch by now and that's why I haven't said anything by
>> > now. Still, giving a thought of what may occur there, I propose few tests
>> > if you have time:
>> > 1. Try to replicate the database in another CouchDB.
>> > 2. If 1 passes, try to replicate to only one node at the time.
>> > 3. If 2 passes, increase the pool of nodes with 1 and repeat the
>> > replication (for sure it will fail at all 3 nodes at the time).
>> >
>> > My idea behind these tests is that it may be that your database may be
>> > corrupted (or seen as corrupted by BigCouch at the second test) and what
>> > you get is just garbage at a certain document. That's why I proposed the
>> > first test. The second test is to see if any of the nodes has a problem
>> in
>> > configuration (or if there is any incompatibility in between your CouchDB
>> > and BigCouch in manipulating your docs). Finally, the third test is to
>> see
>> > if server/node resources limit the number of replications (and at how
>> many
>> > it starts to fail).
>> >
>> > Can you also check the size of the shards at tests 2 and 3?
>> >
>> > If you consider that these tests are irrelevant, please, ignore my
>> > suggestion.
>> >
>> > CGS
>> >
>> >
>> >
>> > On Fri, Apr 13, 2012 at 1:27 PM, Mike Kimber <mkimber@kana.com> wrote:
>> >
>> >> I upped the memory to 6GB on each of the nodes and got exactly the same
>> >> issue in the same time frame i.e. the increased RAM did not seem to by
>> me
>> >> any additional time.
>> >>
>> >> Mike
>> >>
>> >> -----Original Message-----
>> >> From: Robert Newson [mailto:rnewson@apache.org]
>> >> Sent: 12 April 2012 19:34
>> >> To: user@couchdb.apache.org
>> >> Subject: Re: BigCouch - Replication failing with Cannot Allocate memory
>> >>
>> >> 2GB total ram does sound tight. I can only compare to high volume
>> >> production clusters which have much more ram than this. Given that
>> >> beam.smp wanted 1.4 gb and you have 2gb total, do you know where the
>> >> rest one? To couchjs processes, by chance? If so, you can reduce the
>> >> maximum size of that pool in config, I think the default is 50.
>> >>
>> >> On 12 April 2012 18:32, Mike Kimber <mkimber@kana.com> wrote:
>> >> > Ok, I have 3 nodes all load balanced with HAproxy:
>> >> >
>> >> > Centos 5.8 (Virtualised)
>> >> > 2 Cores
>> >> > 2GB RAM
>> >> >
>> >> > I'm trying to replicate about 75K documents which total 6GB when
>> >> compacted (0n Couchdb 1.2 which has compression turned on). I'm told
>> they
>> >> are fairly large documents.
>> >> >
>> >> > When it goes pear shaped Vsmstat starts using a lot of memory:
>> >> >
>> >> > procs -----------memory---------- ---swap-- -----io---- --system--
>> >> -----cpu------
>> >> >  r  b   swpd   free   buff  cache   si   so    bi    bo
  in   cs us
>> sy
>> >> id wa st
>> >> >  1  2 570576   8808    140   7208 2998 2249  3154  2249 1234
 569  1
>>  6
>> >>  2 91  0
>> >> >  0  2 569656   9156    156   7504 2330 1899  2405  1904 1246
 595  1
>>  5
>> >>  9 85  0
>> >> >  1  1 575412   9516    236  14928 1549 2261  3242  2261 1237
 593  1
>>  7
>> >>  1 91  0
>> >> >  0  2 607092  13220    168   8156 3772 9012  3871  9017 1284
 714  1
>> 10
>> >>  4 85  0
>> >> >  1  0 444336 857004    220  10212 5781    0  6202     0 1574
1010 13
>>  7
>> >> 33 47  0
>> >> >  1  0 442176 870684    428  11052 2049    0  2208   140 2561
1541 17
>>  8
>> >> 49 26  0
>> >> >  0  0 442176 813140    460  11968  170    0   348     0
2672 1565 25
>>  9
>> >> 61  4  0
>> >> >  0  1 442176 744972    484  12224 5440    0  5493     7 2432
 900  8
>>  4
>> >> 49 40  0
>> >> >  0  1 442176 714048    484  12296 4547    0  4547     0 1799
 827  4
>>  2
>> >> 50 44  0
>> >> >  0  1 442176 686304    496  12688 5128    0  5222     0 1696
 999  9
>>  2
>> >> 50 40  0
>> >> >  0  3 444000   8712    444  12876  299  368   331   380 1294
 188 22
>> 20
>> >> 36 23  0
>> >> >  0  3 469340  10040    116   7336   29 5087    74  5090 1232
 268  3
>> 22
>> >>  0 75  0
>> >> >  1  2 584356  10220    124   6744 11367 28722 11370 28722 1643
1300  5
>> >> 19 17 59  0
>> >> >  0  1 624908  10640    132   7036 6518 12879  6590 12884 1296
 717  3
>> 10
>> >> 29 58  0
>> >> >  0  2 652556  10948    252  14776 3799 9494  5459  9494 1294
 646  2
>>  9
>> >> 32 57  0
>> >> >  0  2 677784  10648    244  14528 3819 8196  3819  8201 1274
 588  2
>>  7
>> >> 30 61  0
>> >> >  0  2 688460   9512    212   8224 3013 4522  3125  4522 1379
 519  2
>>  7
>> >>  6 84  0
>> >> >  0  3 699164   9888    208   8468 2192 4014  2228  4014 1302
 495  1
>>  6
>> >> 11 83  0
>> >> >  2  0 713104   9004    144   9192 2606 4490  2848  4490 1350
 487  1
>>  8
>> >> 16 75  0
>> >> >
>> >> > It only ever takes out one node at a time and the other nodes seem
to
>> be
>> >> doing very little while the one node is running out of memory.
>> >> >
>> >> > If I kick it off again it processed some more and then spikes the
>> memory
>> >> and fails
>> >> >
>> >> > Thanks
>> >> >
>> >> > Mike
>> >> >
>> >> > PS: hope you enjoyed you couchdb get together!
>> >> >
>> >> > -----Original Message-----
>> >> > From: Robert Newson [mailto:rnewson@apache.org]
>> >> > Sent: 12 April 2012 17:28
>> >> > To: user@couchdb.apache.org
>> >> > Subject: Re: BigCouch - Replication failing with Cannot Allocate
>> memory
>> >> >
>> >> > What kind of load were you putting the machine on?
>> >> >
>> >> > On 12 April 2012 17:24, Robert Newson <rnewson@apache.org> wrote:
>> >> >> Could you show your vm.args file?
>> >> >>
>> >> >> On 12 April 2012 17:23, Robert Newson <rnewson@apache.org>
wrote:
>> >> >>> Unfortunately your request for help coincided with the two
day
>> CouchDB
>> >> >>> Summit. #cloudant and the Issues tab on cloudant/bigcouch are
other
>> >> >>> ways to get bigcouch support, but we happily answer queries
here
>> too,
>> >> >>> when not at the Model UN of CouchDB. :D
>> >> >>>
>> >> >>> B.
>> >> >>>
>> >> >>> On 12 April 2012 17:10, Mike Kimber <mkimber@kana.com>
wrote:
>> >> >>>> Looks like this isn't the right place based on the responses
so
>> far.
>> >> Shame I hoped this was going to help solve our index/view rebuild times
>> etc.
>> >> >>>>
>> >> >>>> Mike
>> >> >>>>
>> >> >>>> -----Original Message-----
>> >> >>>> From: Mike Kimber [mailto:mkimber@kana.com]
>> >> >>>> Sent: 10 April 2012 09:20
>> >> >>>> To: user@couchdb.apache.org
>> >> >>>> Subject: BigCouch - Replication failing with Cannot Allocate
memory
>> >> >>>>
>> >> >>>> I'm not sure if this is the correct place to raise an issue
I am
>> >> having with replicating a standalone couchdb 1.1.1 to a 3 node BigCouch
>> >> cluster? If this is not the correct place please point me in the right
>> >> direction if it is then any one have any ideas why I keep getting the
>> >> following error message when I kick of a replication;
>> >> >>>>
>> >> >>>> eheap_alloc: Cannot allocate 1459620480 bytes of memory
(of type
>> >> "heap").
>> >> >>>>
>> >> >>>> My set-up is:
>> >> >>>>
>> >> >>>> Standalone couchdb 1.1.1 running on Centos 5.7
>> >> >>>>
>> >> >>>> 3 Node BigCouch cluster running on Centos 5.8 with the
following
>> >> local.ini overrides pulling from the Standalone couchdb (78K documents)
>> >> >>>>
>> >> >>>> [httpd]
>> >> >>>> bind_address = XXX.XX.X.XX
>> >> >>>>
>> >> >>>> [cluster]
>> >> >>>> ; number of shards for a new database
>> >> >>>> q = 9
>> >> >>>> ; number of copies of each shard
>> >> >>>> n = 1
>> >> >>>>
>> >> >>>> [couchdb]
>> >> >>>> database_dir = /other/bigcouch/database
>> >> >>>> view_index_dir = /other/bigcouch/view
>> >> >>>>
>> >> >>>> The error is always generate on the third node in the cluster
and
>> the
>> >> server basically max's out on memory before hand. The other nodes seem
>> to
>> >> be doing very little, but are getting data i.e. the shard sizes are
>> >> growing. I've put the copies per shard down to 1 as currently I'm not
>> >> interested in resilience.
>> >> >>>>
>> >> >>>> Any help would be greatly appreciated.
>> >> >>>>
>> >> >>>> Mike
>> >> >>>>
>> >>
>>

Mime
View raw message