couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: [jira] Commented: (COUCHDB-325) Memory Leak in 9.0?
Date Tue, 14 Apr 2009 14:52:50 GMT
Rohit,

When your program is running look at the output of the top command to
see which process is consuming the memory.

I'm not a Rubyist, but is that code just buffering all docs from the
first database and then saving them in one call to bulk docs?

HTH,
Paul Davis

On Tue, Apr 14, 2009 at 10:29 AM, Rohit Amarnath (JIRA) <jira@apache.org> wrote:
>
>    [ https://issues.apache.org/jira/browse/COUCHDB-325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12698786#action_12698786
]
>
> Rohit Amarnath commented on COUCHDB-325:
> ----------------------------------------
>
> Not sure, could you point me in the direction of some tool I could install and run to
give you that info?
>
>
>> Memory Leak in 9.0?
>> -------------------
>>
>>                 Key: COUCHDB-325
>>                 URL: https://issues.apache.org/jira/browse/COUCHDB-325
>>             Project: CouchDB
>>          Issue Type: Bug
>>    Affects Versions: 0.9
>>         Environment: gentoo linux on c1.medium aws instance
>>            Reporter: Rohit Amarnath
>>            Priority: Minor
>>             Fix For: 0.9
>>
>>
>> I am using couchrest to transform a document from one database to another (millions
of records). The memory usage continues to increase till all the memory is used. The process
occasionally fails, but usually completes. I dont know enough about ruby/couchrest/couchdb
to be able to tell you where the memory is increasing from - but  if you give me some direction,
I will be happy to take a look.
>> If I comment out the save, it seems ok.
>> Here is the code:
>> --------------------------------
>> require 'rubygems'
>> require 'couchrest'
>> @db = CouchRest.database!("http://127.0.0.1:5984/xfpds_2008")
>> @db2 = CouchRest.database!("http://127.0.0.1:5984/fpds_2008")
>> @streamer = CouchRest::Streamer.new(@db)
>> @streamer.view("_all_docs") do |row|
>> begin
>> doc = @db.get(row["id"])
>> # remove id so the new database gets fresh document
>> doc.delete("_id")
>> doc.delete("_rev")
>> # change badgerfish notation to assign $ key value to parent
>> doc["amounts"]["obligatedAmount"] =  doc["amounts"]["obligatedAmount"]["$"]
>> doc["amounts"]["baseAndAllOptionsValue"] =  doc["amounts"]["baseAndAllOptionsValue"]["$"]
>> doc["amounts"]["baseAndExercisedOptionsValue"] =  doc["amounts"]["baseAndExercisedOptionsValue"]["$"]
>> ...... A whole bunch of fields
>> # save the document using bulk save
>> response =  @db2.save_doc(doc,true)
>> rescue
>>   # if streamer ends, save last few documents
>>   @db2.bulk_save
>> end
>> end
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>

Mime
View raw message