couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan Roth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (COUCHDB-1167) Importing several documents and subsequently adding attachments fails and leads to crash report on DB server (see unit test)
Date Tue, 17 May 2011 14:50:47 GMT

     [ https://issues.apache.org/jira/browse/COUCHDB-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Stefan Roth updated COUCHDB-1167:
---------------------------------

    Description: 
I just started to explore CouchDB and implemented a few Python unit tests. 

Perhaps I don't understand the concepts yet, but I observed a strange behavior when I try
to add a series of documents, and subsequently want to add attachments to them. The CouchDB
server produces a crash report (1), and some of the document attachments on the DB are empty
afterwards (see test_insertDocsThenAttachments unittest (3) and output (2)). 

(If I add the attachment immediately after the document is saved to the DB everything seems
to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).

Stefan

--------------------------------------------

(1)
=== CouchDB crash report ===
Starting CouchDB...
Eshell V5.8.1  (abort with ^G)
1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
1> Apache CouchDB has started. Time to relax.
1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211
201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1
201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261
201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c
201
1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb
201
1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450
201
1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261
200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3
200
1> [error] [<0.131.0>] {error_report,<0.34.0>,
    {<0.131.0>,crash_report,
     [[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
       {pid,<0.131.0>},
       {registered_name,[]},
       {error_info,
           {error,
               {badmatch,ok},
               [{couch_httpd,handle_request_int,5},
                {mochiweb_http,headers,5},
                {proc_lib,init_p_do_apply,3}]}},
       {ancestors,
           [couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
       {messages,[]},
       {links,[<0.106.0>,#Port<0.1674>]},
       {dictionary,
           [{mochiweb_request_qs,[]},
            {jsonp,undefined},
            {mochiweb_request_cookie,[]}]},
       {trap_exit,false},
       {status,running},
       {heap_size,987},
       {stack_size,24},
       {reductions,13653}],
      []]}}
1> 
=CRASH REPORT==== 17-May-2011::16:17:16 ===
  crasher:
    initial call: mochiweb_socket_server:acceptor_loop/1
    pid: <0.131.0>
    registered_name: []
    exception error: no match of right hand side value ok
      in function  couch_httpd:handle_request_int/5
      in call from mochiweb_http:headers/5
    ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
                  <0.35.0>]
    messages: []
    links: [<0.106.0>,#Port<0.1674>]
    dictionary: [{mochiweb_request_qs,[]},
                  {jsonp,undefined},
                  {mochiweb_request_cookie,[]}]
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 24
    reductions: 13653
  neighbours:
1> [error] [<0.106.0>] {error_report,<0.34.0>,
              {<0.106.0>,std_error,
               {mochiweb_socket_server,235,{child_error,{badmatch,ok}}}}}
1> 
=ERROR REPORT==== 17-May-2011::16:17:16 ===
{mochiweb_socket_server,235,{child_error,{badmatch,ok}}}
1>  

(2)
=== Python unittest output (of failed test) ===
Finding files... done.
Importing test modules ... done.

<TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c'
{'path': '.', 'name': 'doc1'}>
<TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb'
{'path': '.', 'name': 'doc2'}>
<TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450'
{'path': '.', 'name': 'doc3'}>
Unexpected error: <type 'exceptions.AttributeError'>
======================================================================
ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py",
line 129, in test_insertDocsThenAttachments
    self.exportdbdocs(self.db, tmpexportfolder)
  File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py",
line 58, in exportdbdocs
    self.exportdbdocattachment(db, row.key, root)
  File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py",
line 70, in exportdbdocattachment
    f.write(attachment.read())
AttributeError: 'NoneType' object has no attribute 'read'

----------------------------------------------------------------------
Ran 1 test in 0.673s

FAILED (errors=1)


(3)
=== Python unit test code ===
'''
Created on May 17, 2011

@author: sro
'''
import unittest
import couchdb
import filecmp
import shutil
import tempfile
import sys
import os
from couchdb import Document
from uuid import uuid4
from pprint import pprint

class TestDocument(Document):
    def __init__(self):
        self['_id'] = uuid4().hex
        
    def getpath(self):
        return self['path']

    def setpath(self, value):
        self['path'] = value
    
    def getname(self):
        return self['name']
    
    def setname(self, value):
        self['name'] = value

class TestCouchDB(unittest.TestCase):

    def setUp(self):
        # Change according to environment
        serverurl = "http://admin:admin@localhost:5984" 
        dbname = "testdb"
        self.db = self.initdb(serverurl, dbname)
        
    def initdb(self, serverurl, dbname):
        couch = couchdb.Server(serverurl) 
        # Create empty database for tests
        try:
            couch.delete(dbname);
        except couchdb.http.ResourceNotFound:
            print("DB did not exist")
        finally:
            couch.create(dbname);
        db = couch[dbname]
        return db

    def exportdbdocs(self, db, root):
        map_fun = '''function(doc) {
            emit(doc._id, null);
            }'''
        for row in db.query(map_fun):
            self.exportdbdocattachment(db, row.key, root)

    def exportdbdocattachment(self, db, docid, root):
        doc = db.get(docid)
        attachment = db.get_attachment(doc.id, doc['name'])
        
        targetfolder = os.path.join(root, doc['path'])
        if not os.path.exists(targetfolder):
            os.makedirs(targetfolder)
        exportpath = os.path.join(targetfolder, doc['name'])
          
        f = open(exportpath, 'w')
        f.write(attachment.read())
        attachment.close()
        f.close
        
        return exportpath

    def createtestfile(self, path, size):
        f = open(path, 'w')
        f.write('\xff' * size)
        f.close()
        return f
        
    def createfilecollection(self, tmpfolder):
        path = []
        
        path.append(os.path.join(tmpfolder, 'doc1'))
        self.createtestfile(path[0], 100)
        
        path.append(os.path.join(tmpfolder, 'doc2'))
        self.createtestfile(path[1], 100)
        
        path.append(os.path.join(tmpfolder, 'doc3'))
        self.createtestfile(path[2], 100)    
        
        return path  
     
    def test_insertDocsAndAttachmentsInASingleLoop(self):
        # This version of the test works
        try:            
            docs = []
            tmpfolder = tempfile.mkdtemp()
            tmpexportfolder = tmpfolder + '-export'
            
            # Here we first create all docs on the DB and then add attachments to them
            paths = self.createfilecollection(tmpfolder)            
            for i in range(0, len(paths)):
                # We append information about the initial attachment location
                relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)   
                filename = os.path.basename(paths[i])
                
                doc = TestDocument()
                doc.setpath(relpath)
                doc.setname(filename)
                docs.append(doc)
                
                self.db.save(doc)

                fp = open(paths[i])           
                pprint(doc)
                self.db.put_attachment(doc, fp)
                fp.close()
            
            # We export the attachments again and compate them to the originals
            self.exportdbdocs(self.db, tmpexportfolder)
            self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

        except:
            print "Unexpected error:", sys.exc_info()[0]
            raise
        
        finally:
            shutil.rmtree(tmpfolder)
            shutil.rmtree(tmpexportfolder)    

    def test_insertDocsThenAttachments(self):
        # This version of the test crashes CouchDB
        try:            
            docs = []
            tmpfolder = tempfile.mkdtemp()
            tmpexportfolder = tmpfolder + '-export'
            
            # Here we first create all docs on the DB and then add attachments to them
            paths = self.createfilecollection(tmpfolder)            
            for i in range(0, len(paths)):
                # We append information about the initial attachment location
                relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)   
                filename = os.path.basename(paths[i])
                
                doc = TestDocument()
                doc.setpath(relpath)
                doc.setname(filename)
                docs.append(doc)
                
                self.db.save(doc)

            # Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
            # self.db.update(docs)
            
            for i in range(0, len(paths)):
                doc = docs[i]
                fp = open(paths[i])           
                pprint(doc)
                self.db.put_attachment(doc, fp)
                fp.close()
            
            # We export the attachments and compare them to the originals
            self.exportdbdocs(self.db, tmpexportfolder)
            self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

        except:
            print "Unexpected error:", sys.exc_info()[0]
            raise
        
        finally:
            shutil.rmtree(tmpfolder)
            shutil.rmtree(tmpexportfolder)    
            
if __name__ == '__main__':
    suite = unittest.TestSuite()
    suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
    suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
    unittest.TextTestRunner().run(suite)
        

  was:
I just started to explore CouchDB and implemented a few Python unit tests. 

Perhaps I understand the concepts yet, but I observed a strange behavior when I try to add
a series of documents, and subsequently want to add attachments to them. The CouchDB server
produces a crash report (1), and some of the document attachments on the DB are empty afterwards
(see test_insertDocsThenAttachments unittest (3) and output (2)). 

(If I add the attachment immediately after the document is saved to the DB everything seems
to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).

Stefan

--------------------------------------------

(1)
=== CouchDB crash report ===
Starting CouchDB...
Eshell V5.8.1  (abort with ^G)
1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
1> Apache CouchDB has started. Time to relax.
1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211
201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1
201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261
201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c
201
1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb
201
1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450
201
1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261
200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3
200
1> [error] [<0.131.0>] {error_report,<0.34.0>,
    {<0.131.0>,crash_report,
     [[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
       {pid,<0.131.0>},
       {registered_name,[]},
       {error_info,
           {error,
               {badmatch,ok},
               [{couch_httpd,handle_request_int,5},
                {mochiweb_http,headers,5},
                {proc_lib,init_p_do_apply,3}]}},
       {ancestors,
           [couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
       {messages,[]},
       {links,[<0.106.0>,#Port<0.1674>]},
       {dictionary,
           [{mochiweb_request_qs,[]},
            {jsonp,undefined},
            {mochiweb_request_cookie,[]}]},
       {trap_exit,false},
       {status,running},
       {heap_size,987},
       {stack_size,24},
       {reductions,13653}],
      []]}}
1> 
=CRASH REPORT==== 17-May-2011::16:17:16 ===
  crasher:
    initial call: mochiweb_socket_server:acceptor_loop/1
    pid: <0.131.0>
    registered_name: []
    exception error: no match of right hand side value ok
      in function  couch_httpd:handle_request_int/5
      in call from mochiweb_http:headers/5
    ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
                  <0.35.0>]
    messages: []
    links: [<0.106.0>,#Port<0.1674>]
    dictionary: [{mochiweb_request_qs,[]},
                  {jsonp,undefined},
                  {mochiweb_request_cookie,[]}]
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 24
    reductions: 13653
  neighbours:
1> [error] [<0.106.0>] {error_report,<0.34.0>,
              {<0.106.0>,std_error,
               {mochiweb_socket_server,235,{child_error,{badmatch,ok}}}}}
1> 
=ERROR REPORT==== 17-May-2011::16:17:16 ===
{mochiweb_socket_server,235,{child_error,{badmatch,ok}}}
1>  

(2)
=== Python unittest output (of failed test) ===
Finding files... done.
Importing test modules ... done.

<TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c'
{'path': '.', 'name': 'doc1'}>
<TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb'
{'path': '.', 'name': 'doc2'}>
<TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450'
{'path': '.', 'name': 'doc3'}>
Unexpected error: <type 'exceptions.AttributeError'>
======================================================================
ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py",
line 129, in test_insertDocsThenAttachments
    self.exportdbdocs(self.db, tmpexportfolder)
  File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py",
line 58, in exportdbdocs
    self.exportdbdocattachment(db, row.key, root)
  File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py",
line 70, in exportdbdocattachment
    f.write(attachment.read())
AttributeError: 'NoneType' object has no attribute 'read'

----------------------------------------------------------------------
Ran 1 test in 0.673s

FAILED (errors=1)


(3)
=== Python unit test code ===
'''
Created on May 17, 2011

@author: sro
'''
import unittest
import couchdb
import filecmp
import shutil
import tempfile
import sys
import os
from couchdb import Document
from uuid import uuid4
from pprint import pprint

class TestDocument(Document):
    def __init__(self):
        self['_id'] = uuid4().hex
        
    def getpath(self):
        return self['path']

    def setpath(self, value):
        self['path'] = value
    
    def getname(self):
        return self['name']
    
    def setname(self, value):
        self['name'] = value

class TestCouchDB(unittest.TestCase):

    def setUp(self):
        # Change according to environment
        serverurl = "http://admin:admin@localhost:5984" 
        dbname = "testdb"
        self.db = self.initdb(serverurl, dbname)
        
    def initdb(self, serverurl, dbname):
        couch = couchdb.Server(serverurl) 
        # Create empty database for tests
        try:
            couch.delete(dbname);
        except couchdb.http.ResourceNotFound:
            print("DB did not exist")
        finally:
            couch.create(dbname);
        db = couch[dbname]
        return db

    def exportdbdocs(self, db, root):
        map_fun = '''function(doc) {
            emit(doc._id, null);
            }'''
        for row in db.query(map_fun):
            self.exportdbdocattachment(db, row.key, root)

    def exportdbdocattachment(self, db, docid, root):
        doc = db.get(docid)
        attachment = db.get_attachment(doc.id, doc['name'])
        
        targetfolder = os.path.join(root, doc['path'])
        if not os.path.exists(targetfolder):
            os.makedirs(targetfolder)
        exportpath = os.path.join(targetfolder, doc['name'])
          
        f = open(exportpath, 'w')
        f.write(attachment.read())
        attachment.close()
        f.close
        
        return exportpath

    def createtestfile(self, path, size):
        f = open(path, 'w')
        f.write('\xff' * size)
        f.close()
        return f
        
    def createfilecollection(self, tmpfolder):
        path = []
        
        path.append(os.path.join(tmpfolder, 'doc1'))
        self.createtestfile(path[0], 100)
        
        path.append(os.path.join(tmpfolder, 'doc2'))
        self.createtestfile(path[1], 100)
        
        path.append(os.path.join(tmpfolder, 'doc3'))
        self.createtestfile(path[2], 100)    
        
        return path  
     
    def test_insertDocsAndAttachmentsInASingleLoop(self):
        # This version of the test works
        try:            
            docs = []
            tmpfolder = tempfile.mkdtemp()
            tmpexportfolder = tmpfolder + '-export'
            
            # Here we first create all docs on the DB and then add attachments to them
            paths = self.createfilecollection(tmpfolder)            
            for i in range(0, len(paths)):
                # We append information about the initial attachment location
                relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)   
                filename = os.path.basename(paths[i])
                
                doc = TestDocument()
                doc.setpath(relpath)
                doc.setname(filename)
                docs.append(doc)
                
                self.db.save(doc)

                fp = open(paths[i])           
                pprint(doc)
                self.db.put_attachment(doc, fp)
                fp.close()
            
            # We export the attachments again and compate them to the originals
            self.exportdbdocs(self.db, tmpexportfolder)
            self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

        except:
            print "Unexpected error:", sys.exc_info()[0]
            raise
        
        finally:
            shutil.rmtree(tmpfolder)
            shutil.rmtree(tmpexportfolder)    

    def test_insertDocsThenAttachments(self):
        # This version of the test crashes CouchDB
        try:            
            docs = []
            tmpfolder = tempfile.mkdtemp()
            tmpexportfolder = tmpfolder + '-export'
            
            # Here we first create all docs on the DB and then add attachments to them
            paths = self.createfilecollection(tmpfolder)            
            for i in range(0, len(paths)):
                # We append information about the initial attachment location
                relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)   
                filename = os.path.basename(paths[i])
                
                doc = TestDocument()
                doc.setpath(relpath)
                doc.setname(filename)
                docs.append(doc)
                
                self.db.save(doc)

            # Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
            # self.db.update(docs)
            
            for i in range(0, len(paths)):
                doc = docs[i]
                fp = open(paths[i])           
                pprint(doc)
                self.db.put_attachment(doc, fp)
                fp.close()
            
            # We export the attachments and compare them to the originals
            self.exportdbdocs(self.db, tmpexportfolder)
            self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

        except:
            print "Unexpected error:", sys.exc_info()[0]
            raise
        
        finally:
            shutil.rmtree(tmpfolder)
            shutil.rmtree(tmpexportfolder)    
            
if __name__ == '__main__':
    suite = unittest.TestSuite()
    suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
    suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
    unittest.TextTestRunner().run(suite)
        

        Summary: Importing several documents and subsequently adding attachments fails and
leads to crash report on DB server (see unit test)   (was: Importing several document and
adding attachments subsequently fails, and leads to crash report on server (see unit test)
)

> Importing several documents and subsequently adding attachments fails and leads to crash
report on DB server (see unit test) 
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: COUCHDB-1167
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-1167
>             Project: CouchDB
>          Issue Type: Bug
>          Components: Database Core
>    Affects Versions: 1.0.2
>         Environment: Windows 7 and Mac OS X, Python CouchDB API
>            Reporter: Stefan Roth
>            Priority: Critical
>              Labels: attachments, crash
>
> I just started to explore CouchDB and implemented a few Python unit tests. 
> Perhaps I don't understand the concepts yet, but I observed a strange behavior when I
try to add a series of documents, and subsequently want to add attachments to them. The CouchDB
server produces a crash report (1), and some of the document attachments on the DB are empty
afterwards (see test_insertDocsThenAttachments unittest (3) and output (2)). 
> (If I add the attachment immediately after the document is saved to the DB everything
seems to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).
> Stefan
> --------------------------------------------
> (1)
> === CouchDB crash report ===
> Starting CouchDB...
> Eshell V5.8.1  (abort with ^G)
> 1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
> 1> Apache CouchDB has started. Time to relax.
> 1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211
201
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1
201
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261
201
> 1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c
201
> 1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb
201
> 1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450
201
> 1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
> 1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
> 1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
> 1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261
200
> 1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3
200
> 1> [error] [<0.131.0>] {error_report,<0.34.0>,
>     {<0.131.0>,crash_report,
>      [[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
>        {pid,<0.131.0>},
>        {registered_name,[]},
>        {error_info,
>            {error,
>                {badmatch,ok},
>                [{couch_httpd,handle_request_int,5},
>                 {mochiweb_http,headers,5},
>                 {proc_lib,init_p_do_apply,3}]}},
>        {ancestors,
>            [couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
>        {messages,[]},
>        {links,[<0.106.0>,#Port<0.1674>]},
>        {dictionary,
>            [{mochiweb_request_qs,[]},
>             {jsonp,undefined},
>             {mochiweb_request_cookie,[]}]},
>        {trap_exit,false},
>        {status,running},
>        {heap_size,987},
>        {stack_size,24},
>        {reductions,13653}],
>       []]}}
> 1> 
> =CRASH REPORT==== 17-May-2011::16:17:16 ===
>   crasher:
>     initial call: mochiweb_socket_server:acceptor_loop/1
>     pid: <0.131.0>
>     registered_name: []
>     exception error: no match of right hand side value ok
>       in function  couch_httpd:handle_request_int/5
>       in call from mochiweb_http:headers/5
>     ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
>                   <0.35.0>]
>     messages: []
>     links: [<0.106.0>,#Port<0.1674>]
>     dictionary: [{mochiweb_request_qs,[]},
>                   {jsonp,undefined},
>                   {mochiweb_request_cookie,[]}]
>     trap_exit: false
>     status: running
>     heap_size: 987
>     stack_size: 24
>     reductions: 13653
>   neighbours:
> 1> [error] [<0.106.0>] {error_report,<0.34.0>,
>               {<0.106.0>,std_error,
>                {mochiweb_socket_server,235,{child_error,{badmatch,ok}}}}}
> 1> 
> =ERROR REPORT==== 17-May-2011::16:17:16 ===
> {mochiweb_socket_server,235,{child_error,{badmatch,ok}}}
> 1>  
> (2)
> === Python unittest output (of failed test) ===
> Finding files... done.
> Importing test modules ... done.
> <TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c'
{'path': '.', 'name': 'doc1'}>
> <TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb'
{'path': '.', 'name': 'doc2'}>
> <TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450'
{'path': '.', 'name': 'doc3'}>
> Unexpected error: <type 'exceptions.AttributeError'>
> ======================================================================
> ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py",
line 129, in test_insertDocsThenAttachments
>     self.exportdbdocs(self.db, tmpexportfolder)
>   File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py",
line 58, in exportdbdocs
>     self.exportdbdocattachment(db, row.key, root)
>   File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py",
line 70, in exportdbdocattachment
>     f.write(attachment.read())
> AttributeError: 'NoneType' object has no attribute 'read'
> ----------------------------------------------------------------------
> Ran 1 test in 0.673s
> FAILED (errors=1)
> (3)
> === Python unit test code ===
> '''
> Created on May 17, 2011
> @author: sro
> '''
> import unittest
> import couchdb
> import filecmp
> import shutil
> import tempfile
> import sys
> import os
> from couchdb import Document
> from uuid import uuid4
> from pprint import pprint
> class TestDocument(Document):
>     def __init__(self):
>         self['_id'] = uuid4().hex
>         
>     def getpath(self):
>         return self['path']
>     def setpath(self, value):
>         self['path'] = value
>     
>     def getname(self):
>         return self['name']
>     
>     def setname(self, value):
>         self['name'] = value
> class TestCouchDB(unittest.TestCase):
>     def setUp(self):
>         # Change according to environment
>         serverurl = "http://admin:admin@localhost:5984" 
>         dbname = "testdb"
>         self.db = self.initdb(serverurl, dbname)
>         
>     def initdb(self, serverurl, dbname):
>         couch = couchdb.Server(serverurl) 
>         # Create empty database for tests
>         try:
>             couch.delete(dbname);
>         except couchdb.http.ResourceNotFound:
>             print("DB did not exist")
>         finally:
>             couch.create(dbname);
>         db = couch[dbname]
>         return db
>     def exportdbdocs(self, db, root):
>         map_fun = '''function(doc) {
>             emit(doc._id, null);
>             }'''
>         for row in db.query(map_fun):
>             self.exportdbdocattachment(db, row.key, root)
>     def exportdbdocattachment(self, db, docid, root):
>         doc = db.get(docid)
>         attachment = db.get_attachment(doc.id, doc['name'])
>         
>         targetfolder = os.path.join(root, doc['path'])
>         if not os.path.exists(targetfolder):
>             os.makedirs(targetfolder)
>         exportpath = os.path.join(targetfolder, doc['name'])
>           
>         f = open(exportpath, 'w')
>         f.write(attachment.read())
>         attachment.close()
>         f.close
>         
>         return exportpath
>     def createtestfile(self, path, size):
>         f = open(path, 'w')
>         f.write('\xff' * size)
>         f.close()
>         return f
>         
>     def createfilecollection(self, tmpfolder):
>         path = []
>         
>         path.append(os.path.join(tmpfolder, 'doc1'))
>         self.createtestfile(path[0], 100)
>         
>         path.append(os.path.join(tmpfolder, 'doc2'))
>         self.createtestfile(path[1], 100)
>         
>         path.append(os.path.join(tmpfolder, 'doc3'))
>         self.createtestfile(path[2], 100)    
>         
>         return path  
>      
>     def test_insertDocsAndAttachmentsInASingleLoop(self):
>         # This version of the test works
>         try:            
>             docs = []
>             tmpfolder = tempfile.mkdtemp()
>             tmpexportfolder = tmpfolder + '-export'
>             
>             # Here we first create all docs on the DB and then add attachments to them
>             paths = self.createfilecollection(tmpfolder)            
>             for i in range(0, len(paths)):
>                 # We append information about the initial attachment location
>                 relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)   
>                 filename = os.path.basename(paths[i])
>                 
>                 doc = TestDocument()
>                 doc.setpath(relpath)
>                 doc.setname(filename)
>                 docs.append(doc)
>                 
>                 self.db.save(doc)
>                 fp = open(paths[i])           
>                 pprint(doc)
>                 self.db.put_attachment(doc, fp)
>                 fp.close()
>             
>             # We export the attachments again and compate them to the originals
>             self.exportdbdocs(self.db, tmpexportfolder)
>             self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))
>         except:
>             print "Unexpected error:", sys.exc_info()[0]
>             raise
>         
>         finally:
>             shutil.rmtree(tmpfolder)
>             shutil.rmtree(tmpexportfolder)    
>     def test_insertDocsThenAttachments(self):
>         # This version of the test crashes CouchDB
>         try:            
>             docs = []
>             tmpfolder = tempfile.mkdtemp()
>             tmpexportfolder = tmpfolder + '-export'
>             
>             # Here we first create all docs on the DB and then add attachments to them
>             paths = self.createfilecollection(tmpfolder)            
>             for i in range(0, len(paths)):
>                 # We append information about the initial attachment location
>                 relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)   
>                 filename = os.path.basename(paths[i])
>                 
>                 doc = TestDocument()
>                 doc.setpath(relpath)
>                 doc.setname(filename)
>                 docs.append(doc)
>                 
>                 self.db.save(doc)
>             # Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
>             # self.db.update(docs)
>             
>             for i in range(0, len(paths)):
>                 doc = docs[i]
>                 fp = open(paths[i])           
>                 pprint(doc)
>                 self.db.put_attachment(doc, fp)
>                 fp.close()
>             
>             # We export the attachments and compare them to the originals
>             self.exportdbdocs(self.db, tmpexportfolder)
>             self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))
>         except:
>             print "Unexpected error:", sys.exc_info()[0]
>             raise
>         
>         finally:
>             shutil.rmtree(tmpfolder)
>             shutil.rmtree(tmpexportfolder)    
>             
> if __name__ == '__main__':
>     suite = unittest.TestSuite()
>     suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
>     suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
>     unittest.TextTestRunner().run(suite)
>         

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message