Mailing-List: contact dev-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@couchdb.apache.org
Date: Tue, 17 May 2011 14:34:47 +0000 (UTC)
From: "Stefan Roth (JIRA)" <jira@apache.org>
To: dev@couchdb.apache.org
Message-ID: 
 <1767569166.19200.1305642887497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (COUCHDB-1167) Importing several document and
 adding attachments subsequently fails, and leads to crash report on server
 (see unit test)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Importing several document and adding attachments subsequently fails, and leads to crash report on server (see unit test) 
--------------------------------------------------------------------------------------------------------------------------

                 Key: COUCHDB-1167
                 URL: https://issues.apache.org/jira/browse/COUCHDB-1167
             Project: CouchDB
          Issue Type: Bug
          Components: Database Core
    Affects Versions: 1.0.2
         Environment: Windows 7 and Mac OS X, Python CouchDB API
            Reporter: Stefan Roth
            Priority: Critical


I just started to explore CouchDB and implemented a few Python unit tests. 

Perhaps I understand the concepts yet, but I observed a strange behavior when I try to add a series of documents, and subsequently want to add attachments to them. The CouchDB server produces a crash report (1), and some of the document attachments on the DB are empty afterwards (see test_insertDocsThenAttachments unittest (3) and output (2)). 

(If I add the attachment immediately after the document is saved to the DB everything seems to be ok - see test_insertDocsAndAttachmentsInASingleLoop unittest).

Stefan

--------------------------------------------

(1)
=== CouchDB crash report ===
Starting CouchDB...
Eshell V5.8.1  (abort with ^G)
1> Apache CouchDB 1.0.2 (LogLevel=info) is starting.
1> Apache CouchDB has started. Time to relax.
1> [info] [<0.35.0>] Apache CouchDB has started on http://127.0.0.1:5984/
1> [info] [<0.107.0>] 127.0.0.1 - - 'DELETE' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'HEAD' /testdb 200
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261 201
1> [info] [<0.107.0>] 127.0.0.1 - - 'PUT' /testdb/e0abe6e707f34619b292eb152d812211/doc1?rev=1-2a6731bb8fe08d8d55839abdafd3bd3c 201
1> [info] [<0.118.0>] 127.0.0.1 - - 'PUT' /testdb/fb5b47d4e88f4943a94a46ed6bbad0a1/doc2?rev=1-985759e4100dce92706deff32bdacefb 201
1> [info] [<0.129.0>] 127.0.0.1 - - 'PUT' /testdb/b3880c2930704932872ee21309690261/doc3?rev=1-ca7618164e9b385bc12b9de118f34450 201
1> [info] [<0.134.0>] checkpointing view update at seq 4 for testdb _temp
1> [info] [<0.134.0>] checkpointing view update at seq 6 for testdb _temp
1> [info] [<0.131.0>] 127.0.0.1 - - 'POST' /testdb/_temp_view 200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261 200
1> [info] [<0.131.0>] 127.0.0.1 - - 'GET' /testdb/b3880c2930704932872ee21309690261/doc3 200
1> [error] [<0.131.0>] {error_report,<0.34.0>,
    {<0.131.0>,crash_report,
     [[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
       {pid,<0.131.0>},
       {registered_name,[]},
       {error_info,
           {error,
               {badmatch,ok},
               [{couch_httpd,handle_request_int,5},
                {mochiweb_http,headers,5},
                {proc_lib,init_p_do_apply,3}]}},
       {ancestors,
           [couch_httpd,couch_secondary_services,couch_server_sup,<0.35.0>]},
       {messages,[]},
       {links,[<0.106.0>,#Port<0.1674>]},
       {dictionary,
           [{mochiweb_request_qs,[]},
            {jsonp,undefined},
            {mochiweb_request_cookie,[]}]},
       {trap_exit,false},
       {status,running},
       {heap_size,987},
       {stack_size,24},
       {reductions,13653}],
      []]}}
1> 
=CRASH REPORT==== 17-May-2011::16:17:16 ===
  crasher:
    initial call: mochiweb_socket_server:acceptor_loop/1
    pid: <0.131.0>
    registered_name: []
    exception error: no match of right hand side value ok
      in function  couch_httpd:handle_request_int/5
      in call from mochiweb_http:headers/5
    ancestors: [couch_httpd,couch_secondary_services,couch_server_sup,
                  <0.35.0>]
    messages: []
    links: [<0.106.0>,#Port<0.1674>]
    dictionary: [{mochiweb_request_qs,[]},
                  {jsonp,undefined},
                  {mochiweb_request_cookie,[]}]
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 24
    reductions: 13653
  neighbours:
1> [error] [<0.106.0>] {error_report,<0.34.0>,
              {<0.106.0>,std_error,
               {mochiweb_socket_server,235,{child_error,{badmatch,ok}}}}}
1> 
=ERROR REPORT==== 17-May-2011::16:17:16 ===
{mochiweb_socket_server,235,{child_error,{badmatch,ok}}}
1>  

(2)
=== Python unittest output (of failed test) ===
Finding files... done.
Importing test modules ... done.

<TestDocument u'e0abe6e707f34619b292eb152d812211'@u'1-2a6731bb8fe08d8d55839abdafd3bd3c' {'path': '.', 'name': 'doc1'}>
<TestDocument u'fb5b47d4e88f4943a94a46ed6bbad0a1'@u'1-985759e4100dce92706deff32bdacefb' {'path': '.', 'name': 'doc2'}>
<TestDocument u'b3880c2930704932872ee21309690261'@u'1-ca7618164e9b385bc12b9de118f34450' {'path': '.', 'name': 'doc3'}>
Unexpected error: <type 'exceptions.AttributeError'>
======================================================================
ERROR: test_insertDocsThenAttachments (couchdbbugreport.TestCouchDB)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 129, in test_insertDocsThenAttachments
    self.exportdbdocs(self.db, tmpexportfolder)
  File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 58, in exportdbdocs
    self.exportdbdocattachment(db, row.key, root)
  File "/Users/sro/Documents/workspace_couchapp/src/INIDB_Pydev/src/ch/ini/test/couchdbbugreport.py", line 70, in exportdbdocattachment
    f.write(attachment.read())
AttributeError: 'NoneType' object has no attribute 'read'

----------------------------------------------------------------------
Ran 1 test in 0.673s

FAILED (errors=1)


(3)
=== Python unit test code ===
'''
Created on May 17, 2011

@author: sro
'''
import unittest
import couchdb
import filecmp
import shutil
import tempfile
import sys
import os
from couchdb import Document
from uuid import uuid4
from pprint import pprint

class TestDocument(Document):
    def __init__(self):
        self['_id'] = uuid4().hex
        
    def getpath(self):
        return self['path']

    def setpath(self, value):
        self['path'] = value
    
    def getname(self):
        return self['name']
    
    def setname(self, value):
        self['name'] = value

class TestCouchDB(unittest.TestCase):

    def setUp(self):
        # Change according to environment
        serverurl = "http://admin:admin@localhost:5984" 
        dbname = "testdb"
        self.db = self.initdb(serverurl, dbname)
        
    def initdb(self, serverurl, dbname):
        couch = couchdb.Server(serverurl) 
        # Create empty database for tests
        try:
            couch.delete(dbname);
        except couchdb.http.ResourceNotFound:
            print("DB did not exist")
        finally:
            couch.create(dbname);
        db = couch[dbname]
        return db

    def exportdbdocs(self, db, root):
        map_fun = '''function(doc) {
            emit(doc._id, null);
            }'''
        for row in db.query(map_fun):
            self.exportdbdocattachment(db, row.key, root)

    def exportdbdocattachment(self, db, docid, root):
        doc = db.get(docid)
        attachment = db.get_attachment(doc.id, doc['name'])
        
        targetfolder = os.path.join(root, doc['path'])
        if not os.path.exists(targetfolder):
            os.makedirs(targetfolder)
        exportpath = os.path.join(targetfolder, doc['name'])
          
        f = open(exportpath, 'w')
        f.write(attachment.read())
        attachment.close()
        f.close
        
        return exportpath

    def createtestfile(self, path, size):
        f = open(path, 'w')
        f.write('\xff' * size)
        f.close()
        return f
        
    def createfilecollection(self, tmpfolder):
        path = []
        
        path.append(os.path.join(tmpfolder, 'doc1'))
        self.createtestfile(path[0], 100)
        
        path.append(os.path.join(tmpfolder, 'doc2'))
        self.createtestfile(path[1], 100)
        
        path.append(os.path.join(tmpfolder, 'doc3'))
        self.createtestfile(path[2], 100)    
        
        return path  
     
    def test_insertDocsAndAttachmentsInASingleLoop(self):
        # This version of the test works
        try:            
            docs = []
            tmpfolder = tempfile.mkdtemp()
            tmpexportfolder = tmpfolder + '-export'
            
            # Here we first create all docs on the DB and then add attachments to them
            paths = self.createfilecollection(tmpfolder)            
            for i in range(0, len(paths)):
                # We append information about the initial attachment location
                relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)   
                filename = os.path.basename(paths[i])
                
                doc = TestDocument()
                doc.setpath(relpath)
                doc.setname(filename)
                docs.append(doc)
                
                self.db.save(doc)

                fp = open(paths[i])           
                pprint(doc)
                self.db.put_attachment(doc, fp)
                fp.close()
            
            # We export the attachments again and compate them to the originals
            self.exportdbdocs(self.db, tmpexportfolder)
            self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

        except:
            print "Unexpected error:", sys.exc_info()[0]
            raise
        
        finally:
            shutil.rmtree(tmpfolder)
            shutil.rmtree(tmpexportfolder)    

    def test_insertDocsThenAttachments(self):
        # This version of the test crashes CouchDB
        try:            
            docs = []
            tmpfolder = tempfile.mkdtemp()
            tmpexportfolder = tmpfolder + '-export'
            
            # Here we first create all docs on the DB and then add attachments to them
            paths = self.createfilecollection(tmpfolder)            
            for i in range(0, len(paths)):
                # We append information about the initial attachment location
                relpath = os.path.relpath(os.path.dirname(paths[i]), tmpfolder)   
                filename = os.path.basename(paths[i])
                
                doc = TestDocument()
                doc.setpath(relpath)
                doc.setname(filename)
                docs.append(doc)
                
                self.db.save(doc)

            # Bulk insert (instead of self.db.save(docs[i])) crashes DB server as well
            # self.db.update(docs)
            
            for i in range(0, len(paths)):
                doc = docs[i]
                fp = open(paths[i])           
                pprint(doc)
                self.db.put_attachment(doc, fp)
                fp.close()
            
            # We export the attachments and compare them to the originals
            self.exportdbdocs(self.db, tmpexportfolder)
            self.assertTrue(filecmp.dircmp(tmpfolder, tmpexportfolder, False))

        except:
            print "Unexpected error:", sys.exc_info()[0]
            raise
        
        finally:
            shutil.rmtree(tmpfolder)
            shutil.rmtree(tmpexportfolder)    
            
if __name__ == '__main__':
    suite = unittest.TestSuite()
    suite.addTest(TestCouchDB("test_insertDocsAndAttachmentsInASingleLoop"))
    suite.addTest(TestCouchDB("test_insertDocsThenAttachments"))
    unittest.TextTestRunner().run(suite)
        

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira