From user-return-14249-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Sat Dec 18 04:08:51 2010 Return-Path: Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: (qmail 19012 invoked from network); 18 Dec 2010 04:08:51 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 18 Dec 2010 04:08:51 -0000 Received: (qmail 28446 invoked by uid 500); 18 Dec 2010 04:08:50 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 28332 invoked by uid 500); 18 Dec 2010 04:08:49 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 28324 invoked by uid 99); 18 Dec 2010 04:08:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 04:08:49 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [76.13.13.40] (HELO smtp101.prem.mail.ac4.yahoo.com) (76.13.13.40) by apache.org (qpsmtpd/0.29) with SMTP; Sat, 18 Dec 2010 04:08:48 +0000 Received: (qmail 35449 invoked from network); 18 Dec 2010 04:08:27 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=DKIM-Signature:Received:X-Yahoo-SMTP:X-YMail-OSG:X-Yahoo-Newman-Property:Message-ID:Date:From:User-Agent:MIME-Version:To:Subject:References:In-Reply-To:Content-Type; b=QEHh1P0oPY7ET8Q8VjbgMH7uwCwzWHq/GyhyyMGEtOYNH1Fj49Gm8UYOYNr8+//yE8zXHNIRWoAkf355VvR+yrOrIOh2UMLBDsF/wIoVqYBhEDrHt9qyms7yCkYxcuL5+uK2ZjYugvFP4TsGLaBQovXpxYK49SaBANmp01gwSO4= ; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1292645307; bh=oItdhpC1ur6O99qtz9xYhf95Vq71EbTI2rjeN1lD8og=; h=Received:X-Yahoo-SMTP:X-YMail-OSG:X-Yahoo-Newman-Property:Message-ID:Date:From:User-Agent:MIME-Version:To:Subject:References:In-Reply-To:Content-Type; b=xXo9lAW+Rsn82pQaAznIxxIe5MDQ7Wjylek76lFIB5FHrhwENPNosSFv4bd2Xhny0P5bm0xEJ/Rh3YV1e94bHwMu53IeNbVWDgqfpic+/kGKD3Avk35nuRDCw0ZaS82Ztd6GF0BGhkV6vAS81XIwzA3TXxIrD81NV6nUDJq3etU= Received: from MacDaddy.local (csj790@98.207.50.174 with plain) by smtp101.prem.mail.ac4.yahoo.com with SMTP; 17 Dec 2010 20:08:27 -0800 PST X-Yahoo-SMTP: cbef2U2swBCtkSD6VBey7XQqDQ-- X-YMail-OSG: x89KP4sVM1kvfAtGkz0o5eUPxETOt9SanGZW6PORtFtn5Va l1b8pyXfFw4wRCSwj266LXIs2YqyRTAPMpcPckVLoTBURZjFWyktAX2Hl5LK GHMoadF0NER1XAHR._I.0ARWimervJS2dS1NmuBayvp.wWjpBxhwA9H9T3oo zB3FJpZ30uVangInXQvmh0swzUPu9RM5CFTQ3GQRrBYHnBskJehr6iYNnTx9 sy1Dr8emkHF80gYbSPICI0H4M44TZRyGZjZB8VfvZ2sVerarwu.jE3g.Pu75 dnJruf4C1oXWvjhnvlD.__WE.uKSmStkdGkLaKyPrKpHH499MtoLlBf4Qe3L xZxAzSCy6xFd5ntN8ATWbZ1GhJvp2rsu3INQH_3nW X-Yahoo-Newman-Property: ymail-3 Message-ID: <4D0C33B9.1070803@yahoo.com> Date: Fri, 17 Dec 2010 20:08:25 -0800 From: Chris Johnson User-Agent: Postbox 2.1.0 (Macintosh/20101209) MIME-Version: 1.0 To: user@couchdb.apache.org Subject: Re: bulk import References: <4D0BF7CA.70908@yahoo.com> In-Reply-To: Content-Type: multipart/alternative; boundary="------------020208000801060206030703" --------------020208000801060206030703 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit This worked and leads into another question. The file I am trying to batch import has 1.5M docs. No luck. I had to reduce it down to 250K entries before I was able to successfully able to do a batch import. Is there a limit to the number of documents that are supported? Thanks > ------------------------------------------------------------------------ > > Tim King > December 17, 2010 5:07 PM > > > Try adding this header argument after your curl command: > > -H "Content-Type: application/json" > > Tim > > > ------------------------------------------------------------------------ > > Chris Johnson > December 17, 2010 3:52 PM > > > I have an XML doc that I am parsing that results in 5M documents. I > have been successful in serializing this and writing to the database > but this takes an extremely long time. As a result, I am testing the > _bulk_docs import but have not been successful. > > Below is a snapshot of the file that I am creating to do the bulk > import, but when I use it, I get the following error: > > >curl -X POST -d @bulk.json http://192.168.1.133:5984/bulktest/_bulk_docs > >{"error":"bad_content_type","reason":"Content-Type must be > application/json"} > > bulk.json > { > "docs": [ > { "session_id":"849", "policyID":"trust-to-untrust/4", > "in_srcIP":"192.168.199.108", "in_srcPort":"49174", > "in_dstIP":"72.14.213.109", "in_dstPort":"993", > "out_srcIP":"72.14.213.109", "out_srcPort":"993", > "out_dstIP":"98.207.50.174", "out_dstPort":"38324", "protocol":"tcp" }, > { "session_id":"2127", "policyID":"trust-to-untrust/4", > "in_srcIP":"192.168.199.117", "in_srcPort":"49197", > "in_dstIP":"17.148.16.38", "in_dstPort":"993", > "out_srcIP":"17.148.16.38", "out_srcPort":"993", > "out_dstIP":"98.207.50.174", "out_dstPort":"27540", "protocol":"tcp" }, > { "session_id":"2364", "policyID":"trust-to-untrust/4", > "in_srcIP":"192.168.199.117", "in_srcPort":"49309", > "in_dstIP":"98.136.48.116", "in_dstPort":"5050", > "out_srcIP":"98.136.48.116", "out_srcPort":"5050", > "out_dstIP":"98.207.50.174", "out_dstPort":"38335", "protocol":"tcp" }, > ] > } > --------------020208000801060206030703 Content-Type: multipart/related; boundary="------------020204030408070100090103" --------------020204030408070100090103 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit This worked and leads into another question. The file I am trying to batch import has 1.5M docs. No luck. I had to reduce it down to 250K entries before I was able to successfully able to do a batch import. Is there a limit to the number of documents that are supported?

Thanks



Tim King
December 17, 2010 5:07 PM

Try adding this header argument after your curl command:

-H "Content-Type: application/json"

Tim




Chris Johnson
December 17, 2010 3:52 PM

I have an XML doc that I am parsing that results in 5M documents. I have been successful in serializing this and writing to the database but this takes an extremely long time. As a result, I am testing the _bulk_docs import but have not been successful.

Below is a snapshot of the file that I am creating to do the bulk import, but when I use it, I get the following error:

>curl -X POST -d @bulk.json http://192.168.1.133:5984/bulktest/_bulk_docs
>{"error":"bad_content_type","reason":"Content-Type must be application/json"}

bulk.json
{
        "docs": [
{ "session_id":"849", "policyID":"trust-to-untrust/4", "in_srcIP":"192.168.199.108", "in_srcPort":"49174", "in_dstIP":"72.14.213.109", "in_dstPort":"993", "out_srcIP":"72.14.213.109", "out_srcPort":"993", "out_dstIP":"98.207.50.174", "out_dstPort":"38324", "protocol":"tcp" },
{ "session_id":"2127", "policyID":"trust-to-untrust/4", "in_srcIP":"192.168.199.117", "in_srcPort":"49197", "in_dstIP":"17.148.16.38", "in_dstPort":"993", "out_srcIP":"17.148.16.38", "out_srcPort":"993", "out_dstIP":"98.207.50.174", "out_dstPort":"27540", "protocol":"tcp" },
{ "session_id":"2364", "policyID":"trust-to-untrust/4", "in_srcIP":"192.168.199.117", "in_srcPort":"49309", "in_dstIP":"98.136.48.116", "in_dstPort":"5050", "out_srcIP":"98.136.48.116", "out_srcPort":"5050", "out_dstIP":"98.207.50.174", "out_dstPort":"38335", "protocol":"tcp" },
    ]
}

--------------020204030408070100090103 Content-Type: image/jpeg; x-apple-mail-type=stationery; name="compose-unknown-contact.jpg" Content-Transfer-Encoding: base64 Content-ID: Content-Disposition: inline; filename="compose-unknown-contact.jpg" /9j/4AAQSkZJRgABAQEARwBHAAD/4QCARXhpZgAATU0AKgAAAAgABQESAAMAAAABAAEAAAEa AAUAAAABAAAASgEbAAUAAAABAAAAUgEoAAMAAAABAAIAAIdpAAQAAAABAAAAWgAAAAAAAAby AAAAGQAABvIAAAAZAAKgAgAEAAAAAQAAAB2gAwAEAAAAAQAAAB0AAAAA/9sAQwABAQEBAQEB AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEB AQEB/9sAQwEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEB AQEBAQEBAQEBAQEBAQEBAQEB/8AAEQgAHQAdAwERAAIRAQMRAf/EAB8AAAEFAQEBAQEBAAAA AAAAAAABAgMEBQYHCAkKC//EALUQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQci cRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldY WVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrC w8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29/j5+v/EAB8BAAMBAQEBAQEBAQEA AAAAAAABAgMEBQYHCAkKC//EALURAAIBAgQEAwQHBQQEAAECdwABAgMRBAUhMQYSQVEHYXET IjKBCBRCkaGxwQkjM1LwFWJy0QoWJDThJfEXGBkaJicoKSo1Njc4OTpDREVGR0hJSlNUVVZX WFlaY2RlZmdoaWpzdHV2d3h5eoKDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5 usLDxMXGx8jJytLT1NXW19jZ2uLj5OXm5+jp6vLz9PX29/j5+v/aAAwDAQACEQMRAD8A/uY8 A+AfAt54F8F3l54L8JXV3deEvDlzdXVz4c0ee4ubifR7OWe4uJ5bN5ZpppXaSWWRmkkkZndi xJIB5N8QfF/w48MapfaBoXwt8FahqWnyiC6vL7w3oy2EU+zdJEkFvZrPctGWVXPn24Dh05K5 oA7z4a/8Ky+IWn3cw+G/g3TdS02SKO+sj4d0Odds6s0NxBKdOjZ4ZDHIuGTfG6FXJ3IzAHi/ 7UXhzw94f/4Qb+wdB0bRPtf/AAk32v8AsjS7HTftXkf8I/5H2j7HBD53k+dN5XmbvL82XZjz HyAfWXw4/wCSeeA/+xM8L/8ApjsaAPjX4jaFf6b418RR3pR5LvUrnUopVKgTW+oSvcwSEKPk co+JE/gkDKCygMQD2v8AZ60W5trfxDrMq7be6ktNPtm+U+a9sJZrog53hY/PgXJXa7s4BLRu AAcL+1z/AM0+/wC5r/8AdaoA9R0/xzceGvAfgDTrHT1ub2fwB4UukuJ3P2WFZNJhhAaKPEsz hoCxXzIlwyneeRQB4tqtve61qF1qmoyefeXkplmk8oKCxAAVFHCIihURRwqqBz1oA7rwf4z1 rwjaJpkVtb3+lrPLP9nlWSGeMzHdIsFyhZUVnBfEkEwDM+MAgAA4f9pzV4td0r4Z6rDDJbpd r4wPkzFTJG8M/h+3kUlThh5kTlG4LIVYqpJUAHPWHx70OHRvD2l6n8Of7Un0DQNJ0BL7/hL7 uy+0RaVapbrN9mt9HKRec4km8syTNH5nlmaQIGoAn/4X34T/AOiU/wDl86l/8p6AD/hffhP/ AKJT/wCXzqX/AMp6APPfib8TbT4gWnhmxsfDP/CN2vhv+2fKi/tmXWPtH9sS2Ez/ADzWFnJF 5Ulk7fM8/mefgeUsQDgH/9k= --------------020204030408070100090103 Content-Type: image/jpeg; x-apple-mail-type=stationery; name="postbox-contact.jpg" Content-Transfer-Encoding: base64 Content-ID: Content-Disposition: inline; filename="postbox-contact.jpg" /9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoM DAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsN FBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBT/wAAR CAAdAB0DAREAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAA AgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkK FhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWG h4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl 5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREA AgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYk NOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOE hYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk 5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD6+8WeLNR0XWfsdn9kjgS2tmCtYwOctBGz EsyEnJJPJ71+O5/n+ZYLMquHw9W0FaytF7xT6pvdn6Vk+T4HFYGnWrU7yd7u77tdGJoWp+Mf EwkOmWltdLHwzjT7VFB9NzIBn261y4HNOJcyu8JJyS3fLBL72kr+W50YvAZHgbfWI8t/Obf3 Jt/Mzrzxp4h066ktrpbW3uIzh45NMtwyn6eXXn1uJM8w9R0qtVxkt04xv/6SdlPJMqrQVSnT un1Upf5nQadqk+teGba7uhCbj7ZPFvht44sqEhIBCKAcFm/Ov0vhbMMTmOCnVxUuaSk1slpa L6Jd2fDZ/g6GCxUaeHjZOKe7et33v2OZ+IP/ACM7/wDXraf+k0dfl3FX/I4r/wDbv/pMT73I P+RbS+f/AKUz3nwNaQ+H/DljYogVkiBkI53SEZY/mT+GBX7XlODjgMDSoRVmkr+r1b+/8ND8 uzHEyxeKqVm93p6Lb8Dz/wCPOmQu+narEgWRibeU9243J+WH59x6V+f8cYOK9ljIrXWL8+q+ 7X8Ox9jwriZP2mGb0+Jfk/0MHwz/AMiZbf8AYQuf/RcFetwT/wAi6p/jf/pMTzuKf99h/hX5 yMT4g/8AIzv/ANetp/6TR1+fcVf8jiv/ANu/+kxPscg/5FtL5/8ApTPQvCPj6wvdKhS8v4rO 7iQJILlwgYgY3AnAOeuOo/In9NyXiPB4rCwjiKqhUiknzNK9uqbsnfe26+5v4XNMlxNDESdG DlCTbVle3k0rvTv1/Bcn8TfGMHiCa3srKQzW1uSzy/wu/QY9gM898+2T8PxXnVHMZww2Gd4Q u2+je2nktdet9NFd/VcPZXUwUZV66tKWiXZefr26W+Sl8M/8iZbf9hC5/wDRcFfWcE/8i6p/ jf8A6TE+e4p/32H+FfnIt6pp2ja1dC6u7a/+0GKKJvJu0VDsjVAQDESMhQepr0Mw4WwWY4mW KqzkpSts1bRJdYvt3OPB5/isFQjh6cYtK+6d9Xfuu5U/4Rnw7/z7ap/4HR//ABmvP/1Jy7/n 5P74/wDyJ2f6043+SP3P/wCSD/hGfDv/AD7ap/4HR/8Axmj/AFJy7/n5P74//Ih/rTjf5I/c /wD5IvIllZaZDY2MM8UKTSTlriZZGLOqDsi4A2D86+pyvK6OUUXQoNtN31te9kuiXY8DH4+r mNVVaqSaVtL92+rfc//Z --------------020204030408070100090103-- --------------020208000801060206030703--