couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Johnson <csj...@yahoo.com>
Subject Re: bulk import
Date Sat, 18 Dec 2010 04:08:25 GMT
This worked and leads into another question. The file I am trying to 
batch import has 1.5M docs. No luck. I had to reduce it down to 250K 
entries before I was able to successfully able to do a batch import. Is 
there a limit to the number of documents that are supported?

Thanks

> ------------------------------------------------------------------------
>
> 	Tim King <mailto:timking@gmx.net>
> December 17, 2010 5:07 PM
>
>
> Try adding this header argument after your curl command:
>
> -H "Content-Type: application/json"
>
> Tim
>
>
> ------------------------------------------------------------------------
>
> 	Chris Johnson <mailto:csj790@yahoo.com>
> December 17, 2010 3:52 PM
>
>
> I have an XML doc that I am parsing that results in 5M documents. I 
> have been successful in serializing this and writing to the database 
> but this takes an extremely long time. As a result, I am testing the 
> _bulk_docs import but have not been successful.
>
> Below is a snapshot of the file that I am creating to do the bulk 
> import, but when I use it, I get the following error:
>
> >curl -X POST -d @bulk.json http://192.168.1.133:5984/bulktest/_bulk_docs
> >{"error":"bad_content_type","reason":"Content-Type must be 
> application/json"}
>
> bulk.json
> {
>         "docs": [
> { "session_id":"849", "policyID":"trust-to-untrust/4", 
> "in_srcIP":"192.168.199.108", "in_srcPort":"49174", 
> "in_dstIP":"72.14.213.109", "in_dstPort":"993", 
> "out_srcIP":"72.14.213.109", "out_srcPort":"993", 
> "out_dstIP":"98.207.50.174", "out_dstPort":"38324", "protocol":"tcp" },
> { "session_id":"2127", "policyID":"trust-to-untrust/4", 
> "in_srcIP":"192.168.199.117", "in_srcPort":"49197", 
> "in_dstIP":"17.148.16.38", "in_dstPort":"993", 
> "out_srcIP":"17.148.16.38", "out_srcPort":"993", 
> "out_dstIP":"98.207.50.174", "out_dstPort":"27540", "protocol":"tcp" },
> { "session_id":"2364", "policyID":"trust-to-untrust/4", 
> "in_srcIP":"192.168.199.117", "in_srcPort":"49309", 
> "in_dstIP":"98.136.48.116", "in_dstPort":"5050", 
> "out_srcIP":"98.136.48.116", "out_srcPort":"5050", 
> "out_dstIP":"98.207.50.174", "out_dstPort":"38335", "protocol":"tcp" },
>     ]
> }
>

Mime
View raw message