incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From CGS <cgsmcml...@gmail.com>
Subject Re: How to do Bulk-insert from Huge JSON File (460 MB)
Date Tue, 12 Jun 2012 12:06:43 GMT
See -F (http://curl.haxx.se/docs/manpage.html) option. I haven't tried it
by now with CouchDB, but usually it does the trick.

CGS




On Tue, Jun 12, 2012 at 11:58 AM, Mohammad Prabowo <rizalp@gmail.com> wrote:

> Would you please tell me the curl syntax to upload the file as multipart?
> I've been scratching my head, and can't even find it from the curl's
> manual..
>
> On Tue, Jun 12, 2012 at 4:10 PM, CGS <cgsmcmlxxv@gmail.com> wrote:
>
> > You said your file is 465 MB, but cURL is sending only
> >
> > > Content-Length: 439203931
> >
> > which raised a multipart file transmission which never occurred:
> >
> > > Expect: 100-continue
> >
> > Try either multipart file or split your JSON in smaller files (2 files
> > should be enough).
> >
> > CGS
> >
> >
> >
> >
> > On Tue, Jun 12, 2012 at 10:42 AM, Mohammad Prabowo <rizalp@gmail.com>
> > wrote:
> >
> > > I can only see indication in Couch.log as if nothing happened. I've
> tried
> > > using smaller JSON file (4K) and it run successfully.
> > > I'm using couchDB 1.2
> > >
> > > On Tue, Jun 12, 2012 at 11:38 AM, Dave Cottlehuber <dave@muse.net.nz>
> > > wrote:
> > >
> > > > On 12 June 2012 06:27, Mohammad Prabowo <rizalp@gmail.com> wrote:
> > > > > Still no result. I guess i have to split it into smaller
> documents..
> > > > >
> > > > > On Mon, Jun 11, 2012 at 7:03 PM, Robert Newson <rnewson@apache.org
> >
> > > > wrote:
> > > > >
> > > > >>
> > > > >> -d will load the whole file into memory and also interpret it
as
> > > ascii,
> > > > >> which might make it invalid.
> > > > >>
> > > > >> use -T <filename> instead.
> > > > >>
> > > > >> B.
> > > > >>
> > > > >> On 11 Jun 2012, at 12:29, Mohammad Prabowo wrote:
> > > > >>
> > > > >> > Hi. I need to do bulk-insert of document in my CouchDB database.
> > > > >> > I'm trying to follow the manual here:
> > > > >> > http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API
> > > > >> >
> > > > >> > Here is my code:
> > > > >> >
> > > > >> > ~$ DB="http://localhost:5984/employees"
> > > > >> > ~$ curl -H "Content-Type:application/json" -d
> > > > >> @employees_selfContained.json
> > > > >> > -vX POST $DB/_bulk_docs
> > > > >> >
> > > > >> > the file employees_selfContained.json is a huge file = 465
MB.
> > I've
> > > > >> > validated it using JSONLint and found nothing wrong
> > > > >> > Here's the curl's verbose output:
> > > > >> >
> > > > >> > curl -H "Content-Type:application/json" -d
> > > > @employees_selfContained.json
> > > > >> > -vX POST $DB/_bulk_docs
> > > > >> > * About to connect() to 127.0.0.1 port 5984 (#0)
> > > > >> > * Trying 127.0.0.1... connected
> > > > >> > * Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0)
> > > > >> >> POST /employees/_bulk_docs HTTP/1.1
> > > > >> >> User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7
> > > > >> OpenSSL/0.9.8k
> > > > >> > zlib/1.2.3.3 libidn/1.15
> > > > >> >> Host: 127.0.0.1:5984
> > > > >> >> Accept: */*
> > > > >> >> Content-Type:application/json
> > > > >> >> Content-Length: 439203931
> > > > >> >> Expect: 100-continue
> > > > >> >>
> > > > >> > < HTTP/1.1 100 Continue
> > > > >> > * Empty reply from server
> > > > >> > * Connection #0 to host 127.0.0.1 left intact
> > > > >> > curl: (52) Empty reply from server
> > > > >> > * Closing connection #0
> > > > >> >
> > > > >> > How can i do bulk-insert from that Huge single file? I prefer
> not
> > to
> > > > >> split
> > > > >> > the file into smaller size if possible..
> > > > >>
> > > > >>
> > > >
> > > >
> > > > Mohammed,
> > > >
> > > > What do you see in the couch.log ?
> > > >
> > > > I'd be interested to hear if this same upload works against 1.1.1 vs
> > > 1.2.0.
> > > >
> > > > Thanks
> > > > Dave
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message