Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of rizalp@gmail.com designates
 209.85.160.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKrexVKQ4erspdo8YCTcL6QUs0qC1=oqa9aoiU6LXn5AatZJ9Q@mail.gmail.com>
References: 
 <CAKXTtKGegDyrrxwASqcnTbeZKZOMV=nNHRw3Ba+uXow70cPEQw@mail.gmail.com>
	<93C87D8E-D891-4C54-B7AC-A621A68A11C9@apache.org>
	<CAKXTtKE8mi3s33RG5hE5j64bxb0cvtrD8xWBs_4iKTOYOLpLag@mail.gmail.com>
	<CAHZBNKZNJQRpwYo9S7Dis4LOQ3o=3LZmyRQwa-vRBd5e0sO+WQ@mail.gmail.com>
	<CAKXTtKEz2PnRTpf6DRtb3JmVCsk8GYtbZgFasDxhH_fJhj1hgw@mail.gmail.com>
	<CAKrexVKQ4erspdo8YCTcL6QUs0qC1=oqa9aoiU6LXn5AatZJ9Q@mail.gmail.com>
Date: Tue, 12 Jun 2012 16:58:31 +0700
Message-ID: 
 <CAKXTtKHKb57u_1zwJJCUxjWnthC7GNkUEt_YXuwxaDZU9rx2Hg@mail.gmail.com>
Subject: Re: How to do Bulk-insert from Huge JSON File (460 MB)
From: Mohammad Prabowo <rizalp@gmail.com>
To: user@couchdb.apache.org
Content-Type: multipart/alternative; boundary=e89a8ff2518250fef004c243825d

--e89a8ff2518250fef004c243825d
Content-Type: text/plain; charset=ISO-8859-1

Would you please tell me the curl syntax to upload the file as multipart?
I've been scratching my head, and can't even find it from the curl's
manual..

On Tue, Jun 12, 2012 at 4:10 PM, CGS <cgsmcmlxxv@gmail.com> wrote:

> You said your file is 465 MB, but cURL is sending only
>
> > Content-Length: 439203931
>
> which raised a multipart file transmission which never occurred:
>
> > Expect: 100-continue
>
> Try either multipart file or split your JSON in smaller files (2 files
> should be enough).
>
> CGS
>
>
>
>
> On Tue, Jun 12, 2012 at 10:42 AM, Mohammad Prabowo <rizalp@gmail.com>
> wrote:
>
> > I can only see indication in Couch.log as if nothing happened. I've tried
> > using smaller JSON file (4K) and it run successfully.
> > I'm using couchDB 1.2
> >
> > On Tue, Jun 12, 2012 at 11:38 AM, Dave Cottlehuber <dave@muse.net.nz>
> > wrote:
> >
> > > On 12 June 2012 06:27, Mohammad Prabowo <rizalp@gmail.com> wrote:
> > > > Still no result. I guess i have to split it into smaller documents..
> > > >
> > > > On Mon, Jun 11, 2012 at 7:03 PM, Robert Newson <rnewson@apache.org>
> > > wrote:
> > > >
> > > >>
> > > >> -d will load the whole file into memory and also interpret it as
> > ascii,
> > > >> which might make it invalid.
> > > >>
> > > >> use -T <filename> instead.
> > > >>
> > > >> B.
> > > >>
> > > >> On 11 Jun 2012, at 12:29, Mohammad Prabowo wrote:
> > > >>
> > > >> > Hi. I need to do bulk-insert of document in my CouchDB database.
> > > >> > I'm trying to follow the manual here:
> > > >> > http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API
> > > >> >
> > > >> > Here is my code:
> > > >> >
> > > >> > ~$ DB="http://localhost:5984/employees"
> > > >> > ~$ curl -H "Content-Type:application/json" -d
> > > >> @employees_selfContained.json
> > > >> > -vX POST $DB/_bulk_docs
> > > >> >
> > > >> > the file employees_selfContained.json is a huge file = 465 MB.
> I've
> > > >> > validated it using JSONLint and found nothing wrong
> > > >> > Here's the curl's verbose output:
> > > >> >
> > > >> > curl -H "Content-Type:application/json" -d
> > > @employees_selfContained.json
> > > >> > -vX POST $DB/_bulk_docs
> > > >> > * About to connect() to 127.0.0.1 port 5984 (#0)
> > > >> > * Trying 127.0.0.1... connected
> > > >> > * Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0)
> > > >> >> POST /employees/_bulk_docs HTTP/1.1
> > > >> >> User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7
> > > >> OpenSSL/0.9.8k
> > > >> > zlib/1.2.3.3 libidn/1.15
> > > >> >> Host: 127.0.0.1:5984
> > > >> >> Accept: */*
> > > >> >> Content-Type:application/json
> > > >> >> Content-Length: 439203931
> > > >> >> Expect: 100-continue
> > > >> >>
> > > >> > < HTTP/1.1 100 Continue
> > > >> > * Empty reply from server
> > > >> > * Connection #0 to host 127.0.0.1 left intact
> > > >> > curl: (52) Empty reply from server
> > > >> > * Closing connection #0
> > > >> >
> > > >> > How can i do bulk-insert from that Huge single file? I prefer not
> to
> > > >> split
> > > >> > the file into smaller size if possible..
> > > >>
> > > >>
> > >
> > >
> > > Mohammed,
> > >
> > > What do you see in the couch.log ?
> > >
> > > I'd be interested to hear if this same upload works against 1.1.1 vs
> > 1.2.0.
> > >
> > > Thanks
> > > Dave
> > >
> >
>

--e89a8ff2518250fef004c243825d--