From user-return-21192-apmail-couchdb-user-archive=couchdb.apache.org@couchdb.apache.org Tue Jun 12 04:28:06 2012 Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 91B0F9286 for ; Tue, 12 Jun 2012 04:28:06 +0000 (UTC) Received: (qmail 41695 invoked by uid 500); 12 Jun 2012 04:28:05 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 41517 invoked by uid 500); 12 Jun 2012 04:28:05 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 41508 invoked by uid 99); 12 Jun 2012 04:28:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jun 2012 04:28:04 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of rizalp@gmail.com designates 209.85.214.180 as permitted sender) Received: from [209.85.214.180] (HELO mail-ob0-f180.google.com) (209.85.214.180) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jun 2012 04:27:59 +0000 Received: by obbun3 with SMTP id un3so8820578obb.11 for ; Mon, 11 Jun 2012 21:27:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=vKIwqoeOGK/nZjcGyxqpd0SnOk9LJ36eJCc8yBL6N3I=; b=pf/+kBp8ApoyXx+zAKyK0Bx8NFV+LD5dRTX9QpnGKWNzFt4BrDTwh9OdutJFokmcPf 3wDj4FWaEuE17f0MB1YLC6ksIF7DCocpTjAfEtc4niqXkATcRJ9NI241v24SvvovFprW 5MeXCjE/AWQp0cewqN3u6PNJcgnokRLCn0ICx5pD2PtPG8+mKtaskUqk9B6bY4pCs8Zh KurhjNVG9KrNKjcPlWJTD9D1tM6x7iVLpppgotyECXRO5GEyRHvneJ9tTH3AZKBL6cZs oiCjXytwPKu/bQBEtSvvV9rssQI/Vn/v+dVxBtng559uHIT/+k8jGldRlqj9CLPazOzw 4lwQ== MIME-Version: 1.0 Received: by 10.60.8.8 with SMTP id n8mr43201oea.38.1339475258408; Mon, 11 Jun 2012 21:27:38 -0700 (PDT) Received: by 10.76.2.142 with HTTP; Mon, 11 Jun 2012 21:27:38 -0700 (PDT) In-Reply-To: <93C87D8E-D891-4C54-B7AC-A621A68A11C9@apache.org> References: <93C87D8E-D891-4C54-B7AC-A621A68A11C9@apache.org> Date: Tue, 12 Jun 2012 11:27:38 +0700 Message-ID: Subject: Re: How to do Bulk-insert from Huge JSON File (460 MB) From: Mohammad Prabowo To: user@couchdb.apache.org Content-Type: multipart/alternative; boundary=e89a8ff1c6d4f67d3904c23ee2af X-Virus-Checked: Checked by ClamAV on apache.org --e89a8ff1c6d4f67d3904c23ee2af Content-Type: text/plain; charset=ISO-8859-1 Still no result. I guess i have to split it into smaller documents.. On Mon, Jun 11, 2012 at 7:03 PM, Robert Newson wrote: > > -d will load the whole file into memory and also interpret it as ascii, > which might make it invalid. > > use -T instead. > > B. > > On 11 Jun 2012, at 12:29, Mohammad Prabowo wrote: > > > Hi. I need to do bulk-insert of document in my CouchDB database. > > I'm trying to follow the manual here: > > http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API > > > > Here is my code: > > > > ~$ DB="http://localhost:5984/employees" > > ~$ curl -H "Content-Type:application/json" -d > @employees_selfContained.json > > -vX POST $DB/_bulk_docs > > > > the file employees_selfContained.json is a huge file = 465 MB. I've > > validated it using JSONLint and found nothing wrong > > Here's the curl's verbose output: > > > > curl -H "Content-Type:application/json" -d @employees_selfContained.json > > -vX POST $DB/_bulk_docs > > * About to connect() to 127.0.0.1 port 5984 (#0) > > * Trying 127.0.0.1... connected > > * Connected to 127.0.0.1 (127.0.0.1) port 5984 (#0) > >> POST /employees/_bulk_docs HTTP/1.1 > >> User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 > OpenSSL/0.9.8k > > zlib/1.2.3.3 libidn/1.15 > >> Host: 127.0.0.1:5984 > >> Accept: */* > >> Content-Type:application/json > >> Content-Length: 439203931 > >> Expect: 100-continue > >> > > < HTTP/1.1 100 Continue > > * Empty reply from server > > * Connection #0 to host 127.0.0.1 left intact > > curl: (52) Empty reply from server > > * Closing connection #0 > > > > How can i do bulk-insert from that Huge single file? I prefer not to > split > > the file into smaller size if possible.. > > --e89a8ff1c6d4f67d3904c23ee2af--