Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@couchdb.apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com
 designates 209.85.220.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <B1241385C30A7B4DAAED209D35E27B770545FF0F@VF-MBX12.internal.vodafone.com>
References: 
 <B1241385C30A7B4DAAED209D35E27B7705918D4D@VF-MBX12.internal.vodafone.com>
 <CAJ_m3YDU1896vzrq9b7rpif3p-9nJiufsW5uJmsee0maazQkXA@mail.gmail.com>
 <B1241385C30A7B4DAAED209D35E27B770545FF0F@VF-MBX12.internal.vodafone.com>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Fri, 26 Aug 2011 15:16:16 -0500
Message-ID: 
 <CAJ_m3YAiP_6zq2bRxzLaoCCUWyYU-7_dHg-QLcBKminxmKXsiQ@mail.gmail.com>
Subject: Re: Emfile error
To: user@couchdb.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

This issue isn't specific to CouchDB. You'll run into this anytime you
have a network server. The basic lines of attack in increasing
complexity:

1. Make sure and reuse your HTTP connections as much as possible.
CouchDB handles persistent connections appropriately. The client just
has to make sure that its using them. In the code you outlined above
it looks like you're creating an HTTP connection for every request. My
guess would be to reuse those HttpClient instances.

2. Increase the file descriptor limits and ephemeral port ranges if
you're hitting system limits.

3. Decrease the socket maximum segment lifetime so that sockets don't
stay in TIME_WAIT as long (defined as 2*MSL). This has other tricky
ramifications that you'd want to understand before tweaking.

On Fri, Aug 26, 2011 at 2:32 PM, Dias, Cliff, VF-Group
<Cliff.Dias@vodafone.com> wrote:
> I see. So what does this mean for an application which expects a high loa=
d ? Do I need to tell couchDB to be keep the connection alive and reuse the=
 connection ?
>
> I would be interested in how people have implemented a couchDB high load =
application.
>
> Thanks in advance,
>
> Kind regards,
> Cliff
>
>
> -----Original Message-----
> From: Paul Davis [mailto:paul.joseph.davis@gmail.com]
> Sent: Fri 8/26/2011 19:44
> To: user@couchdb.apache.org
> Subject: Re: Emfile error
>
> Sockets are files too. Each HTTP connection you make will require a
> socket unless you reuse that the client and it has HTTP/1.1 support.
> Each socket opened will remain open for a bit before the kernel closes
> it which leads to sockets exhausting the serveres file descriptor
> limit.
>
>
> On Fri, Aug 26, 2011 at 9:17 AM, Dias, Cliff, VF-Group
> <Cliff.Dias@vodafone.com> wrote:
>> Hi,
>> I am trying to run a load test on a couch db instance. I am peforming an
>> http post to insert the data to the DB, but I get an emfile error.
>>
>> I have read the FAQ and I understand that I have to increase the file
>> descriptor limit.
>>
>> I would like to understand why I need to do this ? I am writing to a
>> single database, so I would expect couch DB to recycle the connection.
>>
>> Here is the method I use to write. It is a simple method called 10000
>> times.
>>
>>
>> public void doAdd(Data data) throws HttpException, IOException
>> =A0 =A0 =A0 =A0{
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0PostMethod method;
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0HttpClient client =3D new HttpClient();
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0String url =3D baseURL;
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0method =3D new PostMethod(url);
>>
>>
>>
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Gson gson =3D new
>> GsonBuilder().setPrettyPrinting().create();
>>
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0String jsonOutput =3D gson.toJson(data);
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0method.setRequestBody(jsonOutput);
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0method.setRequestHeader( "content-type",
>> "application/json");
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0long t1 =3D System.currentTimeMillis();
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0client.executeMethod(method);
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0String response =3D method.getResponseBod=
yAsString();
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0long t2 =3D System.currentTimeMillis();
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.out.println("response =3D " + resp=
onse);
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0System.out.println("Time =3D " + (t2-t1))=
;
>>
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0method.releaseConnection();
>>
>> =A0 =A0 =A0 =A0}
>>
>>
>> Kind regards,
>> Cliff
>>
>
>