incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Davis <paul.joseph.da...@gmail.com>
Subject Re: Emfile error
Date Fri, 26 Aug 2011 20:16:16 GMT
This issue isn't specific to CouchDB. You'll run into this anytime you
have a network server. The basic lines of attack in increasing
complexity:

1. Make sure and reuse your HTTP connections as much as possible.
CouchDB handles persistent connections appropriately. The client just
has to make sure that its using them. In the code you outlined above
it looks like you're creating an HTTP connection for every request. My
guess would be to reuse those HttpClient instances.

2. Increase the file descriptor limits and ephemeral port ranges if
you're hitting system limits.

3. Decrease the socket maximum segment lifetime so that sockets don't
stay in TIME_WAIT as long (defined as 2*MSL). This has other tricky
ramifications that you'd want to understand before tweaking.

On Fri, Aug 26, 2011 at 2:32 PM, Dias, Cliff, VF-Group
<Cliff.Dias@vodafone.com> wrote:
> I see. So what does this mean for an application which expects a high load ? Do I need
to tell couchDB to be keep the connection alive and reuse the connection ?
>
> I would be interested in how people have implemented a couchDB high load application.
>
> Thanks in advance,
>
> Kind regards,
> Cliff
>
>
> -----Original Message-----
> From: Paul Davis [mailto:paul.joseph.davis@gmail.com]
> Sent: Fri 8/26/2011 19:44
> To: user@couchdb.apache.org
> Subject: Re: Emfile error
>
> Sockets are files too. Each HTTP connection you make will require a
> socket unless you reuse that the client and it has HTTP/1.1 support.
> Each socket opened will remain open for a bit before the kernel closes
> it which leads to sockets exhausting the serveres file descriptor
> limit.
>
>
> On Fri, Aug 26, 2011 at 9:17 AM, Dias, Cliff, VF-Group
> <Cliff.Dias@vodafone.com> wrote:
>> Hi,
>> I am trying to run a load test on a couch db instance. I am peforming an
>> http post to insert the data to the DB, but I get an emfile error.
>>
>> I have read the FAQ and I understand that I have to increase the file
>> descriptor limit.
>>
>> I would like to understand why I need to do this ? I am writing to a
>> single database, so I would expect couch DB to recycle the connection.
>>
>> Here is the method I use to write. It is a simple method called 10000
>> times.
>>
>>
>> public void doAdd(Data data) throws HttpException, IOException
>>        {
>>                PostMethod method;
>>
>>                HttpClient client = new HttpClient();
>>
>>                String url = baseURL;
>>
>>                method = new PostMethod(url);
>>
>>
>>
>>
>>                Gson gson = new
>> GsonBuilder().setPrettyPrinting().create();
>>
>>
>>                String jsonOutput = gson.toJson(data);
>>
>>                method.setRequestBody(jsonOutput);
>>                method.setRequestHeader( "content-type",
>> "application/json");
>>
>>                long t1 = System.currentTimeMillis();
>>
>>                client.executeMethod(method);
>>
>>                String response = method.getResponseBodyAsString();
>>
>>                long t2 = System.currentTimeMillis();
>>
>>                System.out.println("response = " + response);
>>                System.out.println("Time = " + (t2-t1));
>>
>>
>>                method.releaseConnection();
>>
>>        }
>>
>>
>> Kind regards,
>> Cliff
>>
>
>

Mime
View raw message