couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ralev <vladimir.ra...@gmail.com>
Subject Re: Burst PUT updates
Date Fri, 27 Dec 2013 02:06:40 GMT
It's CouchDB 1.1.1, erlang R14B03 (erts-5.8.4)

The script takes all docs from a view in several DBs, changes one field and
then puts it back. I've tried several versions of the script, one that
maintains the TCP socket and all kinds of reconnect logic, more or less the
same effect. I will review my settings, but there shouldn't be anything too
crazy there. The machine is solid - xeon 4 core server with 16G and there
is very little CPU and IO utilisation, Debian. It's unknown right now if
compaction is correlated. I will have to get back to you on that. This DB
was migrated from a bigcouch, so I guess I will have to rule out some
problem with that too.

The python3 script looks something very close to the minimal script that
demonstrates the issue for me.


def update(db, conn):

    conn.request("GET", "/" + db + "/_design/account/_view/by_name")

    response = conn.getresponse().read().decode()

    rj = json.loads(response)

    for account in rj["rows"]:

        conn.request("GET", "/" + db + "/" + account["id"])

        docresp = conn.getresponse().read().decode()

        jdocresp = json.loads(docresp)

        jdocresp["url"] = "http://myurl0"

        conn.request("PUT","/" + db + "/" + account["id"],
json.dumps(jdocresp))

        conn.getresponse().read()

        print ('Done ' + json.dumps(jdocresp))



def main():

    conn = http.client.HTTPConnection('host',5984, True, 20)

    conn.request("GET", "/_all_dbs")

    response = conn.getresponse()

    body = response.read().decode()

    jdbs = json.loads(body)

    for db in jdbs:

        updatenotify(db, conn)

        conn.close()

        time.sleep(20) # HERE IS THE SLEEP I mentioned

        conn = http.client.HTTPConnection('host', 5984, True, 20)




On Fri, Dec 27, 2013 at 3:27 AM, Stanley Iriele <siriele2x3@gmail.com>wrote:

> What's the hardware... Cocudb version...operating system of your
> project...also that doesn't sound like a lot..sounds like there's something
> funky going on...you're not spinning up new processes for each request are
> you?
> On Dec 26, 2013 7:51 PM, "Vladimir Ralev" <vladimir.ralev@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I have a script that makes a burst of around 100 very fast PUT requests
> > mixed with another 300 or so GET requests against a 4GB database. 40
> > seconds into the script (30-40 PUTs and the CouchDB starts freezing,
> taking
> > longer and longer to respond until it reaches y timeouts). Is this a
> known
> > problem? Is there something I can do about it.
> >
> > The PUTs update existing documents in the DB and there some
> autocompaction.
> > The documents are less than 1KB. Now I run the script with 30 second
> pauses
> > between the writes and that works fine, but it's too slow.
> >
> > Cheers.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message