From users-return-24939-apmail-subversion-users-archive=subversion.apache.org@subversion.apache.org Fri Jul 15 21:10:41 2016 Return-Path: X-Original-To: apmail-subversion-users-archive@minotaur.apache.org Delivered-To: apmail-subversion-users-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6E8C619494 for ; Fri, 15 Jul 2016 21:10:41 +0000 (UTC) Received: (qmail 56427 invoked by uid 500); 15 Jul 2016 21:10:36 -0000 Delivered-To: apmail-subversion-users-archive@subversion.apache.org Received: (qmail 56370 invoked by uid 500); 15 Jul 2016 21:10:36 -0000 Mailing-List: contact users-help@subversion.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list users@subversion.apache.org Delivered-To: moderator for users@subversion.apache.org Received: (qmail 66269 invoked by uid 99); 15 Jul 2016 19:49:16 -0000 X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.821 X-Spam-Level: X-Spam-Status: No, score=-0.821 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=sbdHhD4pnQ5BvCxkKgntCgNTUFhscLM4XHj0ragKipA=; b=ORbeDz8axUTM4u3e8CTGrh/YK5WPKHahgMjE6fJPHYon9tqvpWkjbSAdjQnL7BguCL O/4BvOM1vdFa+XbN0V8nBuoT0+2rmRdzClRvHTojBXylA1GlF8G6LkorRSteI5vdehDK Pqb8rnMXwJsOsmy0livunYVA8ri9m32La/DkPuzEjIxH3+sFLWBFKEVV3kS+vvVcqkdI nOBke0SwsvodtG+oInm8qAcj3jBlb/0sS6hZG0ox/j293aRFrMDg4nyPZfZo4iCa8CCL +7g0W+LbABGTYhV2MEaeikmRftxDtdPSqtWrNY9fFIR8CZaTGUg4GKzfFFqRLbP65uiw z5Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=sbdHhD4pnQ5BvCxkKgntCgNTUFhscLM4XHj0ragKipA=; b=RhBZlMdmpaaDa1W2x738V72aHGTQb8aUDo2cVksjrM0ftuxj6L9gEWENsgSnO4hGgz /7Yjo0jEYnrQQuD/M15lvtzC/U5z8s0PeYIuDpxCeFoz8tP4WntZOuHgcyqINQx+MWNC gscQFCXe4Q0gC0DBzv1uRRDkzjqn2wSTNUxRWaMaYca1iftgG/gvKxn0yqa+32vjCE0L QYhV/zsLTtCuO0/K5CyLMAstf71q1tQzO6dIAA5ATu/uet4BvW9avpEdnUU2xFR/X+CU xLOPxrERSjoL6ovNbUsiolECsYmV3hhmw56cnDy4tkEp2nv96opJR6oePXmbs+Jx9VzO 9Yaw== X-Gm-Message-State: ALyK8tKTA9WIdOb/MmLgBzBvJgyQp8Pkrq4OFJL9medG/0KMehpVbcMKl/sSK9ohQGe1gKaFdc7kDsSdsTK+vA== X-Received: by 10.194.104.202 with SMTP id gg10mr2456636wjb.114.1468612146927; Fri, 15 Jul 2016 12:49:06 -0700 (PDT) MIME-Version: 1.0 From: Tim Tim Date: Fri, 15 Jul 2016 13:49:06 -0600 Message-ID: Subject: Intermittent 'apr_exit_why_e was 2' Precommit Block To: users@subversion.apache.org Content-Type: text/plain; charset=UTF-8 So, I have a most troubling issue: Intermittent blocking by our pre-commit hook, with the only output being: svn: 'pre-commit' hook failed (did not exit cleanly: apr_exit_why_e was 2, exitcode was -1073741502). with no output. This hook is very simple: just checks that the log for the commit contains something (anything), and succeed if there is content in the log. Otherwise fail. Straightforward, nothing special, log requirement on commits. This error only pops up intermittently: a service restart of Apache will give us about 24-48 hours without any errors. A server restart usually buys 1-2 weeks. Either way, this error comes back, and when it happens, there's usually about a 50-50 chance that when the user tries a second time, they do not get this error and can commit successfully, having changed nothing on their end or on the server. When the problem doesn't somehow fix itself, it usually persists for a couple of hours until we restart something server-side. But here's the kicker: The svn binaries that are checking the log for content are separate from the binaries used by the Apache service (Why is a good question. The answer is: I have no clue). Switching which binaries are used doesn't help at all, and makes it strange that an Apache restart alleviates things. I also have found no correlations to the amount of traffic we're receiving to the server, the amount of disk space used, disk i/o, or anything other machine metric. Memory is around 30% used, processor averages 50% load. I've tried troubleshooting the running httpd process and its spawns, found noerrors or unusual activity, and this blocking doesn't seem to affect all repositories, just a random few at a time. It's also extremely hard to reproduce intentionally. This server has ran for years with no issues, and nothing in our environment has changed recently. It also affects both Windows and Linux clients, both in and out of the U.S., and locale settings are universal. I've run stuck after pursuing this for the last 2 months with no success. If anyone could offer some more troubleshooting tips or possibilities, that would be a wonderful blessing. Some details on the server and SVN: Server: Windows 2008 R2 (VM) running Apache 2.2.24 (64bit) Subversion: 1.7.14 and 1.6.2 (Server has1.7.14) Traffic: Servers 7k+ repos (2k actively used), ~26mil hits a day (mostly ALM tools like Jenkins) Disk: 2.5 TB NTFS format