Return-Path: X-Original-To: apmail-accumulo-dev-archive@www.apache.org Delivered-To: apmail-accumulo-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E1CFDE68A for ; Fri, 4 Jan 2013 00:36:15 +0000 (UTC) Received: (qmail 1410 invoked by uid 500); 4 Jan 2013 00:36:15 -0000 Delivered-To: apmail-accumulo-dev-archive@accumulo.apache.org Received: (qmail 1369 invoked by uid 500); 4 Jan 2013 00:36:15 -0000 Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@accumulo.apache.org Delivered-To: mailing list dev@accumulo.apache.org Received: (qmail 1340 invoked by uid 99); 4 Jan 2013 00:36:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jan 2013 00:36:15 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS,UNPARSEABLE_RELAY X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [129.55.12.46] (HELO mx2.ll.mit.edu) (129.55.12.46) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jan 2013 00:36:04 +0000 Received: from LLE2K7-HUB02.mitll.ad.local (LLE2K7-HUB02.mitll.ad.local) by mx2.ll.mit.edu (unknown) with ESMTP id r040ZZ8N017706; Thu, 3 Jan 2013 19:35:41 -0500 From: "Kepner, Jeremy - 0553 - MITLL" To: "dev@accumulo.apache.org" , "vines@apache.org" CC: "Kepner, Jeremy - 0553 - MITLL" Date: Thu, 3 Jan 2013 19:31:44 -0500 Subject: Re: ingest performance oscillations and Xceivers Thread-Topic: ingest performance oscillations and Xceivers Thread-Index: Ac3qEuCOBpmt7z5iTFK7UkMIieANVw== Message-ID: <40FDB156-E32B-41C0-B483-EA69B2178586@ll.mit.edu> References: <20130102191131.GA12538@ll.mit.edu> <7C21BEED-BDBD-40D4-97CF-EF60B776540A@ll.mit.edu> <20130103222534.GA23212@ll.mit.edu> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/mixed; boundary="_002_40FDB156E32B41C0B483EA69B2178586llmitedu_" MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.9.8327,1.0.431,0.0.0000 definitions=2013-01-03_08:2013-01-03,2013-01-03,1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 suspectscore=7 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=6.0.2-1211240000 definitions=main-1301030294 X-Virus-Checked: Checked by ClamAV on apache.org --_002_40FDB156E32B41C0B483EA69B2178586llmitedu_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Attached is the log for a set for 3 ingests. The first hump is with 1 inge= stor, the second hump is with 2 ingestors, and the third hump is with 3 ing= estors. The 3 ingestor case starts oscillating about 21:40. I don't see an= y spikes in any of the fields. It would be nice if there was also a plot o= f Xceivers since I think that anti-correlates. I do sometimes see the hold= times highlighted in red. The values are usually in hundreds of msec. On Jan 3, 2013, at 6:04 PM, John Vines wrote: > How many hard drives and what's your max minor/maor compactions set to? > These can severely limit your performance due to potential disk thrashing= . > If you observe the monitor, when ingest starts trailing, do you see it > undergoing, or possibly being backed up on, any form of compaction? And > lastly, are you see the bug eric mentioned above? See if the monitor is > reporting any warnings like those mentioned. --_002_40FDB156E32B41C0B483EA69B2178586llmitedu_--