Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7000DD1DE for ; Fri, 21 Sep 2012 14:54:36 +0000 (UTC) Received: (qmail 93382 invoked by uid 500); 21 Sep 2012 14:54:36 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 93346 invoked by uid 500); 21 Sep 2012 14:54:36 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 93338 invoked by uid 99); 21 Sep 2012 14:54:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Sep 2012 14:54:36 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of tejay.e.cardon@lmco.com designates 192.31.106.12 as permitted sender) Received: from [192.31.106.12] (HELO mailfo01.lmco.com) (192.31.106.12) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Sep 2012 14:54:28 +0000 Received: from emss09g01.ems.lmco.com ([166.17.13.59]) by mailfo01.lmco.com (8.14.5/8.14.5) with ESMTP id q8LEoOAR013929 for ; Fri, 21 Sep 2012 15:50:26 +0100 Received: from CONVERSION2-DAEMON.lmco.com by lmco.com (PMDF V6.4 #31806) id <0MAP00B01FV0PQ@lmco.com> for user@accumulo.apache.org; Fri, 21 Sep 2012 14:50:24 +0000 (GMT) Received: from HDXHTPN9.us.lmco.com ([158.188.83.19]) by lmco.com (PMDF V6.4 #31806) with ESMTP id <0MAP009CAFVNPX@lmco.com> for user@accumulo.apache.org; Fri, 21 Sep 2012 14:50:12 +0000 (GMT) Received: from HDXDSP33.us.lmco.com ([fe80::d0ea:cdbc:7392:8296]) by HDXHTPN9.us.lmco.com ([fe80::bc2a:fd6f:439:c93f%15]) with mapi id 14.01.0355.002; Fri, 21 Sep 2012 08:50:11 -0600 Date: Fri, 21 Sep 2012 14:50:10 +0000 From: "Cardon, Tejay E" Subject: RE: EXTERNAL: Re: Failing Tablet Servers In-reply-to: X-Originating-IP: [158.188.95.3] To: "user@accumulo.apache.org" Message-id: <57754A39E408FB4994D100B1F4409AD80BE71E21@HDXDSP33.us.lmco.com> MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_kJr2Ipdtzam+b1pIx/ZIjw)" Content-language: en-US Thread-Topic: EXTERNAL: Re: Failing Tablet Servers Thread-Index: Ac2Xc7avSqHaW9BiSki+HQC1xTe+rAAwG20AAAxuqWD//6TCgIAAYESg Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: References: <57754A39E408FB4994D100B1F4409AD80BE71BC3@HDXDSP33.us.lmco.com> <57754A39E408FB4994D100B1F4409AD80BE71DC0@HDXDSP33.us.lmco.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.7.7855,1.0.431,0.0.0000 definitions=2012-09-21_01:2012-09-20,2012-09-21,1970-01-01 signatures=0 X-Virus-Checked: Checked by ClamAV on apache.org --Boundary_(ID_kJr2Ipdtzam+b1pIx/ZIjw) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Alright. So I'm changing it to: 1. moderate size mutations ~1,000 key/values per mutation. 2. Tserver_opts = 5g 3. Memory.maps = 3g 4. Swappiness = 0 (right now I'm at 20) It sounds like those are all settings I should fix anyway, so we'll do them all. I'll report back if that doesn't fix the problem. Thanks again for all the help Tejay Cardon From: Eric Newton [mailto:eric.newton@gmail.com] Sent: Friday, September 21, 2012 8:33 AM To: user@accumulo.apache.org Subject: Re: EXTERNAL: Re: Failing Tablet Servers We regularly send an overwhelming number of small key/value pairs to tablet servers (see the continuous ingest test). If your servers are going down with smaller mutations, send the logs again. I suspect that the tserver is being pushed into swap, and then the GC is taking too long. That causes the tserver to lose its lock in zookeeper. Make sure that swappiness is set to zero. -Eric On Fri, Sep 21, 2012 at 10:12 AM, Cardon, Tejay E > wrote: Jim, Eric, and Adam, Thanks. It sounds like you're all saying the same thing. Originally I was doing each key/value as its own mutation, and it was blowing up much faster (probably due to the volume/overhead of the mutation objects themselves. I'll try refactoring to break them up into something in-between. My keys are small (<25 Bytes), and my values are empty, but I'll aim for ~1,000 key/values per mutation and see how that works out for me. Eric, I was under the impression that the memory.maps setting was not very important when using native maps. Apparently I'm mistaken there. What does this setting control when in a native map setting? And, in general, what's the proper balance between tserver_opts and tserver.memory.maps? With regards to the "Finished gathering information from 24 servers in 27.45 seconds" Do you have any recommendations for how to chase down the bottleneck? I'm pretty sure I'm having GC issues, but I'm not sure what is causing them on the server side. I'm sending a fairly small number of very large mutation objects, which I'd expect to be a moderate problem for the GC, but not a huge one.. Thanks again to everyone for being so responsive and helpful. Tejay Cardon From: Eric Newton [mailto:eric.newton@gmail.com] Sent: Friday, September 21, 2012 8:03 AM To: user@accumulo.apache.org Subject: EXTERNAL: Re: Failing Tablet Servers A few items noted from your logs: tserver.memory.maps.max = 1G If you are giving your processes 10G, you might want to make the map larger, say 6G, and then reduce the JVM by 6G. Write-Ahead Log recovery complete for rz<;zw== (8 mutations applied, 8000000 entries created) You are creating rows with 1M columns. This is ok, but you might want to write them out more incrementally. WARN : Running low on memory That's pretty self-explanatory. I'm guessing that the very large mutations are causing the tablet servers to run out of memory before they are held waiting for minor compactions. Finished gathering information from 24 servers in 27.45 seconds Something is running slow, probably due to GC thrashing. WARN : Lost servers [10.1.24.69:9997[139d46130344b98]] And there's a server crashing, probably due to an OOM condition. Send smaller mutations. Maybe keep it to 200K column updates. You can still have 1M wide rows, just send 5 mutations. -Eric On Thu, Sep 20, 2012 at 5:05 PM, Cardon, Tejay E > wrote: I'm seeing some strange behavior on a moderate (30 node) cluster. I've got 27 tablet servers on large dell servers with 30GB of memory each. I've set the TServer_OPTS to give them each 10G of memory. I'm running an ingest process that uses AccumuloInputFormat in a MapReduce job to write 1,000 rows with each row containing ~1,000,000 columns in 160,000 families. The MapReduce initially runs quite quickly and I can see the ingest rate peak on the monitor page. However, after about 30 seconds of high ingest, the ingest falls to 0. It then stalls out and my map task are eventually killed. In the end, the map/reduce fails and I usually end up with between 3 and 7 of my Tservers dead. Inspecting the tserver.err logs shows nothing, even on the nodes that fail. The tserver.out log shows a java OutOfMemoryError, and nothing else. I've included a zip with the logs from one of the failed tservers and a second one with the logs from the master. Other than the out of memory, I'm not seeing anything that stands out to me. If I reduce the data size to only 100,000 columns, rather than 1,000,000, the process takes about 4 minutes and completes without incident. Am I just ingesting too quickly? Thanks, Tejay Cardon --Boundary_(ID_kJr2Ipdtzam+b1pIx/ZIjw) Content-type: text/html; charset=us-ascii Content-transfer-encoding: 7BIT

Alright.  So I’m changing it to:

1.        moderate size mutations ~1,000 key/values per mutation.

2.       Tserver_opts = 5g

3.       Memory.maps = 3g

4.       Swappiness = 0 (right now I’m at 20)

 

It sounds like those are all settings I should fix anyway, so we’ll do them all.  I’ll report back if that doesn’t fix the problem.

 

Thanks again for all the help

 

Tejay Cardon

From: Eric Newton [mailto:eric.newton@gmail.com]
Sent: Friday, September 21, 2012 8:33 AM
To: user@accumulo.apache.org
Subject: Re: EXTERNAL: Re: Failing Tablet Servers

 

We regularly send an overwhelming number of small key/value pairs to tablet servers (see the continuous ingest test).

 

If your servers are going down with smaller mutations, send the logs again.  I suspect that the tserver is being pushed into swap, and then the GC is taking too long.  That causes the tserver to lose its lock in zookeeper.

 

Make sure that swappiness is set to zero.

 

-Eric

On Fri, Sep 21, 2012 at 10:12 AM, Cardon, Tejay E <tejay.e.cardon@lmco.com> wrote:

Jim, Eric, and Adam,

Thanks.  It sounds like you’re all saying the same thing.  Originally I was doing each key/value as its own mutation, and it was blowing up much faster (probably due to the volume/overhead of the mutation objects themselves.  I’ll try refactoring to break them up into something in-between.  My keys are small (<25 Bytes), and my values are empty, but I’ll aim for ~1,000 key/values per mutation and see how that works out for me.

 

Eric,

I was under the impression that the memory.maps setting was not very important when using native maps.  Apparently I’m mistaken there.  What does this setting control when in a native map setting?  And, in general, what’s the proper balance between tserver_opts and tserver.memory.maps?

 

With regards to the “Finished gathering information from 24 servers in 27.45 seconds”  Do you have any recommendations for how to chase down the bottleneck?  I’m pretty sure I’m having GC issues, but I’m not sure what is causing them on the server side.  I’m sending a fairly small number of very large mutation objects, which I’d expect to be a moderate problem for the GC, but not a huge one..

 

Thanks again to everyone for being so responsive and helpful.

 

Tejay Cardon

 

 

From: Eric Newton [mailto:eric.newton@gmail.com]
Sent: Friday, September 21, 2012 8:03 AM


To: user@accumulo.apache.org
Subject: EXTERNAL: Re: Failing Tablet Servers

 

A few items noted from your logs:

 

tserver.memory.maps.max = 1G

 

If you are giving your processes 10G, you might want to make the map larger, say 6G, and then reduce the JVM by 6G.

 

Write-Ahead Log recovery complete for rz<;zw== (8 mutations applied, 8000000 entries created)

 

You are creating rows with 1M columns.  This is ok, but you might want to write them out more incrementally.

 

WARN : Running low on memory

 

That's pretty self-explanatory.  I'm guessing that the very large mutations are causing the tablet servers to run out of memory before they are held waiting for minor compactions.

 

Finished gathering information from 24 servers in 27.45 seconds

 

Something is running slow, probably due to GC thrashing.

 

WARN : Lost servers [10.1.24.69:9997[139d46130344b98]]

 

And there's a server crashing, probably due to an OOM condition.

 

Send smaller mutations.  Maybe keep it to 200K column updates.  You can still have 1M wide rows, just send 5 mutations.

 

-Eric

 

On Thu, Sep 20, 2012 at 5:05 PM, Cardon, Tejay E <tejay.e.cardon@lmco.com> wrote:

I’m seeing some strange behavior on a moderate (30 node) cluster.  I’ve got 27 tablet servers on large dell servers with 30GB of memory each.  I’ve set the TServer_OPTS to give them each 10G of memory.  I’m running an ingest process that uses AccumuloInputFormat in a MapReduce job to write 1,000 rows with each row containing ~1,000,000 columns in 160,000 families.  The MapReduce initially runs quite quickly and I can see the ingest rate peak on the  monitor page.  However, after about 30 seconds of high ingest, the ingest falls to 0.  It then stalls out and my map task are eventually killed.  In the end, the map/reduce fails and I usually end up with between 3 and 7 of my Tservers dead.

 

Inspecting the tserver.err logs shows nothing, even on the nodes that fail.  The tserver.out log shows a java OutOfMemoryError, and nothing else.  I’ve included a zip with the logs from one of the failed tservers and a second one with the logs from the master.  Other than the out of memory, I’m not seeing anything that stands out to me.

 

If I reduce the data size to only 100,000 columns, rather than 1,000,000, the process takes about 4 minutes and completes without incident.

 

Am I just ingesting too quickly?

 

Thanks,

Tejay Cardon

 

 

--Boundary_(ID_kJr2Ipdtzam+b1pIx/ZIjw)--