Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EFD7810F79 for ; Wed, 7 Aug 2013 18:28:01 +0000 (UTC) Received: (qmail 11027 invoked by uid 500); 7 Aug 2013 18:28:01 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 10994 invoked by uid 500); 7 Aug 2013 18:28:01 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 10986 invoked by uid 99); 7 Aug 2013 18:28:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Aug 2013 18:28:01 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ray.pfaff@apx-labs.com designates 207.46.163.207 as permitted sender) Received: from [207.46.163.207] (HELO na01-bl2-obe.outbound.protection.outlook.com) (207.46.163.207) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Aug 2013 18:27:57 +0000 Received: from CO1PR06MB078.namprd06.prod.outlook.com (10.242.164.20) by CO1PR06MB077.namprd06.prod.outlook.com (10.242.164.19) with Microsoft SMTP Server (TLS) id 15.0.731.16; Wed, 7 Aug 2013 18:27:20 +0000 Received: from CO1PR06MB078.namprd06.prod.outlook.com ([169.254.8.100]) by CO1PR06MB078.namprd06.prod.outlook.com ([169.254.8.100]) with mapi id 15.00.0731.000; Wed, 7 Aug 2013 18:27:20 +0000 From: Ray Pfaff To: "user@accumulo.apache.org" Subject: Re: Communication issue between zookeeper and accumulo Thread-Topic: Communication issue between zookeeper and accumulo Thread-Index: AQHOkrKaFiDDSetXUUSDOpgVwZFwO5mIROmA//+98gCAAEZcgP//z8eAgABLOgD//75pgAAJOX8w//+/GYCAAEOWgP//voOAgABDkYD//75hgIAB3ZyA///B3gA= Date: Wed, 7 Aug 2013 18:27:20 +0000 Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [108.48.202.82] x-forefront-prvs: 0931CB1479 x-forefront-antispam-report: SFV:NSPM;SFS:(24454002)(199002)(189002)(377454003)(47446002)(56816003)(54316002)(54356001)(47976001)(80976001)(77096001)(74662001)(74366001)(53806001)(81542001)(46102001)(74706001)(76786001)(19580405001)(16406001)(50986001)(49866001)(76796001)(83072001)(83322001)(19580395003)(4396001)(51856001)(56776001)(81342001)(47736001)(76176001)(36756003)(66066001)(63696002)(77982001)(74876001)(59766001)(69226001)(79102001)(16236675002)(65816001)(31966008)(24704002);DIR:OUT;SFP:;SCL:1;SRVR:CO1PR06MB077;H:CO1PR06MB078.namprd06.prod.outlook.com;CLIP:108.48.202.82;RD:InfoNoRecords;A:1;MX:1;LANG:en; Content-Type: multipart/alternative; boundary="_000_CE28098875Draypfaffapxlabscom_" MIME-Version: 1.0 X-OriginatorOrg: apx-labs.com X-Virus-Checked: Checked by ClamAV on apache.org --_000_CE28098875Draypfaffapxlabscom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable It's not 100, it's 1024. So if you're asking if it's been raised=85 yes. From: Eric Newton > Reply-To: "user@accumulo.apache.org" > Date: Wednesday, August 7, 2013 2:09 PM To: "user@accumulo.apache.org" > Subject: Re: Communication issue between zookeeper and accumulo Have you set maxClientCnxns as described in the README? You will need to r= estart zookeeper for this to have an effect. -Eric On Tue, Aug 6, 2013 at 1:40 PM, Ray Pfaff > wrote: It's from one of the tablet servers, but looking at one of the zookeeper se= rvers, it's exactly the same From: Sean Busbey > Reply-To: "user@accumulo.apache.org" > Date: Tuesday, August 6, 2013 1:35 PM To: Accumulo User List > Subject: Re: Communication issue between zookeeper and accumulo Is that on the ZK server or the TabletServer? Can we also see the other? On Tue, Aug 6, 2013 at 10:33 AM, Ray Pfaff > wrote: Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ACCEPT icmp -- anywhere anywhere icmp echo-repl= y ACCEPT icmp -- anywhere anywhere icmp echo-requ= est ACCEPT tcp -- anywhere anywhere tcp dpt:nrpe ACCEPT udp -- anywhere anywhere udp dpt:domain Chain FORWARD (policy DROP) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination From: Brendan Heussler > Reply-To: "user@accumulo.apache.org" > Date: Tuesday, August 6, 2013 1:27 PM To: "user@accumulo.apache.org" > Subject: Re: Communication issue between zookeeper and accumulo What is the output of iptables --list? Brendan On Tue, Aug 6, 2013 at 1:25 PM, Ray Pfaff > wrote: Not sure what you mean. I get the error "Fatal ip6_tables not found." I'm= assuming that means disabled? From: , "Charles H." > Reply-To: "user@accumulo.apache.org" > Date: Tuesday, August 6, 2013 1:18 PM To: "user@accumulo.apache.org" > Subject: RE: Communication issue between zookeeper and accumulo And iptables? From: user-return-2837-CHARLES.H.OTT=3Dsaic.com@accumulo.apache.org [mailto:user= -return-2837-CHARLES.H.OTT=3Dsaic.com@accumulo.apache.org] On Behalf Of Ray= Pfaff Sent: Tuesday, August 06, 2013 12:54 PM To: user@accumulo.apache.org Subject: Re: Communication issue between zookeeper and accumulo Yes, it is disabled, so that's not the problem. From: Sean Busbey > Reply-To: "user@accumulo.apache.org" > Date: Tuesday, August 6, 2013 12:48 PM To: Accumulo User List > Subject: Re: Communication issue between zookeeper and accumulo Hi Ray! Can you confirm that IPv6 is disabled? On Tue, Aug 6, 2013 at 9:19 AM, Ray Pfaff > wrote: I'm not sure if I can provide those due to the contract I'm working. I rea= lly don't want to diverge this conversation from the original question I'm = asking (which is a problem even running one tablet server per machine) but = are you saying that setting tserver.port.search =3D true shouldn't be done?= I found this to be an undocumented way of running more than one tablet se= rver per system. I'm still not convinced that this leads to stability issu= es on tablet servers. As I said, it's undocumented. From: Eric Newton > Reply-To: "user@accumulo.apache.org" > Date: Tuesday, August 6, 2013 11:12 AM To: "user@accumulo.apache.org" > Subject: Re: Communication issue between zookeeper and accumulo Interesting. You could not get similar performance improvements by increas= ing the size of the JVM, the number of threads, or the number of tablets pe= r server? If you have details about what configurations you've tried and the performa= nce numbers you found, please open a ticket. This would indicate that we h= ave some unnecessary bottleneck in the tserver. -Eric On Tue, Aug 6, 2013 at 11:00 AM, Ray Pfaff > wrote: Because we found this to be the optimal number of tablet servers in our tes= ting. It performs better than one per machine. I'm not convinced that the= stability issues make it worthwhile. Doesn't affect my problem anyway. I get this error whether I run one or fo= ur tablet servers. Running four just makes it a bigger issue to get back u= p after failure. From: Eric Newton > Reply-To: "user@accumulo.apache.org" > Date: Tuesday, August 6, 2013 10:56 AM To: "user@accumulo.apache.org" > Subject: Re: Communication issue between zookeeper and accumulo I'm running 4 tservers per machine dedicated to the tablet servers Why? -- Sean -- Sean --_000_CE28098875Draypfaffapxlabscom_ Content-Type: text/html; charset="Windows-1252" Content-ID: Content-Transfer-Encoding: quoted-printable
It's not 100, it's 1024.  So if you're asking if it's been raised= =85 yes.

From: Eric Newton <eric.newton@gmail.com>
Reply-To: "user@accumulo.apache.org" <user@accumulo.apache.org>
Date: Wednesday, August 7, 2013 2:0= 9 PM
To: "user@accumulo.apache.org" <user@accumulo.apache.org>
Subject: Re: Communication issue be= tween zookeeper and accumulo

Have you set maxClientCnxns as described in the README? &n= bsp;You will need to restart zookeeper for this to have an effect.

-Eric



On Tue, Aug 6, 2013 at 1:40 PM, Ray Pfaff <ray.pfaff@a= px-labs.com> wrote:
It's from one of the tablet servers, but looking at one of the zookeep= er servers, it's exactly the same

From: Sean Busbey= <busbey@cloude= ra.com>
Reply-To: "user@accumulo.apache.org&quo= t; <user@a= ccumulo.apache.org>
Date: Tuesday, August 6, 2013 1:35 = PM

To: Accumulo User List <user@accumulo.apache= .org>
Subject: Re: Communication issue be= tween zookeeper and accumulo

Is that on the ZK server or the TabletServer? Can we also = see the other?


On Tue, Aug 6, 2013 at 10:33 AM, Ray Pfaff <ray.pfaff@a= px-labs.com> wrote:
Chain INPUT (policy ACCEPT)
target     prot opt source          = ;     destination         
ACCEPT     tcp  --  anywhere       =       anywhere            tcp = dpt:ssh 
ACCEPT     icmp --  anywhere        = ;     anywhere            icmp echo= -reply 
ACCEPT     icmp --  anywhere        = ;     anywhere            icmp echo= -request 
ACCEPT     tcp  --  anywhere       =       anywhere            tcp = dpt:nrpe 
ACCEPT     udp  --  anywhere       =       anywhere            udp = dpt:domain 

Chain FORWARD (policy DROP)
target     prot opt source          = ;     destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source          = ;     destination

From: Brendan Heussler <bheussler@gmail.com&g= t;
Reply-To: "user@accumulo.apache.org&quo= t; <user@a= ccumulo.apache.org>
Date: Tuesday, August 6, 2013 1:27 = PM
To: "user@accumulo.apache.org" <= ;user@accumul= o.apache.org>

Subject: Re: Communication issue be= tween zookeeper and accumulo

What is the output of iptables --list?



Brendan


On Tue, Aug 6, 2013 at 1:25 PM, Ray Pfaff <ray.pfaff@a= px-labs.com> wrote:
Not sure what you mean.  I get the error "Fatal ip6_tables n= ot found."  I'm assuming that means disabled?

From: <Ott>, "Charles H.= " <CHAR= LES.H.OTT@saic.com>
Reply-To: "user@accumulo.apache.org&quo= t; <user@a= ccumulo.apache.org>
Date: Tuesday, August 6, 2013 1:18 = PM
To: "user@accumulo.apache.org" <= ;user@accumul= o.apache.org>
Subject: RE: Communication issue be= tween zookeeper and accumulo

And iptables?=

 

From: user-return-2837-CHARLES.H.OTT=3Dsaic.com@accumulo.apache.org [mailto:user-return-2837-CHARLES.H.OTT=3Dsaic.com@accumulo.= apache.org] On Behalf Of Ray Pfaff
Sent: Tuesday, August 06, 2013 12:54 PM
To: us= er@accumulo.apache.org
Subject: Re: Communication issue between zookeeper and accumulo

 

Yes, it is disabled, so that's not the problem.=

 

From: Sean Busbey <busbey@cloudera.com>
Reply-To: "user@accumulo.apache.org" <user@accumulo.apache.org>
Date: Tuesday, August 6, 2013 12:48 PM
To: Accumulo User List <user@accumulo.apache.org>
Subject: Re: Communication issue between zookeeper and accumulo

 

Hi Ray!

 

Can you confirm that IPv6 is disabled?

 

On Tue, Aug 6, 2013 at 9:19 AM, Ray Pfaff <ray.pfaff@apx-labs.com= > wrote:

I'm not sure if I can provide those due to the contract I= 'm working.  I really don't want to diverge this conversation from the= original question I'm asking (which is a problem even running one tablet server per machine) but are you saying tha= t setting tserver.port.search =3D true shouldn't be done?  I foun= d this to be an undocumented way of running more than one tablet server per= system.  I'm still not convinced that this leads to stability issues on tablet servers.  As I said, it's undocum= ented.

 

Date: Tuesday, August 6, 2013 11:12 AM


To: "user@accumulo.apache.org" <user@accumulo.apache.org>
Subject: Re: Communication issue between zookeeper and accumulo

 

Interesting.  You could not get similar performance = improvements by increasing the size of the JVM, the number of threads, or t= he number of tablets per server?

 

If you have details about what configurations you've trie= d and the performance numbers you found, please open a ticket.  This w= ould indicate that we have some unnecessary bottleneck in the tserver.

 

-Eric

 

 

On Tue, Aug 6, 2013 at 11:00 AM, Ray Pfaff <ray.pfaff@apx-labs.com> wrote:

 



 

--

Sean





--
Sean

 --_000_CE28098875Draypfaffapxlabscom_--