accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: MiniAccumuloClutser Unit Test Problems
Date Fri, 26 Jun 2015 19:32:05 GMT
They should all be using '0' which will cause the OS to allocate a 
dynamic port. I doubt your parallelism is so high on your tests that you 
exhaust the dynamic port range from the OS :)

Parise, Jonathan wrote:
> Should I be doing something to dynamically allocate the ports? I am
> using the MAC constructor that simply takes a File and a root password.
>
>  From what I am seeing in the logs, it does look like ports could be the
> cause. If all the MAC instances try to use the same ports, they would
> conflict.
>
> Jon
>
> *From:*David Medinets [mailto:david.medinets@gmail.com]
> *Sent:* Friday, June 26, 2015 2:36 PM
> *To:* accumulo-user
> *Subject:* Re: MiniAccumuloClutser Unit Test Problems
>
> How are the ports dynamically allocated? If five MAC are started at the
> same time, could they be stepping on each other?
>
> On Fri, Jun 26, 2015 at 11:36 AM, Josh Elser <josh.elser@gmail.com
> <mailto:josh.elser@gmail.com>> wrote:
>
> Makes sense. Thanks for clarifying!
>
> Definitely send us a JIRA issue so we can track the issue and try to get
> to the bottom of it.
>
>
>
> Parise, Jonathan wrote:
>
> Yes each test has a unique directory for the MAC.
>
> I use the following code to ensure that:
>
> storageDirectory_ = new File("atf-temp" + Path.SEPARATOR + "accumulo-test-"
> + UUID.randomUUID().toString());
>
> Basically this makes a new random UUID so that I get very unique paths.
> All of the directories end up in atf-temp. This way I can clean them up
> easily.
>
> Jon
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>]
> Sent: Friday, June 26, 2015 10:50 AM
> To: user@accumulo.apache.org <mailto:user@accumulo.apache.org>
> Subject: Re: MiniAccumuloClutser Unit Test Problems
>
> Did each test class use a unique directory for MAC?
>
> Assuming they're all unique, it sounds like a bug. I know some devs run
> our suite of integration tests in parallel which use MAC out of the box.
> If all of the above is true, please feel free to open up an issue on
> JIRA and we can look into it more there.
>
> https://issues.apache.org/jira/secure/CreateIssue!default.jspa
>
> Thanks!
>
> Parise, Jonathan wrote:
>
> Josh,
>
> Sorry for the confusion. I'll try and explain again with better terminology.
>
> I have several test classes, each of which contain several test methods.
> For each test class I have an @BeforeClass that configures and starts a
> MAC. I also have an @AfterClass that calls MAC.stop().
>
> So the flow is MAC is created, 1-n tests in the class run against it,
> MAC is stopped, repeat for next N tests.
>
> This way several @test methods in the same class use the same MAC. I
> mostly did this to make the tests run faster. I understand that having
> several test methods share a MAC could cause test state pollution, but I
> am careful to avoid that.
>
> The issues I was seeing is that if I simply ran "mvn test", the first
> few tests would pass and then eventually one of the tests would get
> stuck forever and just keep throwing connection errors.
>
> When I changed the maven configuration to force the tests to run
> serially, the issue stopped occurring.
>
> I'm not sure if there tests are expected to work when run in parallel.
> My best guess is that the tests may conflict with each other over ports
> or something like that.
>
> I'd like to understand why changing the test running behavior fixed this
> issue. Also, I think it would be good to document this somewhere. The
> MAC javadoc and also the user's guide should provide details about using
> the MAC for tests and properly configuring those tests.
>
> Does that make more sense?
>
> Jon
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>]
> Sent: Friday, June 26, 2015 10:00 AM
> To: user@accumulo.apache.org <mailto:user@accumulo.apache.org>
> Subject: Re: MiniAccumuloClutser Unit Test Problems
>
> Jonathan,
>
> If you're not seeing consistent behavior starting and stopping a
> MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code
> which shows this problem, that'd be a huge help.
>
> If you can get a list of the processes running when you see this happen
> and cross-reference it with what processes should be running, that would
> also go a long way in trying to debug this.
>
> I am a little confused to your specific situation. You said that you
> construct and start a MAC instance in a BeforeClass and stop it in an
> AfterClass, but then you said that you start and stop it for each test.
> Are you saying that after the third construction and use of a MAC, you
> see problems? Or, are you saying that you stop and start each MAC
> instance before you run the @Test methods?
>
> - Josh
>
> Parise, Jonathan wrote:
>
> Hello,
>
> I have been writing some J-unit tests based on the
> MiniAccumuloCluster class. I'm experiencing some issues when several
> of the tests run back to back. Before I get into the error, let me
> explain how the tests work in general. Also, I am using Accumulo 1.6.2.
>
> Each test has an @BeforeClass method that first creates a new random
> directory. Then makes a new MiniAccumuloCluster instance using that
> directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
>
> There are several @Test methods in each test class. The typical
> pattern for them is that they create any necessary tables, write some
> data into those tables and then scan to verify it was written
> correctly. Basically they are testing that I can serialize and
> deserialize various types of Objects correctly.
>
> Then the test class as an @AfterClass method that calls
> MiniAccumuloCluster.stop(). It also deletes the random directory used
> by the previous test.
>
> The issue I am running into is that generally the first test or two
> run fine. However, the third test usually gets stuck in the
> MiniAccumuloCluster startup. It just keeps complaining about being
> unable to connect. Note that if the test is run independently it
> passes just fine. When run back to back, I see errors like this one
> repeatedly:
>
> 2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
> <http://127.0.0.1:47046>
>
> 2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
> <http://127.0.0.1:10406>
>
> 2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> 2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
> <http://127.0.0.1:10406>
>
> 2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
> <http://127.0.0.1:47046>
>
> 2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
> <http://127.0.0.1:10406>
>
> 2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:10406
> <http://127.0.0.1:10406>
>
> 2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739
> )
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
>
> 2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> socket connection to server localhost/127.0.0.1:47046
> <http://127.0.0.1:47046>
>
> 2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
> zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> connection and attempting reconnect
>
> In the ZooKeeperServerMain.out I see lines like this repeating
> several thousand times:
>
> 2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
> (NIOServerCnxn.java:finishSessionInit(1580)) - Established session
> 0x14e2fe179eb0000 with negotiated timeout 30000 for client
> /127.0.0.1:36311 <http://127.0.0.1:36311>
>
> 2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> - Got user-level KeeperException when processing
> sessionid:0x14e2fe179eb0000 type:create cxid:0x31
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
> Error:KeeperErrorCode = NodeExists for
> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
>
> 2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
> server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> - Got user-level KeeperException when processing
> sessionid:0x14e2fe179eb0000 type:create cxid:0x33
> zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
> Error:KeeperErrorCode = NodeExists for
> /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
>
> I've thought about putting a Thread.sleep() call after the
> MiniAccumuloCluster.stop() call, but that certainly seems brittle.
> I'm not sure if that would improve the situation.
>
> It seems to me that the MiniAccumuloCluster does not behave well when
> instances are started and stopped several times. I am running the
> tests through Maven using the default test behavior.
>
> Could it be something with Maven? Maybe I need to be more explicit
> when telling it how to run the tests?
>
> Anyone have any insight into what is going wrong here? In general is
> my usage pattern correct for MiniAccumuloCluster?
>
> Thanks,
>
> Jon Parise
>
> Senior Software Engineer
>
> Viz | General Dynamics Missons Systems
>
> *This message and/or attachments may include information subject to
> GD Corporate Policies 07-103 and 07-105 and is intended to be
> accessed only by authorized recipients. Use, storage and transmission
> are governed by General Dynamics and its policies. Contractual
> restrictions apply to third parties. Recipients should refer to the
> policies or contract to determine proper handling. Unauthorized
> review, use, disclosure or distribution is prohibited. If you are not
> an intended recipient, please contact the sender and destroy all
> copies of the original message.*
>

Mime
View raw message