accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: MiniAccumuloClutser Unit Test Problems
Date Fri, 26 Jun 2015 14:28:53 GMT
So the problem is with tests for different classes, with a MAC per class,
running in parallel? if so, that sounds like a bug.

On Fri, Jun 26, 2015 at 9:14 AM, Parise, Jonathan <Jonathan.Parise@gd-ms.com
> wrote:

> Josh,
>
> Sorry for the confusion. I'll try and explain again with better
> terminology.
>
> I have several test classes, each of which contain several test methods.
> For each test class I have an @BeforeClass that configures and starts a
> MAC. I also have an @AfterClass that calls MAC.stop().
>
> So the flow is MAC is created, 1-n tests in the class run against it, MAC
> is stopped, repeat for next N tests.
>
> This way several @test methods in the same class use the same MAC. I
> mostly did this to make the tests run faster. I understand that having
> several test methods share a MAC could cause test state pollution, but I am
> careful to avoid that.
>
> The issues I was seeing is that if I simply ran "mvn test", the first few
> tests would pass and then eventually one of the tests would get stuck
> forever and just keep throwing connection errors.
>
> When I changed the maven configuration to force the tests to run serially,
> the issue stopped occurring.
>
> I'm not sure if there tests are expected to work when run in parallel. My
> best guess is that the tests may conflict with each other over ports or
> something like that.
>
> I'd like to understand why changing the test running behavior fixed this
> issue. Also, I think it would be good to document this somewhere. The MAC
> javadoc and also the user's guide should provide details about using the
> MAC for tests and properly configuring those tests.
>
> Does that make more sense?
>
> Jon
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Friday, June 26, 2015 10:00 AM
> To: user@accumulo.apache.org
> Subject: Re: MiniAccumuloClutser Unit Test Problems
>
> Jonathan,
>
> If you're not seeing consistent behavior starting and stopping a
> MiniAccumuloCluster repeatedly, that's a bug. If you can provide a code
> which shows this problem, that'd be a huge help.
>
> If you can get a list of the processes running when you see this happen
> and cross-reference it with what processes should be running, that would
> also go a long way in trying to debug this.
>
> I am a little confused to your specific situation. You said that you
> construct and start a MAC instance in a BeforeClass and stop it in an
> AfterClass, but then you said that you start and stop it for each test.
> Are you saying that after the third construction and use of a MAC, you see
> problems? Or, are you saying that you stop and start each MAC instance
> before you run the @Test methods?
>
> - Josh
>
> Parise, Jonathan wrote:
> > Hello,
> >
> > I have been writing some J-unit tests based on the MiniAccumuloCluster
> > class. I'm experiencing some issues when several of the tests run back
> > to back. Before I get into the error, let me explain how the tests
> > work in general. Also, I am using Accumulo 1.6.2.
> >
> > Each test has an @BeforeClass method that first creates a new random
> > directory. Then makes a new MiniAccumuloCluster instance using that
> > directory as the dir parameter. Then, I call MiniAccumuloCluster.start().
> >
> > There are several @Test methods in each test class. The typical
> > pattern for them is that they create any necessary tables, write some
> > data into those tables and then scan to verify it was written
> > correctly. Basically they are testing that I can serialize and
> > deserialize various types of Objects correctly.
> >
> > Then the test class as an @AfterClass method that calls
> > MiniAccumuloCluster.stop(). It also deletes the random directory used
> > by the previous test.
> >
> > The issue I am running into is that generally the first test or two
> > run fine. However, the third test usually gets stuck in the
> > MiniAccumuloCluster startup. It just keeps complaining about being
> > unable to connect. Note that if the test is run independently it
> > passes just fine. When run back to back, I see errors like this one
> repeatedly:
> >
> > 2015-06-26 09:10:24,352 INFO [main-SendThread(localhost:47046)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:47046
> >
> > 2015-06-26 09:10:24,353 WARN [main-SendThread(localhost:47046)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > java.net.ConnectException: Connection refused
> >
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> >
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> >
> > 2015-06-26 09:10:24,956 INFO [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:10406
> >
> > 2015-06-26 09:10:24,957 WARN [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > 2015-06-26 09:10:51,764 INFO [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:10406
> >
> > 2015-06-26 09:10:51,765 WARN [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > java.net.ConnectException: Connection refused
> >
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> >
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> >
> > 2015-06-26 09:10:52,572 INFO [main-SendThread(localhost:47046)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:47046
> >
> > 2015-06-26 09:10:52,573 WARN [main-SendThread(localhost:47046)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > java.net.ConnectException: Connection refused
> >
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> >
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> >
> > 2015-06-26 09:10:52,890 INFO [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:10406
> >
> > 2015-06-26 09:10:52,891 WARN [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > java.net.ConnectException: Connection refused
> >
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> >
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> >
> > 2015-06-26 09:10:54,191 INFO [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:10406
> >
> > 2015-06-26 09:10:54,192 WARN [main-SendThread(localhost:10406)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffcb2390004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > java.net.ConnectException: Connection refused
> >
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> >
> > at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> >
> > 2015-06-26 09:10:54,471 INFO [main-SendThread(localhost:47046)]
> > zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1058)) - Opening
> > socket connection to server localhost/127.0.0.1:47046
> >
> > 2015-06-26 09:10:54,471 WARN [main-SendThread(localhost:47046)]
> > zookeeper.ClientCnxn (ClientCnxn.java:run(1185)) - Session
> > 0x14e2ffc86d80004 for server null, unexpected error, closing socket
> > connection and attempting reconnect
> >
> > In the ZooKeeperServerMain.out I see lines like this repeating several
> > thousand times:
> >
> > 2015-06-26 08:39:44,810 INFO [SyncThread:0] server.NIOServerCnxn
> > (NIOServerCnxn.java:finishSessionInit(1580)) - Established session
> > 0x14e2fe179eb0000 with negotiated timeout 30000 for client
> > /127.0.0.1:36311
> >
> > 2015-06-26 08:39:45,278 INFO [ProcessThread:-1]
> > server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> > - Got user-level KeeperException when processing
> > sessionid:0x14e2fe179eb0000 type:create cxid:0x31
> > zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> > Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
> > Error:KeeperErrorCode = NodeExists for
> > /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/+r/conf
> >
> > 2015-06-26 08:39:45,300 INFO [ProcessThread:-1]
> > server.PrepRequestProcessor (PrepRequestProcessor.java:pRequest(419))
> > - Got user-level KeeperException when processing
> > sessionid:0x14e2fe179eb0000 type:create cxid:0x33
> > zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
> > Path:/accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
> > Error:KeeperErrorCode = NodeExists for
> > /accumulo/87e85c1f-eb40-4695-b11d-67ed88586648/tables/!0/conf
> >
> > I've thought about putting a Thread.sleep() call after the
> > MiniAccumuloCluster.stop() call, but that certainly seems brittle. I'm
> > not sure if that would improve the situation.
> >
> > It seems to me that the MiniAccumuloCluster does not behave well when
> > instances are started and stopped several times. I am running the
> > tests through Maven using the default test behavior.
> >
> > Could it be something with Maven? Maybe I need to be more explicit
> > when telling it how to run the tests?
> >
> > Anyone have any insight into what is going wrong here? In general is
> > my usage pattern correct for MiniAccumuloCluster?
> >
> > Thanks,
> >
> > Jon Parise
> >
> > Senior Software Engineer
> >
> > Viz | General Dynamics Missons Systems
> >
> > *This message and/or attachments may include information subject to GD
> > Corporate Policies 07-103 and 07-105 and is intended to be accessed
> > only by authorized recipients. Use, storage and transmission are
> > governed by General Dynamics and its policies. Contractual
> > restrictions apply to third parties. Recipients should refer to the
> > policies or contract to determine proper handling. Unauthorized
> > review, use, disclosure or distribution is prohibited. If you are not
> > an intended recipient, please contact the sender and destroy all
> > copies of the original message.*
> >
>



-- 
Sean

Mime
View raw message