accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ott, Charles H." <CHARLES.H....@saic.com>
Subject RE: Thread "shell" Stuck on IO
Date Wed, 17 Oct 2012 21:40:11 GMT
I believe you have already helped me get on the right track...

First, 192.168.0.130 is the IP that the VM came with preconfigured.
I changed the IP for this new environment in RHEL5 and "most" everything
seems to be running... however, the fact that it is reporting
192.168.0.130 tells me that somewhere in the logger configuration it's
still using the old IP?

All of the properties files I have looked at specify the hostname, not
IP... I checked the hosts file and the hostname is resolving the proper
IP, so that shouldn't be an issue.

When I try to start the logger with:

# ./cloudbase.sh logger

 I see:
Failed to initialize log service args=[]
	java.io.IOException: Failed to acquire lock file
		at
cloudbase.server.logger.LogService.<init>(LogService.java:122)
		at
cloudbase.server.logger.LogService.main(LogService.java:83)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
		at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
		at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
		at java.lang.reflect.Method.invoke(Method.java:597)
		at cloudbase.start.Main$1.run(Main.java:73)
		at java.lang.Thread.run(Thread.java:662)

-----Original Message-----
From: user-return-1492-CHARLES.H.OTT=saic.com@accumulo.apache.org
[mailto:user-return-1492-CHARLES.H.OTT=saic.com@accumulo.apache.org] On
Behalf Of Keith Turner
Sent: Wednesday, October 17, 2012 5:09 PM
To: user@accumulo.apache.org
Subject: Re: Thread "shell" Stuck on IO

Is the logger at 192.168.0.130 running.   The stack trace indicates
that the master was attempting to contact the logger at 192.168.0.130 to
initiate log recovery.

On Wed, Oct 17, 2012 at 4:58 PM, Ott, Charles H.
<CHARLES.H.OTT@saic.com> wrote:
> I am using a VMware ESXi 4.1 server  with Cloudbase(Accumulo)  on
RHEL5.
>
> I cannot start with a fresh install because I am somewhat required to 
> use the preconfigured image on the vm. (business rules out of my 
> hands)
>
> Unfortunately the support for this preconfigured instance is not 
> available and I am tasked with getting it working anyway...
>
>
>
> I am able to log into the shell and view the tables, however if  I 
> attempt to create a table or perform a scan, a line return is shown 
> and then it just hangs there until finally throwing the following
error:
>
> WARN thread "shell" stuck on IO to ssdev:9999:9999 (0) for at least 
> 120044 ms.
>
>
>
> I did also discover that 9999 is the property: master.port.client in 
> my conf/accumulo-site.xml
>
>
>
> There is also an event log that was added to the VM with web based UI
> reporting:
>
> Unable to recover
>
192.168.0.130:11224/b4da830b-8ecb-4868-a480-35a39f4af17a(java.io.IOExcep
tion:
> org.apache.thrift.transport.TTransportException:
java.net.ConnectException:
> Connection timed out)
>
>          java.io.IOException:
> org.apache.thrift.transport.TTransportException:
java.net.ConnectException:
> Connection timed out
>
>                  at
> cloudbase.server.tabletserver.log.RemoteLogger.<init>(RemoteLogger.jav
> a:75)
>
>                  at
> cloudbase.server.master.CoordinateRecoveryTask$RecoveryJob.startCopy(C
> oordinateRecoveryTask.java:109)
>
>                  at
> cloudbase.server.master.CoordinateRecoveryTask$RecoveryJob.access$400(
> CoordinateRecoveryTask.java:93)
>
>                  at
> cloudbase.server.master.CoordinateRecoveryTask.recover(CoordinateRecov
> eryTask.java:279)
>
>                  at
> cloudbase.server.master.Master$TabletGroupWatcher.run(Master.java:1155
> )
>
>          Caused by: org.apache.thrift.transport.TTransportException:
> java.net.ConnectException: Connection timed out
>
>                  at
> cloudbase.core.client.impl.ThriftTransportPool.createNewTransport(Thri
> ftTransportPool.java:428)
>
>                  at
> cloudbase.core.client.impl.ThriftTransportPool.getTransport(ThriftTran
> sportPool.java:415)
>
>                  at
> cloudbase.core.client.impl.ThriftTransportPool.getTransport(ThriftTran
> sportPool.java:392)
>
>                  at
> cloudbase.core.util.ThriftUtil.getClient(ThriftUtil.java:58)
>
>                  at
> cloudbase.server.tabletserver.log.RemoteLogger.<init>(RemoteLogger.jav
> a:73)
>
>                  ... 4 more
>
>          Caused by: java.net.ConnectException: Connection timed out
>
>                  at sun.nio.ch.Net.connect(Native Method)
>
>                  at
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)
>
>                  at 
> sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:81)
>
>                  at 
> sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:65)
>
>                  at
> cloudbase.core.util.TTimeoutTransport.create(TTimeoutTransport.java:23
> )
>
>                  at
> cloudbase.core.client.impl.ThriftTransportPool.createNewTransport(Thri
> ftTransportPool.java:426)
>
>                  ... 8 more
>
>
>
>
>
> I have seen posts relating this to the walogs folder not being 
> available, but I have checked that and the .lock file is being created
automatically.
>
> A #netstat | grep 9999 shows no processes using 9999 before logging 
> into the shell... so Im not sure there is a port conflict either.
>
>
>
> Any thoughts on the matter would be greatly appreciated.

Mime
View raw message