accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: scan command hung
Date Thu, 01 Oct 2015 13:57:46 GMT
The client keeps going to server01

  connection to server01

And the server keeps returning nothing...

  Finished scan in 1.001 secs #results=0

This happens when the scan is active on the server side but the scan is
returning no data.  The server will return nothing to the client just to
see if its still there and interested.

A next step would be to jstack the tserver on server01 when this happens
and see what the scan is doing.



On Wed, Sep 30, 2015 at 1:56 PM, z11373 <z11373@outlook.com> wrote:

> Thanks Josh/Keith!
> To answer Josh's questions, that column family only has few rows, the other
> column families that work fine, one of them also only has few rows, and
> another one has million of rows. Not using locality group or any other
> fancy
> configuration.
>
> I tried 'debug on' per suggestion from Keith, and perhaps it's obvious to
> know the culprit from the output.
> Below are the snippets (the first one stopped after returning the result,
> the second one kept going indefinitely):
>
> /// on column family that works fine ///
> ///
> 2015-09-30 09:55:35,281 [shell.Shell] DEBUG: Found no scan iterators to set
> 2015-09-30 09:55:35,307 [impl.TabletLocatorImpl] TRACE: tid=12 oid=7
> Locating tablet  table=22 row=  skipRow=false retry=false
> 2015-09-30 09:55:35,308 [impl.TabletLocatorImpl] TRACE: tid=12 oid=8
> Locating tablet  table=!0 row=22;  skipRow=false retry=false
> 2015-09-30 09:55:35,308 [impl.RootTabletLocator] TRACE: tid=12 oid=9
> Looking up root tablet location in zookeeper.
> 2015-09-30 09:55:35,310 [impl.RootTabletLocator] TRACE: tid=12 oid=9  Found
> root tablet at server01:10014|14eb7844d280183 in 0.001 secs
> 2015-09-30 09:55:35,315 [metadata.MetadataLocationObtainer] TRACE: tid=12
> oid=10  Looking up in +r row=!0;22;  extent=+r<< tserver=server01:10014
> 2015-09-30 09:55:35,320 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:55:35,359 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 562
> 2015-09-30 09:55:35,360 [metadata.MetadataLocationObtainer] TRACE: tid=12
> oid=10  Got 4 results  from +r<< in 0.045 secs
> 2015-09-30 09:55:35,362 [impl.TabletLocatorImpl] TRACE: tid=12 oid=8
> Located tablet !0;~< at server01:10014 in 0.054 secs
> 2015-09-30 09:55:35,363 [metadata.MetadataLocationObtainer] TRACE: tid=12
> oid=11  Looking up in !0 row=22;  extent=!0;~< tserver=server01:10014
> 2015-09-30 09:55:35,363 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:55:35,369 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 720
> 2015-09-30 09:55:35,369 [metadata.MetadataLocationObtainer] TRACE: tid=12
> oid=11  Got 8 results  from !0;~< in 0.006 secs
> 2015-09-30 09:55:35,370 [impl.TabletLocatorImpl] TRACE: tid=12 oid=7
> Located tablet 22;sehttp://dbpedia.org/resource/2008%E2%80%9309_M< at
> server01:10014 in 0.063 secs
> 2015-09-30 09:55:35,370 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:55:35,371 [impl.ThriftScanner] TRACE: tid=12 oid=12  Starting
> scan tserver=server01:10014
> tablet=22;sehttp://dbpedia.org/resource/2008%E2%80%9309_M<
> range=(-inf,+inf)
> ssil=[] ssio={}
> 2015-09-30 09:55:35,387 [impl.ThriftScanner] TRACE: tid=12 oid=12  Finished
> scanning tablet in 0.016 secs #results=0
> 2015-09-30 09:55:35,387 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 272
> 2015-09-30 09:55:35,387 [impl.TabletLocatorImpl] TRACE: tid=12 oid=13
> Locating tablet  table=22
> row=sehttp://dbpedia.org/resource/2008%E2%80%9309_M  skipRow=true
> retry=false
> 2015-09-30 09:55:35,387 [impl.TabletLocatorImpl] TRACE: tid=12 oid=13
> Located tablet
> 22;sehttp://
> dbpedia.org/resource/James_Co;sehttp://dbpedia.org/resource/2008%E2%80%9309_M
> at server01:10014 in 0.000 secs
> 2015-09-30 09:55:35,388 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:55:35,388 [impl.ThriftScanner] TRACE: tid=12 oid=14  Starting
> scan tserver=server01:10014
> tablet=22;sehttp://
> dbpedia.org/resource/James_Co;sehttp://dbpedia.org/resource/2008%E2%80%9309_M
> range=(-inf,+inf) ssil=[] ssio={}
> 2015-09-30 09:55:35,392 [impl.ThriftScanner] TRACE: tid=12 oid=14  Finished
> scanning tablet in 0.004 secs #results=0
> 2015-09-30 09:55:35,392 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 278
> 2015-09-30 09:55:35,392 [impl.TabletLocatorImpl] TRACE: tid=12 oid=15
> Locating tablet  table=22 row=sehttp://dbpedia.org/resource/James_Co
> skipRow=true retry=false
> 2015-09-30 09:55:35,393 [impl.TabletLocatorImpl] TRACE: tid=12 oid=15
> Located tablet
> 22;sehttp://
> dbpedia.org/resource/Royal_L;sehttp://dbpedia.org/resource/James_Co
> at server01:10014 in 0.001 secs
> 2015-09-30 09:55:35,393 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:55:35,393 [impl.ThriftScanner] TRACE: tid=12 oid=16  Starting
> scan tserver=server01:10014
> tablet=22;sehttp://
> dbpedia.org/resource/Royal_L;sehttp://dbpedia.org/resource/James_Co
> range=(-inf,+inf) ssil=[] ssio={}
> 2015-09-30 09:55:35,396 [impl.ThriftScanner] TRACE: tid=12 oid=16  Finished
> scanning tablet in 0.003 secs #results=0
> 2015-09-30 09:55:35,396 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 278
> 2015-09-30 09:55:35,396 [impl.TabletLocatorImpl] TRACE: tid=12 oid=17
> Locating tablet  table=22 row=sehttp://dbpedia.org/resource/Royal_L
> skipRow=true retry=false
> 2015-09-30 09:55:35,396 [impl.TabletLocatorImpl] TRACE: tid=12 oid=17
> Located tablet 22<;sehttp://dbpedia.org/resource/Royal_L at server01:10014
> in 0.000 secs
> 2015-09-30 09:55:35,397 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:55:35,397 [impl.ThriftScanner] TRACE: tid=12 oid=18  Starting
> scan tserver=server01:10014 tablet=22<;sehttp://
> dbpedia.org/resource/Royal_L
> range=(-inf,+inf) ssil=[] ssio={}
> 2015-09-30 09:55:35,410 [impl.ThriftScanner] TRACE: tid=12 oid=18
> Completely finished scan in 0.013 secs #results=0
> 2015-09-30 09:55:35,410 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 272
> 2015-09-30 09:55:35,410 [admin.TableOperations] TRACE: tid=12 oid=19
> Fetching list of tables...
> 2015-09-30 09:55:35,412 [admin.TableOperations] TRACE: tid=12 oid=19
> Fetched 23 table names in 0.002 secs
> 2015-09-30 09:55:35,413 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:55:35,413 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 138
> 2015-09-30 09:55:35,413 [admin.TableOperations] TRACE: tid=12 oid=20
> Fetching list of namespaces...
> 2015-09-30 09:55:35,414 [admin.TableOperations] TRACE: tid=12 oid=20
> Fetched 2 namespaces in 0.001 secs
>
>
>
> /// on column family that hung ///
> ///
> 2015-09-30 09:59:33,609 [admin.TableOperations] TRACE: tid=12 oid=25
> Checking if table BVAULT_RDF_STATS exists...
> 2015-09-30 09:59:33,612 [admin.TableOperations] TRACE: tid=12 oid=25
> Checked existance of true in 0.002 secs
> 2015-09-30 09:59:33,613 [impl.ThriftTransportPool] TRACE: Creating new
> connection to connection to server01:10014
> 2015-09-30 09:59:33,628 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 4968
> 2015-09-30 09:59:33,628 [admin.TableOperations] TRACE: tid=12 oid=26
> Checking if table BVAULT_RDF_STATS exists...
> 2015-09-30 09:59:33,630 [admin.TableOperations] TRACE: tid=12 oid=26
> Checked existance of true in 0.002 secs
> 2015-09-30 09:59:33,630 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:59:33,644 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 4968
> 2015-09-30 09:59:33,644 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:59:33,658 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 4968
> 2015-09-30 09:59:33,658 [admin.TableOperations] TRACE: tid=12 oid=27
> Checking if table BVAULT_RDF_STATS exists...
> 2015-09-30 09:59:33,660 [admin.TableOperations] TRACE: tid=12 oid=27
> Checked existance of true in 0.002 secs
> 2015-09-30 09:59:33,661 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:59:33,675 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 4968
> 2015-09-30 09:59:33,675 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:59:33,676 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 130
> 2015-09-30 09:59:33,678 [shell.Shell] DEBUG: Found no scan iterators to set
> 2015-09-30 09:59:33,682 [impl.TabletLocatorImpl] TRACE: tid=12 oid=28
> Locating tablet  table=22 row=  skipRow=false retry=false
> 2015-09-30 09:59:33,682 [impl.TabletLocatorImpl] TRACE: tid=12 oid=28
> Located tablet 22;sehttp://dbpedia.org/resource/2008%E2%80%9309_M< at
> server01:10014 in 0.000 secs
> 2015-09-30 09:59:33,682 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:59:33,682 [impl.ThriftScanner] TRACE: tid=12 oid=29  Starting
> scan tserver=server01:10014
> tablet=22;sehttp://dbpedia.org/resource/2008%E2%80%9309_M<
> range=(-inf,+inf)
> ssil=[] ssio={}
> 2015-09-30 09:59:34,685 [impl.ThriftScanner] TRACE: tid=12 oid=29  Finished
> scan in 1.003 secs #results=0 scanid=4197848475135257903
> 2015-09-30 09:59:34,685 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 242
> 2015-09-30 09:59:34,685 [impl.TabletLocatorImpl] TRACE: tid=12 oid=30
> Locating tablet  table=22 row=  skipRow=false retry=false
> 2015-09-30 09:59:34,686 [impl.TabletLocatorImpl] TRACE: tid=12 oid=30
> Located tablet 22;sehttp://dbpedia.org/resource/2008%E2%80%9309_M< at
> server01:10014 in 0.000 secs
> 2015-09-30 09:59:34,686 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:59:34,686 [impl.ThriftScanner] TRACE: tid=12 oid=31
> Continuing scan tserver=server01:10014 scanid=4197848475135257903
> 2015-09-30 09:59:35,696 [impl.ThriftScanner] TRACE: tid=12 oid=31  Finished
> scan in 1.010 secs #results=0 scanid=4197848475135257903
> 2015-09-30 09:59:35,696 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 122
> 2015-09-30 09:59:35,696 [impl.TabletLocatorImpl] TRACE: tid=12 oid=32
> Locating tablet  table=22 row=  skipRow=false retry=false
> 2015-09-30 09:59:35,697 [impl.TabletLocatorImpl] TRACE: tid=12 oid=32
> Located tablet 22;sehttp://dbpedia.org/resource/2008%E2%80%9309_M< at
> server01:10014 in 0.001 secs
> 2015-09-30 09:59:35,697 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:59:35,697 [impl.ThriftScanner] TRACE: tid=12 oid=33
> Continuing scan tserver=server01:10014 scanid=4197848475135257903
> 2015-09-30 09:59:36,699 [impl.ThriftScanner] TRACE: tid=12 oid=33  Finished
> scan in 1.001 secs #results=0 scanid=4197848475135257903
> 2015-09-30 09:59:36,699 [impl.ThriftTransportPool] TRACE: Returned
> connection server01:10014 (120000) ioCount : 122
> 2015-09-30 09:59:36,699 [impl.TabletLocatorImpl] TRACE: tid=12 oid=34
> Locating tablet  table=22 row=  skipRow=false retry=false
> 2015-09-30 09:59:36,699 [impl.TabletLocatorImpl] TRACE: tid=12 oid=34
> Located tablet 22;sehttp://dbpedia.org/resource/2008%E2%80%9309_M< at
> server01:10014 in 0.000 secs
> 2015-09-30 09:59:36,699 [impl.ThriftTransportPool] TRACE: Using existing
> connection to server01:10014
> 2015-09-30 09:59:36,699 [impl.ThriftScanner] TRACE: tid=12 oid=35
> Continuing scan tserver=server01:10014 scanid=4197848475135257903
> 2015-09-30 09:59:37,701 [impl.ThriftScanner] TRACE: tid=12 oid=35  Finished
> scan in 1.002 secs #results=0 scanid=4197848475135257903
> ...
> ...
> ...
> (keep going)
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/scan-command-hung-tp15286p15296.html
> Sent from the Developers mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message