accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Wall <mjw...@gmail.com>
Subject Re: scan of small table stopped working (hangs)
Date Tue, 23 May 2017 12:42:05 GMT
Jim

I suspect your installation used files in /tmp that have been cleaned up.
Look at your hdfs-site.xml and ensure the dfs.name.dir and dfs.data.dir are
not in /tmp.  Then look at your zoo.cfg directory and ensure the dataDir is
defined and not /tmp.

Mike

On Tue, May 23, 2017 at 2:02 AM o haya <ohaya@yahoo.com> wrote:

> Hi,
>
> I noticed that in the monitor_nodejs.whatever.com.debug.log file I am
> seeing these messages:
>
> 2017-05-23 01:56:51,035 [impl.ThriftScanner] DEBUG:  Failed to locate
> tablet for table : !0 row : ~err_
> 2017-05-23 01:56:57,173 [monitor.Monitor] INFO :  Failed to obtain problem
> reports
> java.lang.RuntimeException:
> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
>         at
> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:160)
>         at
> org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:252)
>         at
> org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:310)
>         at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:374)
>         at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:519)
>         at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
>         at
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:243)
>         at
> org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:79)
>         at
> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:150)
>         ... 6 more
> 2017-05-23 01:56:57,512 [impl.ThriftScanner] DEBUG:  Failed to locate
> tablet for table : !0 row : ~err_
> 2017-05-23 01:57:03,604 [monitor.Monitor] INFO :  Failed to obtain problem
> reports
> java.lang.RuntimeException:
> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
>         at
> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:160)
>         at
> org.apache.accumulo.server.problems.ProblemReports$3.hasNext(ProblemReports.java:252)
>         at
> org.apache.accumulo.server.problems.ProblemReports.summarize(ProblemReports.java:310)
>         at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:374)
>         at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:519)
>         at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.accumulo.core.client.impl.ThriftScanner$ScanTimedOutException
>         at
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:243)
>         at
> org.apache.accumulo.core.client.impl.ScannerIterator$Reader.run(ScannerIterator.java:79)
>         at
> org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:150)
>         ... 6 more
> 2017-05-23 01:57:03,944 [impl.ThriftScanner] DEBUG:  Failed to locate
> tablet for table : !0 row : ~err_
>
>
>
>
> --------------------------------------------
> On Tue, 5/23/17, o haya <ohaya@yahoo.com> wrote:
>
>  Subject: scan of small table stopped working (hangs)
>  To: user@accumulo.apache.org
>  Date: Tuesday, May 23, 2017, 1:56 AM
>
>  Hi,
>
>  I have a small Accumulo test instance,
>  that I installed following this page:
>
>
> https://www.digitalocean.com/community/tutorials/how-to-install-the-big-data-friendly-apache-accumulo-nosql-database-on-ubuntu-14-04
>
>  I've been doing testing and was able to
>  successfully run the SimpleIngestClient from the minicluster
>  project and create a table, testTable.
>
>  After that, I could use the Accumulo
>  shell and scan the table, after logging in as a user, "demo"
>  and running "scan -t testTable".
>
>  However, tonight, after I brought up
>  the machine and started Hadoop
>  (/apps/hadoop/sbin/start-dfs.sh), then zookeeper
>  (/apps/zookeeper/bin/zkServer.sh start) and finally,
>  Accumulo (/apps/accumulo/bin/start-all.sh).
>
>  And then I started the Accumulo shell
>  and logged in as "demo" and did "scan -t testTable" and it
>  just hangs, with no output.
>
>  I found another thread that suggested
>  enabling "debug on" and here is the output:
>
>  root@accumulo1>
>  debug on
>  2017-05-23 01:44:37,524
>  [admin.TableOperations] TRACE: tid=13 Fetching list of
>  tables...
>  2017-05-23 01:44:37,526
>  [admin.TableOperations] TRACE: tid=13 Fetched 5 table names
>  in 0.001 secs
>  2017-05-23 01:44:37,526
>  [rpc.ThriftUtil] TRACE: Opening normal transport
>  2017-05-23 01:44:37,532
>  [impl.ThriftTransportPool] TRACE: Creating new connection to
>  connection to localhost:9997
>  2017-05-23 01:44:37,534
>  [impl.ThriftTransportPool] TRACE: Returned connection
>  localhost:9997 (120000) ioCount: 170
>  2017-05-23 01:44:37,534
>  [admin.TableOperations] TRACE: tid=13 Fetching list of
>  namespaces...
>  2017-05-23 01:44:37,534
>  [admin.TableOperations] TRACE: tid=13 Fetched 2 namespaces
>  in 0.000 secs
>  root@accumulo1>
>  scan -t testTable
>  2017-05-23 01:44:51,547
>  [admin.TableOperations] TRACE: tid=13 Checking if table
>  testTable exists...
>  2017-05-23 01:44:51,547
>  [admin.TableOperations] TRACE: tid=13 Checked existance of
>  true in 0.000 secs
>  2017-05-23 01:44:51,548
>  [rpc.ThriftUtil] TRACE: Opening normal transport
>  2017-05-23 01:44:51,549
>  [impl.ThriftTransportPool] TRACE: Creating new connection to
>  connection to localhost:9997
>  2017-05-23 01:44:51,565
>  [impl.ThriftTransportPool] TRACE: Returned connection
>  localhost:9997 (120000) ioCount: 6808
>  2017-05-23 01:44:51,565
>  [admin.TableOperations] TRACE: tid=13 Checking if table
>  testTable exists...
>  2017-05-23 01:44:51,566
>  [admin.TableOperations] TRACE: tid=13 Checked existance of
>  true in 0.000 secs
>  2017-05-23 01:44:51,566
>  [impl.ThriftTransportPool] TRACE: Using existing connection
>  to localhost:9997
>  2017-05-23 01:44:51,582
>  [impl.ThriftTransportPool] TRACE: Returned connection
>  localhost:9997 (120000) ioCount: 6808
>  2017-05-23 01:44:51,583
>  [impl.ThriftTransportPool] TRACE: Using existing connection
>  to localhost:9997
>  2017-05-23 01:44:51,594
>  [impl.ThriftTransportPool] TRACE: Returned connection
>  localhost:9997 (120000) ioCount: 6808
>  2017-05-23 01:44:51,594
>  [admin.TableOperations] TRACE: tid=13 Checking if table
>  testTable exists...
>  2017-05-23 01:44:51,594
>  [admin.TableOperations] TRACE: tid=13 Checked existance of
>  true in 0.000 secs
>  2017-05-23 01:44:51,594
>  [impl.ThriftTransportPool] TRACE: Using existing connection
>  to localhost:9997
>  2017-05-23 01:44:51,608
>  [impl.ThriftTransportPool] TRACE: Returned connection
>  localhost:9997 (120000) ioCount: 6808
>  2017-05-23 01:44:51,609
>  [impl.ThriftTransportPool] TRACE: Using existing connection
>  to localhost:9997
>  2017-05-23 01:44:51,614
>  [impl.ThriftTransportPool] TRACE: Returned connection
>  localhost:9997 (120000) ioCount: 130
>  2017-05-23 01:44:51,620 [shell.Shell]
>  DEBUG: Found no scan iterators to set
>  2017-05-23 01:44:51,644
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:44:51,644
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:44:51,644
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:44:51,658
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.014 secs
>  2017-05-23 01:44:51,659
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.014 secs
>  2017-05-23 01:44:51,659
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.015 secs
>  2017-05-23 01:44:51,659
>  [impl.ThriftScanner] DEBUG: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:44:51,759
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:44:51,759
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:44:51,759
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:44:51,759
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.000 secs
>  2017-05-23 01:44:51,759
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:44:51,760
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:44:51,760
>  [impl.ThriftScanner] TRACE: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:44:51,952
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:44:51,952
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:44:51,952
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:44:51,952
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.000 secs
>  2017-05-23 01:44:51,952
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:44:51,953
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:44:51,953
>  [impl.ThriftScanner] TRACE: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:44:52,302
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:44:52,302
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:44:52,302
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:44:52,303
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.000 secs
>  2017-05-23 01:44:52,303
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:44:52,303
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.001 secs
>  2017-05-23 01:44:52,303
>  [impl.ThriftScanner] TRACE: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:44:52,960
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:44:52,960
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:44:52,960
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:44:52,960
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.000 secs
>  2017-05-23 01:44:52,960
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:44:52,960
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:44:52,960
>  [impl.ThriftScanner] TRACE: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:44:54,282
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:44:54,282
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:44:54,282
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:44:54,282
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.000 secs
>  2017-05-23 01:44:54,282
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:44:54,282
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:44:54,282
>  [impl.ThriftScanner] TRACE: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:44:56,898
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:44:56,898
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:44:56,898
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:44:56,898
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.000 secs
>  2017-05-23 01:44:56,898
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:44:56,898
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:44:56,898
>  [impl.ThriftScanner] TRACE: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:45:02,204
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:45:02,204
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:45:02,204
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:45:02,204
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.000 secs
>  2017-05-23 01:45:02,204
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:45:02,205
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:45:02,205
>  [impl.ThriftScanner] TRACE: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:45:07,560
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:45:07,560
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:45:07,560
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:45:07,560
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.000 secs
>  2017-05-23 01:45:07,561
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:45:07,561
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:45:07,561
>  [impl.ThriftScanner] TRACE: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:45:13,002
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:45:13,002
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:45:13,002
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:45:13,002
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.000 secs
>  2017-05-23 01:45:13,002
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:45:13,002
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:45:13,002
>  [impl.ThriftScanner] TRACE: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:45:17,530
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:45:17,530
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:45:17,530
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:45:17,530
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.000 secs
>  2017-05-23 01:45:17,530
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:45:17,530
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:45:17,530
>  [impl.ThriftScanner] TRACE: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:45:22,802
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:45:22,803
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:45:22,803
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:45:22,803
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.000 secs
>  2017-05-23 01:45:22,803
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:45:22,803
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:45:22,803
>  [impl.ThriftScanner] TRACE: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:45:28,247
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=7 row= skipRow=false retry=false
>  2017-05-23 01:45:28,247
>  [impl.TabletLocatorImpl] TRACE: tid=13 Locating tablet
>  table=!0 row=7; skipRow=false retry=false
>  2017-05-23 01:45:28,247
>  [impl.RootTabletLocator] TRACE: tid=13 Looking up root
>  tablet location in zookeeper.
>  2017-05-23 01:45:28,248
>  [impl.RootTabletLocator] TRACE: tid=13 Found root tablet at
>  null in 0.000 secs
>  2017-05-23 01:45:28,248
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:45:28,248
>  [impl.TabletLocatorImpl] TRACE: tid=13 Located tablet null
>  at null in 0.000 secs
>  2017-05-23 01:45:28,248
>  [impl.ThriftScanner] TRACE: Failed to locate tablet for
>  table : 7 row :
>  2017-05-23 01:45:30,781 [ipc.Client]
>  DEBUG: stopping client from cache: org.apache.hadoop.ipc.Client@a95842c
>  2017-05-23 01:45:30,782 [ipc.Client]
>  DEBUG: removing client from cache: org.apache.hadoop.ipc.Client@a95842c
>  2017-05-23 01:45:30,782 [ipc.Client]
>  DEBUG: stopping actual client because no more references
>  remain: org.apache.hadoop.ipc.Client@a95842c
>  2017-05-23 01:45:30,782 [ipc.Client]
>  DEBUG: Stopping client
>
>
>  Can anyone tell from the above why the
>  scan command is no longer working?
>
>  Thanks,
>  Jim
>
>

Mime
View raw message