accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Wall <mjw...@gmail.com>
Subject Re: Node reports assignment failed for tablet
Date Sun, 18 Mar 2018 13:14:23 GMT
Hi Nick,

Looks like you are on the right track with the recovery file, which are
created when WALs have to be replayed, see
http://accumulo.apache.org/1.8/accumulo_user_manual.html#_recovery.  Maybe
try deleting hdfs://master01:9000/user/accumulo/accumulo/recovery/
8bd07d5c-710f-4072-b351-8ce09d771237/finished and give it 10 min or so.  I
could be that one or both of those part files are bad, so your next step
could be to remove the hdfs://master01:9000/user/accumulo/accumulo/recovery/
8bd07d5c-710f-4072-b351-8ce09d771237/ directory entirely.  Again, give
Accumulo 10 min or more.  I don't recall how to track
8bd07d5c-710f-4072-b351-8ce09d771237 back to the WALs, maybe look for the
first occurrence of that in the logs to see if the WALs are still there.
If not, maybe move hdfs://master01:9000/user/accumulo/accumulo/recovery/
8bd07d5c-710f-4072-b351-8ce09d771237/ instead of delete. If you can figure
out what is in the those WALs you will know what to replay.

Good luck

Mike

On Sat, Mar 17, 2018 at 7:14 PM Nick Wise <Nicholas.Wise@sa.catapult.org.uk>
wrote:

> Hello,
>
>
>
> I’m seeing a lot of errors such as the following across my production
> cluster, which has 30 nodes and is running Accumulo 1.7 on Hadoop 2.7.1.
> The system has been running for many months without error.  I would
> appreciate any guidance that can be given particularly if I should, or
> indeed should not, stop the cluster in order to resolve.
>
>
>
> There are over 60 billion elements in the table associated with this
> tablet, and rebuilding from scratch would be very difficult.  I can stand
> some data loss and re-ingest recent data, if that restores service.
>
>
>
> From /usr/local/accumulo/logs/master_master02.log:
>
>
>
> 2018-03-17 22:02:16,527 [master.Master] ERROR: node11:9997 reports
> assignment failed for tablet
> i6;00~1~posDataFeature~dwf~2016;00~1~posDataFeature~dun~201404
>
> 2018-03-17 22:02:16,560 [master.Master] ERROR: node25:9997 reports
> assignment failed for tablet
> i7;^A^@^C51008b0d-1fc7-4742-bc4f-67ec280c7ebc^@80000152d;^A^@^C50fb7a13-1f0f-4943-a94f-26fcd8d15439^@8000014d1e
>
> 2018-03-17 22:02:16,574 [master.Master] ERROR: node30:9997 reports
> assignment failed for tablet
> i7;^A^@^C52d07ff3-677b-44b2-bef5-23d027946401^@80000159fc;^A^@^C52cc4173-6e57-4ef0-81a7-879e77a7d820^@80000152
>
> 2018-03-17 22:02:16,586 [master.Master] ERROR: node06:9997 reports
> assignment failed for tablet
> i6;02~1~posDataFeature~gcn~20170228;02~1~posDataFeature~gbv~201608
>
> 2018-03-17 22:02:16,589 [master.Master] ERROR: node16:9997 reports
> assignment failed for tablet
> i6;00~1~posDataFeature~tbr~2016081;00~1~posDataFeature~t9w~201604
>
> 2018-03-17 22:02:16,616 [master.Master] ERROR: node26:9997 reports
> assignment failed for tablet
> i6;02~1~posDataFeature~dxf~201607;02~1~posDataFeature~dvt~201602
>
> 2018-03-17 22:02:16,694 [master.Master] ERROR: node17:9997 reports
> assignment failed for tablet
> i6;17~0~posDataFeature~k42~201505;17~0~posDataFeature~k3u~20160818
>
> 2018-03-17 22:02:16,778 [master.Master] ERROR: node07:9997 reports
> assignment failed for tablet
> i6;10~0~posDataFeature~u33~2017122;10~0~posDataFeature~u1x~20160207
>
> 2018-03-17 22:02:16,810 [master.Master] ERROR: node05:9997 reports
> assignment failed for tablet
> i6;10~1~posDataFeature~rqe~20140805;10~1~posDataFeature~rqc~20160309
>
> 2018-03-17 22:02:16,825 [master.Master] ERROR: node18:9997 reports
> assignment failed for tablet
> i6;06~1~posDataFeature~6pn~2015082;06~1~posDataFeature~6nx~20170514
>
> 2018-03-17 22:02:16,827 [master.Master] ERROR: node33:9997 reports
> assignment failed for tablet
> i6;09~1~posDataFeature~fu4~20170624;09~1~posDataFeature~fgc~20160928
>
> 2018-03-17 22:02:16,859 [master.Master] ERROR: node31:9997 reports
> assignment failed for tablet
> i7;^A^@^Cc5e97d3d-f3d0-4c80-acfb-b4de1d2aaa0e^@8000014a62;^A^@^Cc5e44ab1-a0a1-4d9c-abce-250a27209c15^@80000156fd
>
> 2018-03-17 22:02:16,920 [master.Master] ERROR: node14:9997 reports
> assignment failed for tablet
> i6;22~0~posDataFeature~xqs~2016;22~0~posDataFeature~xn5~2016012
>
> 2018-03-17 22:02:16,938 [master.Master] ERROR: node22:9997 reports
> assignment failed for tablet
> i6;23~1~posDataFeature~kdm~2015013;23~1~posDataFeature~kdh~20160708
>
> 2018-03-17 22:02:16,981 [master.Master] ERROR: node15:9997 reports
> assignment failed for tablet
> i6;07~1~posDataFeature~w7e~2016092;07~1~posDataFeature~w49
>
> 2018-03-17 22:02:17,194 [master.Master] ERROR: node21:9997 reports
> assignment failed for tablet
> i7;^A^@^C92dba50b-219b-46d0-932a-a12a58ca830f^@800001523;^A^@^C92da7a96-7011-41d0-82ea-1a39ae7b1a6d^@8000015959
>
> 2018-03-17 22:02:17,209 [master.Master] ERROR: node13:9997 reports
> assignment failed for tablet
> i7;^A^@^Cc05;^A^@^Cc04d6645-9a4b-42b8-a8cc-58ff19e0b957^@800001465
>
> 2018-03-17 22:02:17,574 [master.Master] ERROR: node30:9997 reports
> assignment failed for tablet
> i6;17~1~posDataFeature~e7;17~1~posDataFeature~e4r~20160509
>
> 2018-03-17 22:02:17,586 [master.Master] ERROR: node06:9997 reports
> assignment failed for tablet
> i6;01~1~posDataFeature~d3v~20160222;01~1~posDataFeature~d3f~201707072
>
> 2018-03-17 22:02:17,590 [master.Master] ERROR: node16:9997 reports
> assignment failed for tablet
> i7;^A^@^C88fe01d0-243b-487c-ab0d-30b02e8ccf69^@80000152c;^A^@^C88f2eba5-9c1f-4230-8dd2-d1a9f0f659bd^@8000015d
>
> 2018-03-17 22:02:17,617 [master.Master] ERROR: node26:9997 reports
> assignment failed for tablet
> i6;05~1~posDataFeature~d1f;05~1~posDataFeature~d0t~201507
>
> 2018-03-17 22:02:17,694 [master.Master] ERROR: node17:9997 reports
> assignment failed for tablet
> i7;^A^@^C3df85419-e448-4fa3-87b3-ec95edae204b^@80000153;^A^@^C3df47e59-05af-43e9-a650-50685f66ec0e^@80000153d
>
> 2018-03-17 22:02:17,827 [master.Master] ERROR: node33:9997 reports
> assignment failed for tablet
> i7;^A^@^C50f11344-0f80-435e-a6fa-7312619e1535^@80000142b8;^A^@^C50eb8126-a75c-4d89-9767-2a6a71c6bfac^@800001552a
>
> 2018-03-17 22:02:17,859 [master.Master] ERROR: node31:9997 reports
> assignment failed for tablet
> i6;05~0~posDataFeature~pz8;05~0~posDataFeature~my0~20141
>
> 2018-03-17 22:02:17,920 [master.Master] ERROR: node14:9997 reports
> assignment failed for tablet
> i6;14~1~posDataFeature~7ej~201606;14~1~posDataFeature~7ds~201608
>
> 2018-03-17 22:02:17,938 [master.Master] ERROR: node22:9997 reports
> assignment failed for tablet
> i6;18~1~posDataFeature~u3f~2016031;18~1~posDataFeature~u3b~20150816
>
> 2018-03-17 22:02:17,981 [master.Master] ERROR: node15:9997 reports
> assignment failed for tablet
> i6;14~0~posDataFeature~wtq~20151019;14~0~posDataFeature~wt3~20170605
>
> 2018-03-17 22:02:18,194 [master.Master] ERROR: node21:9997 reports
> assignment failed for tablet
> i6;11~0~posDataFeature~dyq~201607;11~0~posDataFeature~drm~201507
>
>
>
> From /usr/local/accumulo/logs/ tserver_node11.log
>
>
>
> 2018-03-17 22:02:16,516 [tserver.TabletServer] INFO : adding tablet
> i6;00~1~posDataFeature~dwf~2016;00~1~posDataFeature~dun~201404 back to the
> assignment pool (retry 129)
>
> 2018-03-17 22:02:16,517 [tserver.TabletServer] INFO : node11:9997: got
> assignment from master:
> i6;00~1~posDataFeature~dwf~2016;00~1~posDataFeature~dun~201404
>
> 2018-03-17 22:02:16,521 [tablet.Tablet] INFO : Starting Write-Ahead Log
> recovery for i6;00~1~posDataFeature~dwf~2016;00~1~posDataFeature~dun~201404
>
> 2018-03-17 22:02:16,521 [tserver.TabletServer] INFO : Looking for
> hdfs://master01:9000/user/accumulo/accumulo/recovery/8bd07d5c-710f-4072-b351-8ce09d771237/finished
>
> 2018-03-17 22:02:16,521 [log.SortedLogRecovery] INFO : Looking at
> mutations from
> hdfs://master01:9000/user/accumulo/accumulo/recovery/8bd07d5c-710f-4072-b351-8ce09d771237
> for i6;00~1~posDataFeature~dwf~2016;00~1~posDataFeature~dun~201404
>
> 2018-03-17 22:02:16,526 [tserver.TabletServer] WARN : exception trying to
> assign tablet
> i6;00~1~posDataFeature~dwf~2016;00~1~posDataFeature~dun~201404
> hdfs://master01:9000/user/accumulo/accumulo/tables/i6/t-011gdek
>
> java.lang.RuntimeException: java.io.IOException: java.io.EOFException
>
>         at
> org.apache.accumulo.tserver.tablet.Tablet.<init>(Tablet.java:639)
>
>         at
> org.apache.accumulo.tserver.tablet.Tablet.<init>(Tablet.java:449)
>
>         at
> org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.java:2157)
>
>         at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>
>         at
> org.apache.accumulo.tserver.ActiveAssignmentRunnable.run(ActiveAssignmentRunnable.java:61)
>
>         at
> org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
>         at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>
>         at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.io.IOException: java.io.EOFException
>
>         at
> org.apache.accumulo.tserver.log.TabletServerLogger.recover(TabletServerLogger.java:456)
>
>         at
> org.apache.accumulo.tserver.TabletServer.recover(TabletServer.java:3012)
>
>         at
> org.apache.accumulo.tserver.tablet.Tablet.<init>(Tablet.java:589)
>
>         ... 9 more
>
> Caused by: java.io.EOFException
>
>         at java.io.DataInputStream.readFully(DataInputStream.java:197)
>
>         at java.io.DataInputStream.readFully(DataInputStream.java:169)
>
>         at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1848)
>
>         at
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1813)
>
>         at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1762)
>
>         at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:443)
>
>         at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:399)
>
>         at
> org.apache.accumulo.tserver.log.MultiReader.<init>(MultiReader.java:113)
>
>         at
> org.apache.accumulo.tserver.log.SortedLogRecovery.recover(SortedLogRecovery.java:105)
>
>         at
> org.apache.accumulo.tserver.log.TabletServerLogger.recover(TabletServerLogger.java:454)
>
>         ... 11 more
>
> 2018-03-17 22:02:16,526 [tserver.TabletServer] WARN : java.io.IOException:
> java.io.EOFException
>
> 2018-03-17 22:02:16,526 [tserver.TabletServer] WARN : failed to open
> tablet i6;00~1~posDataFeature~dwf~2016;00~1~posDataFeature~dun~201404
> reporting failure to master
>
> 2018-03-17 22:02:16,526 [tserver.TabletServer] WARN : rescheduling tablet
> load in 600.00 seconds
>
>
>
>
>
> The same structure error is occurring on many (if not all, all that I have
> so far checked) nodes across the cluster.  From what I have looked at
> 8bd07d5c-710f-4072-b351-8ce09d771237 appears to be a common feature, while
> the other elements vary.
>
>
>
> The file
> hdfs://master01:9000/user/accumulo/accumulo/recovery/8bd07d5c-710f-4072-b351-8ce09d771237/finished
> exists and is zero bytes.  The folder
> hdfs://master01:9000/user/accumulo/accumulo/recovery/8bd07d5c-710f-4072-b351-8ce09d771237
> has two further folders within, part-r-00000 and part-r-00001.  Both have
> files within called data and index.
>
>
>
> The data file in part-r-00000 is 1071KB ends abruptly, thus:
>
>
>
>        +  2xœc```
> ------------------------------
>     f   Z
> ------------------------------
>        "  3
> Khdfs://master01:9000/user/accumulo/accumulo/tables/i7/t-010qx03/F05htsj1.rfxœc```
>
> ------------------------------
>
>
>        #  3xœc```
> ------------------------------
>     f   Z
> ------------------------------
>        $  3
> Khdfs://master01:9000/user/accumulo/accumulo/tables/i7/t-010qx03/F05huonw.rfxœc```
>
> ------------------------------
>
>
>        %  3xœc```
> ------------------------------
>     f   Z
> ------------------------------
>        &  3 Khdfs://master01:9000/u
>
>
>
> The index file in part-r-00000 is zero bytes.
>
>
>
> The data file in part-r-00001 is 31906KB and ends thus (which looks
> reasonable to me):
>
>
>
> ------------------------------
>
>    xœc```lP22ª3¨+È/vI,ItKM,)-J­+54©320´006434.b*MUI5HK1661ÒM³05Õ5ILLÓµ0KNÓM5²HL23H1624e``pc
> [1] .D(ÞÊ ´=I_´û#£ƒ _«
> ------------------------------
> PŸƒçά
> ------------------------------
> @ZÓˆ±÷šÛD ^í)•   Ì
>
> ------------------------------
>
>           xœc```jP22ª3¨+È/vI,ItKM,)-J­+54©320´006434.b*ÎS17HK1661ÒM³05Õ5±0KÑM2N5ÖMJ³02LI34KM5g``pc
> [1] .D(ÞÎÈÀÀ˜¤/ºo#£
> ------------------------------
> ßÔç
> @}
> ž«š iM#ÆÞkn“ˆs[J±J*Â: s uÉ&ºII&)ºÉf‰ ©‰Ææ–f¦fp· ¡xÄm[1]å
@·ñžÛÜ v[—
> ------------------------------
> Üm“70Rv
>
>
>
> The index file in part-t-00001 is 5KB and equally looks reasonable.
>
>
>
> Any help or direction that you might be able to give would be most
> gratefully received.
>
>
>
> Best regards,
>
>
>
> Nick
>
>
>
>
> This email (and any attachments) may contain confidential information and
> is intended solely for the recipient(s) to whom the email is addressed. If
> you received this email in error, please inform us immediately and delete
> the email and all attachments without further using, copying or disclosing
> the information. This email and any attachments are believed to be, but
> cannot be guaranteed to be, secure or virus-free. Satellite Applications
> Catapult Limited is registered in England & Wales. Company Number: 7964746.
> Registered office: Electron Building, Fermi Avenue, Harwell Oxford, Didcot,
> Oxfordshire OX11 0QR.
>

Mime
View raw message