asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taewoo Kim <wangs...@gmail.com>
Subject Re: An exception: Input/output error?
Date Mon, 30 Jan 2017 19:10:32 GMT
For the future references, I am sending this message.

As Yingyi mentioned, "Input/Output Error" happened because of a disk error.
I checked "/var/log/messages" as Yingyi suggested. For this time, it didn't
say anything special. I tried the "kmesg" command that were suggested by
Ian and found that there was a disk error. The message is as follows:

INFO: task flush-8:16:14342 blocked for more than 120 seconds.
      Not tainted 2.6.32-642.1.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-8:16    D 0000000000000000     0 14342      2 0x00000080
 ffff8802093e33e0 0000000000000046 0000000000000000 ffff88003de4d8e0
 0000000000000000 ffff88015e063520 0044e48cad574dea ffff88005e399840
 ffff88015e063520 0000000583cf9f6a ffff88015e063ad8 ffff8802093e3fd8
Call Trace:
 [<ffffffff81547db3>] io_schedule+0x73/0xc0
... ... ... ... (omitted) ... ... ... ...
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
ata2: lost interrupt (Status 0x51)
ata2.00: limiting speed to UDMA/33:PIO4
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x40000000 action 0x6 frozen
ata2: SError: { }
ata2.00: failed command: READ DMA EXT
ata2.00: cmd 25/00:08:30:ed:af/00:00:25:00:00/e0 tag 0 dma 4096 in
         res 40/00:93:35:ed:af/00:00:25:00:00/05 Emask 0x4 (timeout)
ata2.00: status: { DRDY }
ata2: hard resetting link
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata2.00: configured for UDMA/33
sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 1:0:0:0: [sdb]  Sense Key : Aborted Command [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
        72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
        25 af ed 35
sd 1:0:0:0: [sdb]  Add. Sense: No additional sense information
sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 25 af ed 30 00 00 08 00
*end_request: I/O error, dev sdb, sector 632286512*


Best,
Taewoo

On Sun, Jan 29, 2017 at 6:14 PM, Taewoo Kim <wangsaeu@gmail.com> wrote:

> @Yingyi: Thanks. I will look into that error.
>
> Best,
> Taewoo
>
> On Sun, Jan 29, 2017 at 5:50 PM, Yingyi Bu <buyingyi@gmail.com> wrote:
>
>> Hi Taewoo,
>>
>>     This error message usually indicates that there are some disk
>> failures.
>>     Please check /var/log/messages on the machine to see if that's the
>> case.
>>
>> Best,
>> Yingyi
>>
>> On Sun, Jan 29, 2017 at 4:00 PM, Taewoo Kim <wangsaeu@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > I'm trying to load json files into a dataset. I have nine nodes so that
>> I
>> > divide a 250GB file into nine files. After one hour, suddenly, I see the
>> > following exception on one node. Has anyone experienced this symptom?
>> >
>> > org.apache.hyracks.api.exceptions.HyracksDataException:
>> > org.apache.hyracks.api.exceptions.HyracksDataException: Input/output
>> error
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.runInParallel(SuperActivityOperatorNodePushable.java:228)
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.initialize(SuperActivityOperatorNodePushable.java:84)
>> > at org.apache.hyracks.control.nc.Task.run(Task.java:273)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(
>> > ThreadPoolExecutor.java:1142)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> > ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Caused by: java.util.concurrent.ExecutionException:
>> > org.apache.hyracks.api.exceptions.HyracksDataException: Input/output
>> error
>> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.runInParallel(SuperActivityOperatorNodePushable.java:222)
>> > ... 5 more
>> > Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
>> > Input/output error
>> > at
>> > org.apache.asterix.external.operators.ExternalDataScanOperatorDescri
>> > ptor$1.initialize(ExternalDataScanOperatorDescriptor.java:65)
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.lambda$initialize$0(SuperActivityOperatorNodePushable.java:86)
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.lambda$runInParallel$2(SuperActivityOperatorNodePushable.java:216)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > ... 3 more
>> > Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
>> > Input/output error
>> > at
>> > org.apache.asterix.external.dataflow.RecordDataFlowController.start(
>> > RecordDataFlowController.java:63)
>> > at
>> > org.apache.asterix.external.dataset.adapter.GenericAdapter.start(
>> > GenericAdapter.java:37)
>> > at
>> > org.apache.asterix.external.operators.ExternalDataScanOperatorDescri
>> > ptor$1.initialize(ExternalDataScanOperatorDescriptor.java:62)
>> > ... 6 more
>> > Caused by: java.io.IOException: Input/output error
>> > at java.io.FileInputStream.readBytes(Native Method)
>> > at java.io.FileInputStream.read(FileInputStream.java:255)
>> > at
>> > org.apache.asterix.external.input.stream.LocalFSInputStream.read(
>> > LocalFSInputStream.java:126)
>> > at
>> > org.apache.asterix.external.input.stream.AsterixInputStreamReader.read(
>> > AsterixInputStreamReader.java:95)
>> > at
>> > org.apache.asterix.external.input.stream.AsterixInputStreamReader.read(
>> > AsterixInputStreamReader.java:67)
>> > at
>> > org.apache.asterix.external.input.record.reader.stream.
>> > SemiStructuredRecordReader.hasNext(SemiStructuredRecordReader.java:83)
>> > at
>> > org.apache.asterix.external.dataflow.RecordDataFlowController.start(
>> > RecordDataFlowController.java:52)
>> > ... 8 more
>> > java.lang.InterruptedException
>> > at
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.
>> > acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
>> > at java.util.concurrent.Semaphore.acquire(Semaphore.java:467)
>> > at org.apache.hyracks.control.nc.Task.run(Task.java:310)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(
>> > ThreadPoolExecutor.java:1142)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> > ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> > java.lang.InterruptedException
>> > at
>> > java.util.concurrent.locks.AbstractQueuedSynchronizer.
>> > acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302)
>> > at java.util.concurrent.Semaphore.acquire(Semaphore.java:467)
>> > at org.apache.hyracks.control.nc.Task.run(Task.java:310)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(
>> > ThreadPoolExecutor.java:1142)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> > ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Jan 29, 2017 3:44:26 PM org.apache.hyracks.control.nc.Joblet close
>> >
>> >
>> > Best,
>> > Taewoo
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message