kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Long (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KYLIN-3431) Avoid FileInputStream/FileOutputStream
Date Wed, 04 Jul 2018 09:00:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532442#comment-16532442
] 

Chao Long edited comment on KYLIN-3431 at 7/4/18 8:59 AM:
----------------------------------------------------------

I have replaced FileInputStream/FileOutputStream with FIles.newInputStream/newOutputStream,
but when I run CI, I get error message like this:

2018-07-04 06:10:41,485 INFO [main] threadpool.DefaultScheduler:192 : Shutting down DefaultScheduler
....
 2018-07-04 06:10:41,487 ERROR [Scheduler 107702879 Job b036fdd2-73e2-4463-b1a6-841018297f1c-21]
dao.ExecutableDao:219 : error update job output id:b036fdd2-73e2-4463-b1a6-841018297f1c-00
 java.nio.channels.ClosedByInterruptException
 at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
 at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
 at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
 at java.nio.channels.Channels.writeFully(Channels.java:101)
 at java.nio.channels.Channels.access$000(Channels.java:61)
 at java.nio.channels.Channels$1.write(Channels.java:174)
 at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793)
 at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
 at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
 at org.apache.kylin.common.persistence.FileResourceStore.putResourceImpl(FileResourceStore.java:161)
 at org.apache.kylin.common.persistence.FileResourceStore.checkAndPutResourceImpl(FileResourceStore.java:177)
 at org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
 at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
 at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
 at org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
 at org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
 at org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
 at org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:490)
 at org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:403)
 at org.apache.kylin.job.execution.AbstractExecutable.setEndTime(AbstractExecutable.java:419)
 at org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinished(AbstractExecutable.java:121)
 at org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinishedWithRetry(AbstractExecutable.java:98)
 at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:175)
 at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
 at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
 at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

 

code block: FileResourceStore.java->putResourceImpl()

@Override
 protected void putResourceImpl(String resPath, InputStream content, long ts) throws IOException
{
       synchronized (FileResourceStore.class) {
       File f = file(resPath);
       f.getParentFile().mkdirs();
       try (final OutputStream out = Files.newOutputStream(f.toPath())){ 

         IOUtils.copy(content, out);

         f.setLastModified(ts);    

     }

   }
 }

 

After checking the code, I find method DefaultSchedulerTest.java->testSchedulerStop() cause
this problem.  I also find "FIle.newInputStream" use "channal" to do his work, and If a
class implements the "InterruptibleChannel" interface, then when the thread on the channel
is interrupted, the channel will be closed and the thread will throws "ClosedByInterruptException"
exception.

So can I catch this exception["ClosedByInterruptException"] in method["putResourceImpl"] or
any other suggestions provided?


was (Author: wayne0101):
I have replaced FileInputStream/FileOutputStream with FIles.newInputStream/newOutputStream,
but when I run CI, I get error message like this:

2018-07-04 06:10:41,485 INFO [main] threadpool.DefaultScheduler:192 : Shutting down DefaultScheduler
....
2018-07-04 06:10:41,487 ERROR [Scheduler 107702879 Job b036fdd2-73e2-4463-b1a6-841018297f1c-21]
dao.ExecutableDao:219 : error update job output id:b036fdd2-73e2-4463-b1a6-841018297f1c-00
java.nio.channels.ClosedByInterruptException
 at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
 at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
 at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
 at java.nio.channels.Channels.writeFully(Channels.java:101)
 at java.nio.channels.Channels.access$000(Channels.java:61)
 at java.nio.channels.Channels$1.write(Channels.java:174)
 at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793)
 at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
 at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
 at org.apache.kylin.common.persistence.FileResourceStore.putResourceImpl(FileResourceStore.java:161)
 at org.apache.kylin.common.persistence.FileResourceStore.checkAndPutResourceImpl(FileResourceStore.java:177)
 at org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
 at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
 at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
 at org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:88)
 at org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:216)
 at org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
 at org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:490)
 at org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:403)
 at org.apache.kylin.job.execution.AbstractExecutable.setEndTime(AbstractExecutable.java:419)
 at org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinished(AbstractExecutable.java:121)
 at org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinishedWithRetry(AbstractExecutable.java:98)
 at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:175)
 at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
 at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
 at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

 

code block: FileResourceStore.java->putResourceImpl()

@Override
protected void putResourceImpl(String resPath, InputStream content, long ts) throws IOException
{
      synchronized (FileResourceStore.class) {
      File f = file(resPath);
      f.getParentFile().mkdirs();
      try (final OutputStream out = Files.newOutputStream(f.toPath())) {
            IOUtils.copy(content, out);
            f.setLastModified(ts);
      }
   }
}

 

After checking the code, I find method DefaultSchedulerTest.java->testSchedulerStop() cause
this problem.  I also find "FIle.newInputStream" use "channal" to do his work, and If a
class implements the "InterruptibleChannel" interface, then when the thread on the channel
is interrupted, the channel will be closed and the thread will throws "ClosedByInterruptException"
exception.

So can I catch this exception["ClosedByInterruptException"] in method["putResourceImpl"] or
any other suggestions provided?

> Avoid FileInputStream/FileOutputStream
> --------------------------------------
>
>                 Key: KYLIN-3431
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3431
>             Project: Kylin
>          Issue Type: Improvement
>            Reporter: Ted Yu
>            Priority: Major
>             Fix For: v2.5.0
>
>         Attachments: image-2018-07-04-16-41-54-234.png
>
>
> They rely on finalizers (before Java 11), which create unnecessary GC load. The alternatives,
{{Files.newInputStream}}, are as easy to use and don't have this issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message