edgent-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dale LaBossiere (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (EDGENT-306) With edgent Tester, failed to read file content from the newly appeared files from a directory
Date Mon, 07 Nov 2016 20:01:58 GMT

    [ https://issues.apache.org/jira/browse/EDGENT-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15645258#comment-15645258

Dale LaBossiere commented on EDGENT-306:

I suspect there are still some misunderstandings as there is no absolute requirement that
directoryWatcher()/textFileReader() can only be used for files created with textFileWriter().

At this time textFileReader() only supports files whose content is static.  i.e., textFileReader()
only makes a single pass reading a file's content and converting it to tuples.  As noted in
its javadoc "A file is not subsequently monitored for additional lines".   So, perhaps your
code ended up providing textFileReader() with a pathname of a file whose contents was not
yet "complete" (as it is in the "pre-existing" case)?  

I thought there might already be a jira for a "hot" textFileReader() capability but didn't
notice one - i.e. a textFileReader() that can function like "tail -f ...", watching and reporting
a file's content that's being incrementally added.  In the absence of hot-textFileReader()
once could use Connectors.CommandStreams (e.g., CommandStreams.generate(...)) to create a
long-running "tail -f" command and convert its output to tuples.

directoryWatcher() generates a file name in its output stream only once -- when the file is
first noticed.  This is subject to platform's JRE java.nio.file.WatchService behavior as noted
in the directoryWatcher() javadoc.  As noted in the javadoc, directoryWatcher() ignores "hidden"
files -- File.isHidden()==true.

textFileWriter() / FileWriterPolicy creates files in a manner that works with the above. 
i.e., by default, a file that's under construction, an "active file", has a "hidden file"
name.  Only once the file's content is "complete" is the file "cycled" and renamed to a non-hidden
file that directoryWatcher() will see/report and hence when textFileReader() receives the
pathname the file is complete and will be fully processed.

I hope that helps clarify things.
In light of that information should/can this jira be closed as a "usage mistake" or do you
believe there is a bug present?

> With edgent Tester, failed to read file content from the newly appeared files from a
> -----------------------------------------------------------------------------------------------
>                 Key: EDGENT-306
>                 URL: https://issues.apache.org/jira/browse/EDGENT-306
>             Project: Edgent
>          Issue Type: Bug
>          Components: Test
>    Affects Versions: 0.4.0
>            Reporter: Rui Shen
>         Attachments: CopyOfFileStreamsTest.java, FileReaderApp.java, M2.png
> I got a trouble when using edgent Tester. The scenario that I am trying to test is:
> 1) There is a empty directory (e.g "testdata")
> 2) Use FileStreams.directoryWatcher to watch the directory "testdata" and use FileStreams.textFileReader
to read content of newly appeared files in the directory.
> I am trying to use edgent Tester to write a unit test of this scenario (I checked org.apache.edgent.test.connectors.file.FileStreamsTest
and didn't find such scenario). But I found my unit test always failed at reading the content
of newly appeared files. But if the files are pre-existing, it works. 
> I attached two files. To reproduce this issue, you can run testDirectoryWatcherOrder
in CopyOfFileStreamsTest.java. And you will see the following error. FileReaderApp.java is
a standard alone version of the above test scenario - it works well (once you start it, generate
some files with content in the directory, you can see the file content are printed in console).

> So could you please let me know if this is a defect of edgent Tester or my usage mistake?

This message was sent by Atlassian JIRA

View raw message