nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: NiFi copying windows .part files
Date Wed, 06 Dec 2017 07:47:54 GMT
Imagine a filename construct where you wanted to pick up any file that
begins with the phrase 'start' but does NOT end in the phrase 'part'.

The name is of a form 'begin.middle.end'.

This filename start.middle.ok would get picked up.

This filename start.middle.part would not.

The pattern for that example would be
  start\..+\.(?!part)

The key part of that is the negative lookahead for ensuring it does
not end in part.

Thanks

On Wed, Dec 6, 2017 at 2:29 AM, Ravi Papisetti (rpapiset)
<rpapiset@cisco.com> wrote:
> Yeah..that is good idea, but we are already using this option to copy file with certain
prefix. Not sure how I can use this field to meet both exclusion and inclusion criterion.
>
> Any thoughts.
>
> Thanks,
> Ravi Papisetti
>
> On 06/12/17, 1:26 AM, "Joe Witt" <joe.witt@gmail.com> wrote:
>
>     Ravi
>
>     Please use the 'File Filter' property of ListFile to control ignoring
>     filenames until they no longer end in 'part'.
>
>     Thanks
>
>     On Wed, Dec 6, 2017 at 2:14 AM, Ravi Papisetti (rpapiset)
>     <rpapiset@cisco.com> wrote:
>     > Hi,
>     >
>     >
>     >
>     > We are using Apache NiFi 1.3.0
>     >
>     >
>     >
>     > We have a process flow to copy files from NFS to HDFS (with processors
>     > ListFile, FetchFile and PutHDFS)
>     >
>     >
>     >
>     > In the NiFi process flow, ListFile is configured to listen to a directory on
>     > NFS. When a file (ex: x.csv) is being copied from a windows machine to NFS
>     > (while transfer is in the middle), a part file(x.csv.part) is created at NFS
>     > until transfer is complete.
>     >
>     >
>     >
>     > ListFile has picked up this x.csv.part file and fetchFile picked up this to
>     > transfer to HDFS, didn’t update the file name back to x.csv in HDFS when
>     > transfer is complete.
>     >
>     >
>     >
>     > But, in case a file from linux file system, while file copy to NFS is in
>     > progress it created (.x.csv) and when transfer is complete, at both NFS and
>     > HDFS, filename is updated to x.csv (from .x.csv).
>     >
>     >
>     >
>     > Any thought how we can configure ListFile not to pickup these part files or
>     > any configurations in NiFi that fixes file names for these windows part
>     > files?
>     >
>     >
>     >
>     > Appreciate your help.
>     >
>     >
>     >
>     > Thanks,
>     >
>     > Ravi Papisetti
>
>

Mime
View raw message