nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Payne <marka...@hotmail.com>
Subject Re: Reg: Get files from ftp
Date Wed, 11 May 2016 13:07:28 GMT
Sourav,

If your NiFi instance is clustered, it will store the information in ZooKeeper. If not clustered,
it
will store the state in a local file. This is done because in a cluster, you typically want
to run your
List*** Processors on Primary Node only, and this allows another node to pick up where the
previous
one left off if the Primary Node changes. Of course, storing all of the files that have been
listed can become
very verbose so it stores only a small amount of data -- the timestamp of the latest file
discovered
and the timestamp of the latest file process/listed. It can then use this information to determine
if files
are new or modified without storing much info.

Thanks
-Mark

> On May 11, 2016, at 12:39 AM, Sourav Gulati <sourav.gulati@impetus.co.in> wrote:
> 
> Thanks Matthew,
> A quick question: Where does it store the state of files already listed?
> 
> 
> Regards,
> Sourav Gulati
> 
> -----Original Message-----
> From: Matthew Clarke [mailto:matt.clarke.138@gmail.com]
> Sent: Wednesday, May 11, 2016 3:37 AM
> To: dev@nifi.apache.org
> Subject: Re: Reg: Get files from ftp
> 
> The list type processors are designed to use NiFi state management to keep from listing
the same files twice. The fetch type processors with retrieve files based on the FlowFiles
it is fed. Typically those FlowFiles it works from come from the corresponding list processor.
> On May 10, 2016 8:56 AM, "Mark Payne" <markap14@hotmail.com> wrote:
> 
>> Sourav,
>> 
>> Sure. Within the nifi-standard-processors bundle are a few classes
>> that would be important here.
>> First is the AbstractListProcessor. You'll want to use this as your
>> base class for ListFTP. Also, FetchFileTransfer will be the class that
>> you'll extend for the FetchFTP processor.
>> 
>> The ListSFTP and FetchSFTP are great examples to look at as examples.
>> 
>> Additionally, the GetFTP and GetSFTP are good examples to look at as
>> to how the FTP & SFTP implementations differ. They basically differ in
>> the Property Descriptors provided and the FileTransfer object that is
>> used.
>> 
>> If you have any questions, please feel free to reach out to this
>> mailing list. Very happy to help however we can!
>> 
>> Thanks
>> -Mark
>> 
>> 
>>> On May 10, 2016, at 1:30 AM, Sourav Gulati
>>> <sourav.gulati@impetus.co.in>
>> wrote:
>>> 
>>> Sure Mark. I am interested to work on it. Please provide some
>>> pointers
>> regarding that.
>>> 
>>> Also, I will check if Sftp can be used. So ListSFTP / FetchSFTP
>>> won't
>> pick files more than once?
>>> 
>>> Regards,
>>> Sourav Gulati
>>> 
>>> -----Original Message-----
>>> From: Mark Payne [mailto:markap14@hotmail.com]
>>> Sent: Monday, May 09, 2016 5:34 PM
>>> To: dev@nifi.apache.org
>>> Subject: Re: Reg: Get files from ftp
>>> 
>>> Sourav,
>>> 
>>> We have begun transitioning from many of the Get*** Processors to
>> List*** and Fetch*** Processors.
>>> There is a ListSFTP / FetchSFTP processor set but not currently a
>> List/Fetch FTP. Is SFTP a possibility for you? Would you be interested
>> in working on a List/Fetch FTP Processor set?
>>> 
>>> Thanks
>>> -Mark
>>> 
>>>> On May 9, 2016, at 5:48 AM, Sourav Gulati
>>>> <sourav.gulati@impetus.co.in>
>> wrote:
>>>> 
>>>> Hi Team,
>>>> 
>>>> I need a suggestion.
>>>> 
>>>> I want to get files from ftp server for which GetFtp processor is
>> available. However, as I cannot delete files from source, I need to
>> put a check that this processor does not pick a file more than once.
>> What is the best way to do that?
>>>> 
>>>> Regards,
>>>> Sourav Gulati
>>>> 
>>>> 
>>>> ________________________________
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited
>> when received in error. Impetus does not represent, warrant and/or
>> guarantee, that the integrity of this communication has been
>> maintained nor that the communication is free of errors, virus, interception or interference.
>>> 
>>> 
>>> ________________________________
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> NOTE: This message may contain information that is confidential,
>> proprietary, privileged or otherwise protected by law. The message is
>> intended solely for the named addressee. If received in error, please
>> destroy and notify the sender. Any use of this email is prohibited
>> when received in error. Impetus does not represent, warrant and/or
>> guarantee, that the integrity of this communication has been
>> maintained nor that the communication is free of errors, virus, interception or interference.
>> 
>> 
> 
> ________________________________
> 
> 
> 
> 
> 
> 
> NOTE: This message may contain information that is confidential, proprietary, privileged
or otherwise protected by law. The message is intended solely for the named addressee. If
received in error, please destroy and notify the sender. Any use of this email is prohibited
when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity
of this communication has been maintained nor that the communication is free of errors, virus,
interception or interference.


Mime
View raw message