nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sourav Gulati <sourav.gul...@impetus.co.in>
Subject RE: Reg: Tailing a remote file
Date Thu, 21 Apr 2016 12:42:33 GMT
Simon,

I see an issue with tail processor .

I have to give the exact name of the file to tail. I can give something like /home/user/abc*.log

Is there anyway I can do this?

Regards,
Sourav Gulati

-----Original Message-----
From: Simon Ball [mailto:sball@hortonworks.com]
Sent: Thursday, April 21, 2016 3:40 PM
To: dev@nifi.apache.org
Subject: Re: Reg: Tailing a remote file

If you're using a rolling file name pattern you will want to tail the current log file, and
essentially ignore the rolled history, unless you need to represses or miss data.

TailFile is a local processor, so you would run a nifi agent on the box that has the files
on, use tail file -> remote process group -> input port on core nifi instance ->
wherever your data is going next.

The local/remote setting comes up when clustering nifi and using something like an NFS mount
to get to the files being tailed, which is a different way of doing this. The remote will
lead to state about where you've gotten to being stored differently to support the cluster
continuing to tail from alternative nifi cluster nodes.

Simon


> On 21 Apr 2016, at 11:38, Sourav Gulati <sourav.gulati@impetus.co.in> wrote:
>
> Hi Simon/Team.
>
> Please provide some good example of tail processor.
>
> I need some example for " Rolling Filename Pattern".
>
> Also, as per my knowledge TailFile processor is used for tailing local file then what
does location attribute signifies, if I set it to remote.
>
> Regards,
> Sourav Gulati
>
> -----Original Message-----
> From: Simon Ball [mailto:sball@hortonworks.com]
> Sent: Wednesday, April 20, 2016 2:27 PM
> To: dev@nifi.apache.org
> Subject: Re: Reg: Tailing a remote file
>
> Site-to-site is in pretty much every version of nifi released under apache.
>
>> On 20 Apr 2016, at 10:48, Sourav Gulati <sourav.gulati@impetus.co.in> wrote:
>>
>> Thanks Simon. Which version of Nifi support site to site?
>>
>> Regards,
>> Sourav Gulati
>>
>> -----Original Message-----
>> From: Simon Ball [mailto:sball@hortonworks.com]
>> Sent: Wednesday, April 20, 2016 1:28 PM
>> To: dev@nifi.apache.org
>> Subject: Re: Reg: Tailing a remote file
>>
>> The best way to do this would be to run a nifi agent on the remote system, using
the TailFile processor, and then use site-to-site to get that into your core nifi.
>>
>> Simon
>>
>>
>>> On 20 Apr 2016, at 09:49, Sourav Gulati <sourav.gulati@impetus.co.in> wrote:
>>>
>>> Hi Team,
>>>
>>> We want to process logs from a file which is getting updated continuously on
remote system.
>>>
>>> What is the best way to do that in NiFi?
>>>
>>> Regards,
>>> Sourav Gulati
>>>
>>>
>>> ________________________________
>>>
>>>
>>>
>>>
>>>
>>>
>>> NOTE: This message may contain information that is confidential, proprietary,
privileged or otherwise protected by law. The message is intended solely for the named addressee.
If received in error, please destroy and notify the sender. Any use of this email is prohibited
when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity
of this communication has been maintained nor that the communication is free of errors, virus,
interception or interference.
>>
>> ________________________________
>>
>>
>>
>>
>>
>>
>> NOTE: This message may contain information that is confidential, proprietary, privileged
or otherwise protected by law. The message is intended solely for the named addressee. If
received in error, please destroy and notify the sender. Any use of this email is prohibited
when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity
of this communication has been maintained nor that the communication is free of errors, virus,
interception or interference.
>
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential, proprietary, privileged
or otherwise protected by law. The message is intended solely for the named addressee. If
received in error, please destroy and notify the sender. Any use of this email is prohibited
when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity
of this communication has been maintained nor that the communication is free of errors, virus,
interception or interference.
>

________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

Mime
View raw message