nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Witt <joe.w...@gmail.com>
Subject Re: Help with SFTP processor
Date Thu, 06 Apr 2017 16:18:56 GMT
Jim,

Glad you've made progress on the SFTP side.  Please file a JIRA with
your suggestions for the docs and the ideal case then is you'd file a
Pull Request (https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide)
which actually provides the suggested documentation changes.

For the ListFile/FetchFile -> PutSFTP[1] side the key property on
PutSFTP to set is 'Remote Path'.  You'll want this value to have the
base directory you need to write to which could be './' or could be
'some/place/to/write/to' and you'll also want it to reflect the
directory structure from which you fetched the file locally.  This
will be available to you from the 'path' attribute of the flowfile.
This is set by the ListFile processor (see writes attributes) [2].

So putting these together you want your PutSFTP processor to have as a
value for 'Remote Path' something like "thebasedir/fordata/${path}".


[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.PutSFTP/index.html
[2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListFile/index.html

On Thu, Apr 6, 2017 at 11:43 AM, James Keeney <nextvestor@gmail.com> wrote:
> Joe and Juan -
>
> Thank you very much for the help. It turned out to be the prompt for
> verifying the authenticity of the host.
>
> With that fixed, I have a new question:
>
> I'm using ListFile and FetchFile to identify new files as they are added to
> a directory. When they are I am using SFTP to transfer to another server
> behind the firewall. I'd like to be able to preserve the directory structure
> when I transfer the files. I set Create Directory to true but the SFTP
> transfer is always putting the files in the root.
>
> Any ideas?
>
> Here is how I resolve the first issue:
>
> I added the host keys to the known_hosts file and that did the trick.
>
> From a documentation perspective I'd suggest adding a little more guidance
> for people who are not familiar with SFTP. It was not clear to me what the
> two parameters are:
>
> Host Key File
> Private Key Path
>
> Since I didn't understand the use of the known_hosts file (using key-scan to
> get their keys and adding those to the file I had no idea what to put in
> Host Key File. Also, private key path confused me since it is the public key
> that is being shared. Also, we might want to highlight that the SFTP process
> needs to go forward without prompts. I had gotten use to the prompt so when
> I tested I didn't even think about that issue.
>
> How might I go about adding my sweat equity to update the documentation to
> make it a little clearer? Let me know and I will take a crack at expandign
> the information to help other users.
>
> Thanks.
>
> Jim K.
>
> On Tue, Apr 4, 2017 at 12:31 PM Joe Witt <joe.witt@gmail.com> wrote:
>>
>> definitely agree with Juan's suggestion to get more details on what
>> the actual authentication process is when trying ssh -vvv.  Also, be
>> sure to check what order of authorization occurs.  It is possible it
>> is trying keyboard-interactive before the certs and this could create
>> problems so ordering there, on the server side, will really matter.
>>
>> On Tue, Apr 4, 2017 at 12:27 PM, Juan Sequeiros <hellojuan@gmail.com>
>> wrote:
>> > Good afternoon,
>> >
>> > I would try this command from command line:
>> >
>> > ssh -vvv -i <path to rsa file> user@server
>> >
>> > Example:
>> >
>> > ssh -vvv -i /some/path/.ssh/id_rsa nifi@10.10.10.10
>> >
>> > If that works then I would double check the "private key path" property
>> > of
>> > your GetSFTP it should point to the fully qualified file to the private
>> > key
>> > path.
>> >
>> > If it does not work then the -vvv option should give you more error.
>> >
>> >
>> >
>> > On Tue, Apr 4, 2017 at 9:47 AM James Keeney <nextvestor@gmail.com>
>> > wrote:
>> >>
>> >> I am using SFTP to transfer files between two servers. I have tried
>> >> multiple configurations to try to get the authentication to work but i
>> >> keep
>> >> getting the Auth Fail error. I'm able to go onto the Nifi server sftp
>> >> over
>> >> to the destination server but I cannot get it to work in Nifi. I'm just
>> >> not
>> >> sure how to debug this so I was hoping someone could help. Here are the
>> >> setting I'm using (I've replaced all the important details with
>> >> placeholders):
>> >>
>> >> Hostname: <Hostname. I've tried both IP and hostname without luck>
>> >> Port: 22 <I've confirmed this by connecting directly from server to
>> >> server)
>> >> username: <username>
>> >> Host key File: <location on the server of the public key file>
>> >>
>> >> The error I am receiving is below. Any help would be greatly
>> >> appreciated.
>> >>
>> >> Thanks.
>> >>
>> >> Jim K
>> >>
>> >>
>> >>
>> >>
>> >> 2017-03-29 13:59:51,705 ERROR [Timer-Driven Process Thread-1]
>> >> o.a.nifi.processors.standard.PutSFTP
>> >> PutSFTP[id=fcdd2eb4-015a-1000-80c5-7406e6fca4c5] Unable to transfer
>> >>
>> >> StandardFlowFileRecord[uuid=a3569afa-7c80-4cec-9239-2424172e30d1,claim=StandardContentClaim
>> >> [resourceClaim=StandardResourceClaim[id=1490795991637-151,
>> >> container=default, section=151], offset=0,
>> >> length=2917388],offset=0,name=tulips_248_042214.jpg,size=2917388] to
>> >> remote
>> >> host <servername> due to
>> >> org.apache.nifi.processor.exception.ProcessException: IOException
>> >> thrown
>> >> from PutSFTP[id=fcdd2eb4-015a-1000-80c5-7406e6fca4c5]:
>> >> java.io.IOException:
>> >> Failed to obtain connection to remote host due to
>> >> com.jcraft.jsch.JSchException: Auth fail: java.io.IOException: Failed
>> >> to
>> >> obtain connection to remote host due to com.jcraft.jsch.JSchException:
>> >> Auth
>> >> fail; routing to failure: java.io.IOException: Failed to obtain
>> >> connection
>> >> to remote host due to com.jcraft.jsch.JSchException: Auth fail
>> >> 2017-03-29 13:59:51,706 ERROR [Timer-Driven Process Thread-1]
>> >> o.a.nifi.processors.standard.PutSFTP
>> >> java.io.IOException: Failed to obtain connection to remote host due to
>> >> com.jcraft.jsch.JSchException: Auth fail
>> >>         at
>> >>
>> >> org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:447)
>> >> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
>> >>         at
>> >>
>> >> org.apache.nifi.processors.standard.util.SFTPTransfer.put(SFTPTransfer.java:529)
>> >> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
>> >>         at
>> >>
>> >> org.apache.nifi.processors.standard.PutFileTransfer$1.process(PutFileTransfer.java:135)
>> >> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
>> >>         at
>> >>
>> >> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2082)
>> >> ~[na:na]
>> >>         at
>> >>
>> >> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2053)
>> >> ~[na:na]
>> >>         at
>> >>
>> >> org.apache.nifi.processors.standard.PutFileTransfer.onTrigger(PutFileTransfer.java:127)
>> >> ~[nifi-standard-processors-1.1.0.jar:1.1.0]
>> >>         at
>> >>
>> >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> >> [nifi-api-1.1.0.jar:1.1.0]
>> >>         at
>> >>
>> >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
>> >> [nifi-framework-core-1.1.0.jar:1.1.0]
>> >>         at
>> >>
>> >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>> >> [nifi-framework-core-1.1.0.jar:1.1.0]
>> >>         at
>> >>
>> >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>> >> [nifi-framework-core-1.1.0.jar:1.1.0]
>> >>         at
>> >>
>> >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
>> >> [nifi-framework-core-1.1.0.jar:1.1.0]
>> >>         at
>> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> >> [na:1.8.0_121]
>> >>         at
>> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> >> [na:1.8.0_121]
>> >>         at
>> >>
>> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> >> [na:1.8.0_121]
>> >>         at
>> >>
>> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> >> [na:1.8.0_121]
>> >>         at
>> >>
>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> >> [na:1.8.0_121]
>> >>         at
>> >>
>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> >> [na:1.8.0_121]
>
> --
> Jim Keeney
> President, FitterWeb
> E: jim@fitterweb.com
> M: 703-568-5887
>
> FitterWeb Consulting
> Are you lean and agile enough?

Mime
View raw message