Mailing-List: contact users-help@nifi.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@nifi.apache.org
MIME-Version: 1.0
In-Reply-To: <CAA2dGFuqRoJGbWNy7E4TRVOq+4WUOeSt0X7-1PoaOVoW+55fzQ@mail.gmail.com>
References: <CAA2dGFuqRoJGbWNy7E4TRVOq+4WUOeSt0X7-1PoaOVoW+55fzQ@mail.gmail.com>
From: Koji Kawamura <ijokarumawak@gmail.com>
Date: Thu, 7 Sep 2017 14:45:17 +0900
Message-ID: <CA+WJ-+LGNW-3pSd3S8+tewUF1whLrByYQPiMp2tqf9szBRX0Uw@mail.gmail.com>
Subject: Re: Slow FTP and SFTP nifi transfer rates
To: users@nifi.apache.org
Content-Type: text/plain; charset="UTF-8"
archived-at: Thu, 07 Sep 2017 05:45:32 -0000

Hello Gino,

Thanks for sharing your findings on FTP performance.

How did you measure send rate from NiFi to your FTP server?

Sending multiple FlowFiles would provide less throughput compared to
sending one big FlowFile, as PutFTP and PutSFTP make connection to
each incoming FlowFile. The overhead of establishing connection each
time might be the performance difference you see with mput command.

Those processors can decide which FTP servers to use based on incoming
FlowFiles' attribute when NiFi Expression Language is used.

If that's the case, there are some room for performance improvement by
keeping underlying FTP(S) client instance so that it can be reused
among multiple onTrigger() call.

A possible work-around would be using MergeContent beforehand and send
it as a single file, if your use-case allows that.

Thanks,
Koji

On Thu, Sep 7, 2017 at 12:15 PM, Gino Lisignoli <glisignoli@gmail.com> wrote:
> I have this weird issue with PutFTP and PutSFTP transfer rates.
>
> What I am seeing is that no matter what files I transfer from One server to
> another over a single connection the maximum rates I can send are 300Mbps
> for PutFTP and 1Gbps for PutSFTP.
>
> The sending nifi is installed on Centos 7, running on a Dell R730, 190GB
> Ram, 16 Cores @ 2.4GHz and 4x10Gb nics bonded. The sending nifi has it's
> content repository on a ramdisk, and the receiving server is receiving to a
> ramdisk (for testing, to remove disk IO out of the equation).
>
> When I do a ftp send manually (without nifi) with mput I get ftp rates of
> ~8Gbs and sftp rates of 2.2Gbs (Which seems slow anyway).
>
> I would have expected transfer rates similar with nifi.
>
> Is there any way to work out why these rates are so much slower, but also so
> consistent? I'm using Nifi-1.30