Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@nifi.apache.org
Content-Type: multipart/alternative;
	boundary=Apple-Mail-EC0D26EB-43FF-4838-8856-92728763E62E
Mime-Version: 1.0 (1.0)
Subject: Re: Nifi Capability for Fast transfer of Data
From: Andy LoPresto <alopresto.apache@gmail.com>
In-Reply-To: <CAE0YeNPHRZj8j-K-WQ7ByvpSVfTO-UC0orq1N-qcdsX--bzkDQ@mail.gmail.com>
Date: Fri, 25 Nov 2016 17:37:55 -0800
Cc: users@nifi.apache.org
Content-Transfer-Encoding: 7bit
Message-Id: <2315C40A-9438-4643-BFC4-9EDC208E6AF6@gmail.com>
References: <CAE0YeNPHRZj8j-K-WQ7ByvpSVfTO-UC0orq1N-qcdsX--bzkDQ@mail.gmail.com>
To: dev@nifi.apache.org
archived-at: Sat, 26 Nov 2016 01:38:09 -0000

--Apple-Mail-EC0D26EB-43FF-4838-8856-92728763E62E
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable

Unless my back of the envelope math is way off, to transfer 50GB (400Gb) per=
 second, you would need 40 parallel 10GbE connections, assuming absolutely n=
o overhead. Your precision for "a few seconds" would need to be 40+ seconds u=
sing a single 10 GbE link and optimal transmission speed.=20

=46rom the Apache NiFi Overview document:=20

"for something concrete and broadly applicable, consider the out-of-the-box d=
efault implementations. These are all persistent with guaranteed delivery an=
d do so using local disk. So being conservative, assume roughly 50MB per sec=
ond read/write rate on modest disks or RAID volumes within a typical server.=
 NiFi for a large class of dataflows then should be able to efficiently reac=
h 100MB per second or more of throughput. "

Those numbers are at least 18 months old, so with a robust cluster of 8 high=
-performance machines and an optimized flow to balance computation across al=
l the boxes, I would ballpark a perfect world estimate at 1Gbps. My last kno=
wledge of HDFS write speeds was around 10-20Gbps. Again, if your tolerance f=
or the full process is 40-50 seconds, NiFi should be able to keep up, but yo=
ur uplink will probably be the long pole in the tent here.=20

Feel free to correct any poor assumptions or bad math above.=20

Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Nov 24, 2016, at 20:48, shweta Aggarwal <shweta.agg1982@gmail.com> wrot=
e:
>=20
> Hi folks,
>=20
> We have a requirement in one of our time critical application wherein we
> are looking for transferring upto 40-50 GBs worth images
> within few seconds between remote machine and HDFS.
>=20
> Assuming network connectivity between the two is on a 10Gbe link and NIC
> and socket buffers tuned optimally to give best performance , does Nifi
> have a capability  to support desired performance using a combination of
> "getFile" and "putHDFS" on a high ended cluster of  >8 nodes.
>=20
> We are also exploring a combination of HDFS+GrdiFTP for fast transfer of
> images from remote machine to HDFS cluster.
>=20
> Any thoughts or pointers shall be helpful.
>=20
> Thanks!!

--Apple-Mail-EC0D26EB-43FF-4838-8856-92728763E62E--