nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lee Laim <lee.l...@gmail.com>
Subject Re: Calculating the theoretical throughput of a Nifi server
Date Mon, 17 Oct 2016 13:02:22 GMT
Hi Ali,

I observed ~1GB/sec on a test PutFile processor using an enterprise PCIe  NVMe ssd on a single
instance on desktop class hardware.  I plan to run more in depth tests on server class hardware
but will likely be on 1 Gb network. I should note I'm not sure exactly how much provenance
was being written. 

The nifi-0.7.0  instance was a fresh install with no major configuration changes.  I was using
the generate flowfiles processor to generate 100MB flowfiles and writing as fast as possible
with a PutFile processor.

The SSD posted the following on the AS-SSD benchmark (completely unoptimized):
1.8GB/sec for sequential write;
2.3GB/sec for 4K random write (64 treads);
114MB/sec for 4K random write (1 thread)

On the PCie bus, you should easily surpass 800 MB/sec, especially if your flowfiles are large
and you have few provenance events.  The theoretical bandwidth is 985 MB/sec/lane, up to 16
lanes; I was running x4.  The NVMe standard should also help with smaller flowfiles.  

Hope this helps,
Lee



> On Oct 17, 2016, at 12:06 PM, Ali Nazemian <alinazemian@gmail.com> wrote:
> 
> Dear Nifi users/developers,
> Hi,
> 
> I was wondering how can I calculate the theoretical throughput of a Nifi server? let's
suppose we can eliminate different bottlenecks such as the file flow rep and provenance repo
bottleneck by using a very high-end SSD.  Moreover, assume that a very high-end network infrastructure
is available. In this case, is it possible to reach 800MB throughput per second per each server?
Suppose each server comes with 24 disk slots. 16 disk slots are used for creating 8 x RAID1(SAS
10k) mount points and are dedicated to the content repo. Let's say each content repo can achieve
100 MB throughput. May I say the total throughput per each server can be 8x100=800MBps?  Is
it possible to reach this amount of throughput practically?
> Thank you very much.
> 
> Best regards,
> Ali

Mime
View raw message