nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Milan Das <m...@interset.com>
Subject Re: Finding Performance bottleneck issue
Date Sat, 17 Mar 2018 21:20:20 GMT
Thanks for your help Mike Thomsen & Joe Witt.
Performance problem was in the way I was each line from flow input. My logic was : io.InputStream->io.BufferedReader->
while reader.readLine() loop.
Now I am using guava library.
I have changed the code to use “String inputString = CharStreams.toString(new InputStreamReader(in,
"UTF-8"));”
I gained 20 times  performance.


Regards,
Milan Das





On 3/16/18, 12:07 PM, "Mike Thomsen" <mikerthomsen@gmail.com> wrote:

    That seems like a very reasonable use case. You said:
    
    > I see that my processor is actually queuing up  records at source.
    
    Are you saying that the processor isn't able to process them that quickly
    such that you're seeing a big backlog in the input queue?
    
    On Fri, Mar 16, 2018 at 11:56 AM, Milan Das <mdas@interset.com> wrote:
    
    > Hi Mike,
    > My processor is processing windows Text event as below and creating a JSON
    > out of it.
    > Also I am applying simple JoltTransformer (Simple just Shift and Default)
    > to convert to different  JSON (no hierarchy) .
    >
    > Output have the following:
    > 1. Original text
    > 2. Converted JSON
    > 3. JOLT transformed JSON
    > 4. Failure
    >
    >
    > Steps in program:
    > 1. Converting the event to Java Map (using regex: "([^:=]*)[:=]([^:=]*)")
    > 2. Map to Json (using Gson)
    > 3. Jolt transfeormation
    >
    >
    >
    > Example event:
    >
    > Examples of 4626
    > User / Device claims information.
    >
    > Subject:
    >     Security ID:     %1
    >     Account Name:    %2
    >     Account Domain:  %3
    >     Logon ID:        %4
    >
    > Logon Type:          %9
    >
    > New Logon:
    >     Security ID:     %5
    >     Account Name:    %6
    >     Account Domain:  %7
    >     Logon ID:        %8
    >
    > Event in sequence:   %10 of %11
    >
    > User Claims:         %12
    >
    > Device Claims:       %13
    >
    > The subject fields indicate the account on the local system which
    > requested the logon. This is most commonly a service such as the Server
    > service, or a local process such as Winlogon.exe or Services.exe.
    >
    > The logon type field indicates the kind of logon that occurred. The most
    > common types are 2 (interactive) and 3 (network).
    >
    > The New Logon fields indicate the account for whom the new logon was
    > created, i.e. the account that was logged on.
    >
    > This event is generated when the Audit User/Device claims subcategory is
    > configured and the user’s logon token contains user/device claims
    > information. The Logon ID field can be used to correlate this event with
    > the corresponding user logon event as well as to any other security audit
    > events generated during this logon session.
    >
    >
    >
    > Regards,
    > Milan Das
    >
    >
    > On 3/16/18, 10:56 AM, "Mike Thomsen" <mikerthomsen@gmail.com> wrote:
    >
    >     Milan,
    >
    >     Can you share some details about where you are running into problems?
    > Like
    >     a basic description of what it's trying to do?
    >
    >     On Fri, Mar 16, 2018 at 10:39 AM, Milan Das <mdas@interset.com> wrote:
    >
    >     > I have a custom processor, it works as expected. But I feel there is
    > some
    >     > performance measure need to be done. I see that my processor is
    > actually
    >     > queuing up  records at source.
    >     >
    >     > Is there a run a load  test and do performance measure for Custom
    >     > Processor?
    >     >
    >     >
    >     >
    >     > Regards,
    >     >
    >     > Milan Das
    >     >
    >     >
    >
    >
    >
    >
    



Mime
View raw message