nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tony Kurc <trk...@gmail.com>
Subject RE: Issue writing file (~50mb) to azure data lake with Nifi
Date Tue, 06 Sep 2016 21:40:59 GMT
I was referring to this: "Then we wrote a java sdk script to upload this
same file without Nifi into the data lake and it worked successfully."

Is that code somewhere?

On Sep 6, 2016 5:38 PM, "Kumiko Yada" <Kumiko.Yada@ds-iq.com> wrote:

> I didn’t add any test code.  This custom controller and processor is
> working for a small size file.
>
>
>
> Thanks
>
> Kumiko
>
>
>
> *From:* Tony Kurc [mailto:trkurc@gmail.com]
> *Sent:* Tuesday, September 6, 2016 2:32 PM
> *To:* users@nifi.apache.org
> *Cc:* Joe Witt <joe.witt@gmail.com>; #Operations Automation and Tools <#
> opstools@ds-iq.com>; Kevin Verhoeven <Kevin.Verhoeven@ds-iq.com>
> *Subject:* RE: Issue writing file (~50mb) to azure data lake with Nifi
>
>
>
> I didn't see the test script that worked in the source code - did I miss
> it, or is it not in the tree?
>
>
>
> On Sep 6, 2016 3:17 PM, "Kumiko Yada" <Kumiko.Yada@ds-iq.com> wrote:
>
> Joe,
>
>
>
> Here is the log (there was no callstack related to this error) and code,
> https://github.com/kyada1/dl_sdkworkaround/tree/master/
> nifi-azure-dlstore-bundle.
>
>
>
> 2016-09-06 12:06:50,508 INFO [NiFi Web Server-19] c.s.j.s.i.application.WebApplicationImpl
> Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM'
>
> 2016-09-06 12:07:00,991 INFO [StandardProcessScheduler Thread-1] o.a.n.c.s.TimerDrivenSchedulingAgent
> Scheduled PutFileAzureDLStore[id=00dd95dc-0157-1000-8ab1-2de88c159b55] to
> run with 1 threads
>
> 2016-09-06 12:07:01,545 INFO [Flow Service Tasks Thread-1]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@28f414a1 // Another save
> pending = false
>
> 2016-09-06 12:07:01,904 INFO [pool-27-thread-1] c.m.aad.adal4j.AuthenticationAuthority
> [Correlation ID: 564fb5ec-643b-43e6-ab68-59f259a4843a] Instance discovery
> was successful
>
> 2016-09-06 12:08:05,988 ERROR [Timer-Driven Process Thread-1]
> n.a.d.processors.PutFileAzureDLStore PutFileAzureDLStore[id=
> 00dd95dc-0157-1000-8ab1-2de88c159b55] File was not created: /kumiko/test/20160906120701022.txt
> com.microsoft.azure.management.datalake.store.models.AdlsErrorException:
> Invalid status code 404
>
> 2016-09-06 12:08:12,541 INFO [Provenance Maintenance Thread-1] o.a.n.p.PersistentProvenanceRepository
> Created new Provenance Event Writers for events starting with ID 618
>
> 2016-09-06 12:08:12,838 INFO [Provenance Repository Rollover Thread-1]
> o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal
> files (1 records) into single Provenance Log File
> .\provenance_repository\617.prov in 305 milliseconds
>
> 2016-09-06 12:08:12,838 INFO [Provenance Repository Rollover Thread-1]
> o.a.n.p.PersistentProvenanceRepository Successfully Rolled over
> Provenance Event file containing 1 records
>
> 2016-09-06 12:08:21,148 INFO [NiFi Web Server-28] o.a.n.controller.StandardProcessorNode
> Stopping processor: class nifi.azure.dlstore.processors.
> PutFileAzureDLStore
>
>
>
> Thanks
>
> Kumiko
>
>
>
> *From:* Joe Witt [mailto:joe.witt@gmail.com]
> *Sent:* Friday, September 2, 2016 2:56 PM
> *To:* users@nifi.apache.org
> *Cc:* #Operations Automation and Tools <#opstools@ds-iq.com>; Kevin
> Verhoeven <Kevin.Verhoeven@ds-iq.com>
> *Subject:* Re: Issue writing file (~50mb) to azure data lake with Nifi
>
>
>
> Lorenzo
>
> Without seeing the code and logs it would be very difficult to help.
> nifi has no trouble by design writing large files (GBs) to many things
> including hdfs so the issue is probably in how this client library
> interacts with the data stream.
>
>
>
> On Sep 2, 2016 4:19 PM, "Lorenzo Peder" <Lorenzo.Peder@ds-iq.com> wrote:
>
> Hi All,
>
>
>
> We’ve run into an issue uploading a larger file (~50Mb) into an Azure Data
> Lake using a custom processor in nifi 0.7-1.0. This custom processor has
> worked consistently for smaller files, but once encountered with this
> larger file, it spits http error 404 (file not found). Eventually a minor
> portion of the file wrote to the data lake.
>
> We used fiddler to capture network traffic between Nifi and the Azure data
> lake while the processor was running and captured http error 204 (no
> contents).
>
> Then we wrote a java sdk script to upload this same file without Nifi into
> the data lake and it worked successfully.
>
> These findings lead us to believe that this issue is occurring within
> Nifi, if someone could please point us in the right direction in resolving
> this issue it would be greatly appreciated.
>
>
>
> Thank you,
>
>
>
> Lorenzo Peder
>
> Operations Analyst, Campaign Operations & Services
>
>
>
> 425.974.1363 : Office
>
> 425.260.5027 : Mobile
>
> www.ds-iq.com
>
>
>
> Dynamic Shopper Intelligence
>
>
>
> This e-mail may contain confidential or privileged information.
>
> If you are not the intended recipient, please notify the sender
> immediately and then delete this message.
>
>
>
>
>
>

Mime
View raw message