nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Processor classpath
Date Fri, 07 Jul 2017 15:17:48 GMT
Hi James,

I'm saying that it should not find a core-site.xml in NiFi's conf
directory because the conf directory is not on the classpath of
processors.

Are you saying you have tested putting it there and believe it is finding it?

-Bryan

On Fri, Jul 7, 2017 at 11:08 AM, James Srinivasan
<james.srinivasan@gmail.com> wrote:
> Hi Bryan,
>
> That all makes sense...except this is my own NiFi processor, not one
> of the built-in Hadoop ones:
>
> https://github.com/jrs53/geomesa-nifi
>
> So I'm still confused as to why core-site.xml is picked up from NiFi's
> conf directory, but very happy that it is!
>
> On 5 July 2017 at 21:55, Bryan Bende <bbende@gmail.com> wrote:
>> James,
>>
>> Just to clarify... the conf directory of NiFi is not on the classpath
>> of processors.
>>
>> The behavior with core-site.xml is because the Hadoop client used by
>> NiFi provides this behavior, and not NiFi itself. Basically if you
>> create a new Hadoop Configuration object and you don't provide it with
>> a list of resources, it will find any that are on the classpath, which
>> in this case would be any that are bundled in JARs inside the given
>> NAR, or anything in NiFI's lib directory. Putting stuff in NiFi's lib
>> directory makes it visible to every processor and is generally
>> dangerous, which is why the existing Hadoop processors have a property
>> for the path to the core-site.xml file and then load it like so:
>>
>> String resource = ... // get the value of the property where the path is
>> Configuration config = new Configuration();
>> config.addResource(new Path(resource.trim()));
>>
>> Hope that helps.
>>
>> -Bryan
>>
>>
>>
>> On Wed, Jul 5, 2017 at 4:15 PM, James Srinivasan
>> <james.srinivasan@gmail.com> wrote:
>>> Thanks, I ended up stracing the NiFi process to see where it was
>>> actually trying to load my config file from. Helpfully, the list of
>>> locations included the NiFi conf directory, which is just perfect for
>>> me.
>>>
>>> I don't want to keep the config file (core-site.xml for Hadoop) in my
>>> NAR itself since it will obviously change between different
>>> deployments. I was inspired by:
>>>
>>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java#L69-L70
>>>
>>> Which says if that property isn't set, then NiFi searches the
>>> classpath for the necessary file.
>>>
>>> Thanks again,
>>>
>>> James
>>>
>>> On 5 July 2017 at 00:07, Jeremy Dyer <jdye64@gmail.com> wrote:
>>>> Hi James,
>>>>
>>>> NiFi doesn't do anything too crazy from what Java itself does. The "funky"
>>>> part is the isolated classpath loaders. Really since your building a custom
>>>> processor I would also assume your building a custom NAR. If that is the
>>>> case you can read that resource from the "nar" just like you would any "jar"
>>>> in java. The HDFS processor your referenced loads an external file that
>>>> isn't bundled in the actual classpath at all. If that is the functionality
>>>> you want I suggest following the template from the HDFS processors as you
>>>> stated. Otherwise just carryon as usual and pretend its a regular java
>>>> application.
>>>>
>>>> On Tue, Jul 4, 2017 at 4:31 PM, James Srinivasan
>>>> <james.srinivasan@gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I'm developing a processor which needs to read some of its config from
>>>>> the classpath. Reading the docs etc., NiFi's classpath is a little
>>>>> funky - where's the best (least worst?) location for such files? I
>>>>> note that the HDFS processors can read their config (core-site.xml
>>>>> etc) from the classpath, but I can't find where that actually is.
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> James
>>>>
>>>>

Mime
View raw message