metron-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Re: Define a function that can be used in Stellar
Date Fri, 02 Feb 2018 14:49:41 GMT
I think if we understand the use case, we may be able to think of a more
general set of functionality for stellar to meet this and other cases.

Will this configuration change?  Do you need to track that change without
reloading?  How *much* is in the configuration?  Do we want
people putting their own keys in the global and have to manage that
upgrading ( I think it will get blown away )?






On February 2, 2018 at 09:38:28, Casey Stella (cestella@gmail.com) wrote:

We use a guava cache to cache the data for 24 hours.  You can see how it's
done here:
https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/ObjectGet.java

We also do something like this in GEO_GET as well, but it's a bit more
complex.

On Fri, Feb 2, 2018 at 9:35 AM, Simon Elliston Ball <
simon@simonellistonball.com> wrote:

> I forgot we added OBJECT_GET. How does the caching work on that?
>
> Simn
>
>
> On 2 Feb 2018, at 14:33, Nick Allen <nick@nickallen.org> wrote:
>
> There are many functions that use the global configuration.  For example,
> GET_GEO in org.apache.metron.enrichment.stellar.GeoEnrichmentFunctions.
> There might be a better example, but that is one is staring at me at the
> moment.
>
> There is an OBJECT_GET function defined in org.apache.metron.enrichment.stellar.ObjectGet
> that was purpose-built to retrieve files from HDFS.  If you wanted to
> retrieve a configuration from HDFS that would be a good example (if you
> can't just use that functions directly).
>
> On Fri, Feb 2, 2018 at 8:50 AM Ali Nazemian <alinazemian@gmail.com> wrote:
>
>> Is there any Stellar function already been implemented in Metron that has
>> a config file associated with it? I am trying to get an idea of how it
>> works.
>>
>> On 3 Feb. 2018 00:44, "Simon Elliston Ball" <simon@simonellistonball.com>
>> wrote:
>>
>>> Depends how you write the function class, but most likely, yes. Hence
>>> global config option.
>>>
>>> Simon
>>>
>>> On 2 Feb 2018, at 13:42, Ali Nazemian <alinazemian@gmail.com> wrote:
>>>
>>> Does it mean every time the function gets called it will load the
>>> config, but if I use the global one it will only read it one time and it
>>> will be available in memory?
>>>
>>> On 2 Feb. 2018 21:53, "Simon Elliston Ball" <simon@simonellistonball.com>
>>> wrote:
>>>
>>>> Shouldn’t be. The one this I would point out though is that you don’t
>>>> necessarily know which supervisor you will be running from, so pulling from
>>>> HDFS would make sense. That said, the performance implications are probably
>>>> not great. A good option here would be to have the config available in the
>>>> global config for example and refer to that, since most instances of
>>>> stellar apply global config to their context.
>>>>
>>>> Simon
>>>>
>>>>
>>>> On 2 Feb 2018, at 07:14, Ali Nazemian <alinazemian@gmail.com> wrote:
>>>>
>>>> Will be any problem if the Stellar function we want to implement need
>>>> to load an external config file?
>>>>
>>>> Cheers,
>>>> Ali
>>>>
>>>> On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian <alinazemian@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks, All.
>>>>>
>>>>> Yes, Nick. It is highly related to our use case and the way that we
>>>>> are going to enrich events with assets and vulnerability properties.
It is
>>>>> not a general case at all.
>>>>>
>>>>> Cheers,
>>>>> Ali
>>>>>
>>>>> On Thu, Jan 18, 2018 at 5:43 AM, Matt Foley <mattf@apache.org>
wrote:
>>>>>
>>>>>> Besides the example code Simon mentioned at
>>>>>> https://github.com/apache/metron/tree/master/metron-
>>>>>> stellar/stellar-3rd-party-example ,
>>>>>> there is some documentation at http://metron.apache.org/
>>>>>> current-book/metron-stellar/stellar-common/3rdPartyStellar.html
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From:* Nick Allen <nick@nickallen.org>
>>>>>> *Reply-To:* "user@metron.apache.org" <user@metron.apache.org>
>>>>>> *Date:* Wednesday, January 17, 2018 at 4:46 AM
>>>>>> *To:* "user@metron.apache.org" <user@metron.apache.org>
>>>>>> *Subject:* Re: Define a function that can be used in Stellar
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> If something we have already does not fit the bill, I would recommend
>>>>>> creating that function in Java.   Since you described it as "a bit
complex"
>>>>>> and "the logic would be complicated" I don't see any value in defining
>>>>>> something like this in Stellar with named functions.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Best
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball <
>>>>>> simon@simonellistonball.com> wrote:
>>>>>>
>>>>>> Have you looked at the recent TLSH functions in Stellar? We already
>>>>>> have that for similarity preserving hashes.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Simon
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 17 Jan 2018, at 12:35, Ali Nazemian <alinazemian@gmail.com>
wrote:
>>>>>>
>>>>>> It is a bit complex. We want to create a function that accepts a
list
>>>>>> of arguments for an asset and generate an asset identifier that can
be used
>>>>>> as a row_key for the enrichment store. The logic would be complicated,
>>>>>> though. We may need to include some sort of similarity aware hash
function
>>>>>> as a part of this custom function.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 17, 2018 at 10:32 PM, Nick Allen <nick@nickallen.org>
>>>>>> wrote:
>>>>>>
>>>>>> Ali - Can you describe the logic that you are trying to perform?
That
>>>>>> would be useful as a use case to help drive a discussion around creating
>>>>>> named functions in Stellar.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian <alinazemian@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Thanks, Simon. We have already got a script to deal with classpath
>>>>>> management for the parsers. We should be able to use it for this
extension
>>>>>> as well.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yeah, I agree. It will be much easier to define functions on the
fly
>>>>>> and use them afterwards. It could be defined as Lambda or custom
function.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Ali
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 17, 2018 at 9:42 PM, Simon Elliston Ball <
>>>>>> simon@simonellistonball.com> wrote:
>>>>>>
>>>>>> https://github.com/apache/metron/tree/master/metron-
>>>>>> stellar/stellar-3rd-party-example gives good details on how to add
a
>>>>>> stellar function.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Stellar will pick up an annotated function on its class path, so
to
>>>>>> add function there is no need to rebuild metron module, but you do
need
>>>>>> your modules on the classpath, and, pending 777, to deal with things
like
>>>>>> class path clash in your dependencies.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Another idea worth discussion on the dev list is probably the notion
>>>>>> of defining stellar functions in stellar, which would be a much simpler
>>>>>> solution than custom java functions if you can already express you
logic in
>>>>>> stellar.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Simon
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 17 Jan 2018, at 10:37, Ali Nazemian <alinazemian@gmail.com>
wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Simon,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yes, that is exactly what we are looking for. Is there any example
>>>>>> regarding adding a Stellar function in Java? Hopefully, we don't
need to
>>>>>> rebuild the corresponding modules for this?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Ali
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 17, 2018 at 8:40 PM, Simon Elliston Ball <
>>>>>> simon@simonellistonball.com> wrote:
>>>>>>
>>>>>> At present you can certainly create custom stellar functions in Java.
>>>>>> I’m guessing however that what you’re looking to do is create
a kind of
>>>>>> function that combines a number of stellar functions to avoid repetition,
>>>>>> or to ensure consistency of certain parameters for example. Is that
what
>>>>>> you’re looking for? Maybe some sort of syntax to create a named
stellar
>>>>>> function similar to the way we create lambdas?
>>>>>>
>>>>>> Simon
>>>>>>
>>>>>>
>>>>>> > On 17 Jan 2018, at 07:25, Ali Nazemian <alinazemian@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > Hi all,
>>>>>> >
>>>>>> > Is there any way that we can define a function that can be used
>>>>>> rather than duplicating a logic multiple times?
>>>>>> >
>>>>>> > Cheers,
>>>>>> > Ali
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> A.Nazemian
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> A.Nazemian
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> A.Nazemian
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> A.Nazemian
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> A.Nazemian
>>>>
>>>>
>>>>
>>>
>

Mime
View raw message