metron-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Elliston Ball <si...@simonellistonball.com>
Subject Re: Define a function that can be used in Stellar
Date Fri, 02 Feb 2018 13:44:14 GMT
Depends how you write the function class, but most likely, yes. Hence global config option.


Simon

> On 2 Feb 2018, at 13:42, Ali Nazemian <alinazemian@gmail.com> wrote:
> 
> Does it mean every time the function gets called it will load the config, but if I use
the global one it will only read it one time and it will be available in memory?
> 
> On 2 Feb. 2018 21:53, "Simon Elliston Ball" <simon@simonellistonball.com <mailto:simon@simonellistonball.com>>
wrote:
> Shouldn’t be. The one this I would point out though is that you don’t necessarily
know which supervisor you will be running from, so pulling from HDFS would make sense. That
said, the performance implications are probably not great. A good option here would be to
have the config available in the global config for example and refer to that, since most instances
of stellar apply global config to their context. 
> 
> Simon
> 
> 
>> On 2 Feb 2018, at 07:14, Ali Nazemian <alinazemian@gmail.com <mailto:alinazemian@gmail.com>>
wrote:
>> 
>> Will be any problem if the Stellar function we want to implement need to load an
external config file?
>> 
>> Cheers,
>> Ali
>> 
>> On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian <alinazemian@gmail.com <mailto:alinazemian@gmail.com>>
wrote:
>> Thanks, All.
>> 
>> Yes, Nick. It is highly related to our use case and the way that we are going to
enrich events with assets and vulnerability properties. It is not a general case at all.
>> 
>> Cheers,
>> Ali
>> 
>> On Thu, Jan 18, 2018 at 5:43 AM, Matt Foley <mattf@apache.org <mailto:mattf@apache.org>>
wrote:
>> Besides the example code Simon mentioned at https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
<https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example>
,
>> there is some documentation at http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
<http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html>
>>  
>> 
>> From: Nick Allen <nick@nickallen.org <mailto:nick@nickallen.org>>
>> Reply-To: "user@metron.apache.org <mailto:user@metron.apache.org>" <user@metron.apache.org
<mailto:user@metron.apache.org>>
>> Date: Wednesday, January 17, 2018 at 4:46 AM
>> To: "user@metron.apache.org <mailto:user@metron.apache.org>" <user@metron.apache.org
<mailto:user@metron.apache.org>>
>> Subject: Re: Define a function that can be used in Stellar
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>> If something we have already does not fit the bill, I would recommend creating that
function in Java.   Since you described it as "a bit complex" and "the logic would be complicated"
I don't see any value in defining something like this in Stellar with named functions.
>> 
>>  
>> 
>> Best
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>> On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball <simon@simonellistonball.com
<mailto:simon@simonellistonball.com>> wrote:
>> 
>> Have you looked at the recent TLSH functions in Stellar? We already have that for
similarity preserving hashes.
>> 
>>  
>> 
>> Simon
>> 
>>  
>> 
>> 
>> On 17 Jan 2018, at 12:35, Ali Nazemian <alinazemian@gmail.com <mailto:alinazemian@gmail.com>>
wrote:
>> 
>> It is a bit complex. We want to create a function that accepts a list of arguments
for an asset and generate an asset identifier that can be used as a row_key for the enrichment
store. The logic would be complicated, though. We may need to include some sort of similarity
aware hash function as a part of this custom function.
>> 
>>  
>> 
>> On Wed, Jan 17, 2018 at 10:32 PM, Nick Allen <nick@nickallen.org <mailto:nick@nickallen.org>>
wrote:
>> 
>> Ali - Can you describe the logic that you are trying to perform? That would be useful
as a use case to help drive a discussion around creating named functions in Stellar.
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>> On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian <alinazemian@gmail.com <mailto:alinazemian@gmail.com>>
wrote:
>> 
>> Thanks, Simon. We have already got a script to deal with classpath management for
the parsers. We should be able to use it for this extension as well.
>> 
>>  
>> 
>> Yeah, I agree. It will be much easier to define functions on the fly and use them
afterwards. It could be defined as Lambda or custom function. 
>> 
>>  
>> 
>> Regards,
>> 
>> Ali
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>> On Wed, Jan 17, 2018 at 9:42 PM, Simon Elliston Ball <simon@simonellistonball.com
<mailto:simon@simonellistonball.com>> wrote:
>> 
>> https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
<https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example>
gives good details on how to add a stellar function.
>> 
>>  
>> 
>> Stellar will pick up an annotated function on its class path, so to add function
there is no need to rebuild metron module, but you do need your modules on the classpath,
and, pending 777, to deal with things like class path clash in your dependencies. 
>> 
>>  
>> 
>> Another idea worth discussion on the dev list is probably the notion of defining
stellar functions in stellar, which would be a much simpler solution than custom java functions
if you can already express you logic in stellar. 
>> 
>>  
>> 
>> Simon
>> 
>>  
>> 
>> 
>> 
>> 
>> On 17 Jan 2018, at 10:37, Ali Nazemian <alinazemian@gmail.com <mailto:alinazemian@gmail.com>>
wrote:
>> 
>>  
>> 
>> Hi Simon,
>> 
>>  
>> 
>> Yes, that is exactly what we are looking for. Is there any example regarding adding
a Stellar function in Java? Hopefully, we don't need to rebuild the corresponding modules
for this?
>> 
>>  
>> 
>> Cheers,
>> 
>> Ali
>> 
>>  
>> 
>> On Wed, Jan 17, 2018 at 8:40 PM, Simon Elliston Ball <simon@simonellistonball.com
<mailto:simon@simonellistonball.com>> wrote:
>> 
>> At present you can certainly create custom stellar functions in Java. I’m guessing
however that what you’re looking to do is create a kind of function that combines a number
of stellar functions to avoid repetition, or to ensure consistency of certain parameters for
example. Is that what you’re looking for? Maybe some sort of syntax to create a named stellar
function similar to the way we create lambdas?
>> 
>> Simon
>> 
>> 
>> > On 17 Jan 2018, at 07:25, Ali Nazemian <alinazemian@gmail.com <mailto:alinazemian@gmail.com>>
wrote:
>> >
>> > Hi all,
>> >
>> > Is there any way that we can define a function that can be used rather than
duplicating a logic multiple times?
>> >
>> > Cheers,
>> > Ali
>> 
>> 
>> 
>> 
>>  
>> 
>> --
>> 
>> A.Nazemian
>> 
>>  
>> 
>> 
>> 
>> 
>>  
>> 
>> --
>> 
>> A.Nazemian
>> 
>> 
>> 
>> 
>>  
>> 
>> --
>> 
>> A.Nazemian
>> 
>> 
>> 
>> 
>> -- 
>> A.Nazemian
>> 
>> 
>> 
>> -- 
>> A.Nazemian
> 


Mime
View raw message