nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe@Moosheimer.com" <...@Moosheimer.com>
Subject Re: [DISCUSS] Advanced search capabilities
Date Fri, 21 Feb 2020 22:37:36 GMT
Martin,

correct me if I’m wrong, but Neo4J is not open source. Wouldn't it be generally better to
use Janusgraph for this?
Especially since many major companies use Janusgraph: IBM Cloud, Netflix, Uber...

A complete and fully functional lineage is already available with Apache Atlas. In my opinion
it would be more complicated to build it in NiFi than to integrate Apache Atlas, whereas it
is already standard in NiFi to bridge to Atlas.

Or did I misunderstand something?

Mit freundlichen Grüßen / best regards
Kay-Uwe Moosheimer

> Am 21.02.2020 um 19:42 schrieb Martin Ebert <martin.irgang@gmx.de>:
> 
> We still think about building a graph based search (Neo4j) in top of NiFi.
> Would be also fantastic to have it within NiFi.
> 
> There are plenty of examples
> https://blog.grandstack.io/using-neo4js-full-text-search-with-graphql-e3fa484de2ea
> From the idea it could go in this direction - of course much more
> rudimentary. Then one would have the possibility to have only the results
> displayed as text or to find out exploratory connections (graph layout).
> The built-in data lineage function of NiFi would also benefit from the
> power of Neo4j.
> 
> Simon Bence <simonbence.dev@gmail.com> schrieb am Fr., 21. Feb. 2020, 19:00:
> 
>> Dear Community,
>> 
>> In my project, I do use relatively high number of processors and process
>> groups. The current search function on the NiFi UI has no  capabilitites to
>> narrow the results based on the group, which would make the results more
>> relevant, so I would like to propose a possible solution. Please if you
>> have any comment on this, do not hesitate to share it.
>> 
>> The general approach would be to keep the current text box and extend the
>> server side capabilities to process search query in the similar manner for
>> example the Google search behaves.This extensions I would call "filters".
>> For now I am interested in the ones I will mention below, but I think, it
>> is only a matter of small work for further extend the solution with further
>> ones.
>> 
>> In order to distinguish the filters from the rest of the search query, I
>> propose to put them at the beginning of the query and use the
>> [a-zA-Z0-9\.]{1..n}\:[a-zA-Z0-9\.]{1..n} format. For example a filter might
>> look the following: lorem:ipsum
>> 
>> Adding this, the search query should look like the following:
>> 
>> filter1:value filter2:value rest of the query
>> 
>> As for processing the filters, I suggest the following behaviour:
>> 
>> - Without filters the current behaviour should be kept
>> - Everything after the filters should be handled as the search term
>> - After the first "non filter word", anything should be considered as part
>> of the search term (meaning: to keep the text parsing simple, I would not
>> go in the direction to support filters at the end of the query, etc.)
>> - The ordering of the filters should have no effect on the result
>> - Filter duplications should be eliminated
>> - In case a filter appears multiple times in the query, the first occasion
>> will be used
>> - Unknown filters should be ignored
>> - Only adding filters will not end up with result, at least one character
>> must appear as search term
>> 
>> Suggested filters:
>> 
>> scope
>> Narrows the search based on the user's currently active process group. The
>> allowed values are: "all" and "here". All produces the current behaviour,
>> thus no filtering happens, but "here" should use the current process group
>> as "root" of the search, ignoring everything else (including parent group).
>> Note: This needs a minimal frontend change, because as I did see, currently
>> the current group is not sent with the search query.
>> 
>> group
>> Narrows the search for a given processing group, if it exists. The
>> behaviour is recursive, thus the result will include the contained groups
>> as well. If it is a non-existing group, the result list should be empty.
>> 
>> properties
>> Controls if properties values are included or not. If not provided, the
>> property values will be included. This is because in a lot of cases there
>> is a huge number of results come from property names.
>> 
>> - Valid values for inclusion: yes, true, include, 1
>> - Valid values for exclusion: no, none, false, exclude, 0
>> 
>> It is possible that the range of possible values should be limited (and not
>> being ambiguous), but I see a merit of "permissiveness" here as it is
>> simpler to remember.
>> 
>> Also some example:
>> 
>> 1.
>> scope:here properties:exclude lorem ipsum
>> This should search only in the current group (and it's children), excluding
>> properties and return with components containing the "lorem ipsum"
>> expression.
>> 
>> 2.
>> group:myGroup someQuery
>> This should result the finding of components with someQuery expression, but
>> only within the myGroup group, even if it is not the active one.
>> 
>> 3.
>> scope:all properties:include lorem
>> This should behave the same as "lorem" without filters.
>> 
>> Thanks for reading, I am interested to hear your opinion!
>> 
>> Kind regards,
>> Bence
>> 


Mime
View raw message