nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Ebert <martin.irg...@gmx.de>
Subject Re: [DISCUSS] Advanced search capabilities
Date Sat, 22 Feb 2020 07:00:22 GMT
Hi Mike,
that is a fair point. You would actually raise the minimum requirements of
Nifi accordingly if you wanted to use a graph. As an additional
application, as we are currently planning, Neo4j is nevertheless a good
choice and there is nothing to be said against making it open source. The
open source version of Neo4j should be sufficient for this.


Mike Thomsen <mikerthomsen@gmail.com> schrieb am Sa., 22. Feb. 2020, 02:36:

> Martin,
>
> In theory, a graph database would be superior here. Absolutely. In
> practice, none of the tech out there is better than the current
> Lucene-based approach in terms of ease of development and integration and
> low memory footprint. Adding Neo4J or JanusGraph would cause a huge jump in
> the minimum requirements to run NiFi. Possibly to the point where Xms and
> Xmx would have to start at 2GB for people getting started.
>
> It's been a long time since I've played with Atlas and the Atlas
> integration, but if that doesn't work you can build in support for Cypher
> and Gremlin by adding -Pinclude-graph to a 1.10 or 1.11 build. In 1.10, one
> of the NARs was overlooked in that profile, so you'd need to add it back to
> the profile. That was fixed in 1.11. The ExecuteGraphQuery processor will
> allow you to execute Cypher or Gremlin commands/scripts depending on which
> controller service/driver you configure.
>
> On Fri, Feb 21, 2020 at 6:42 PM Martin Ebert <martin.irgang@gmx.de> wrote:
>
> > We still think about building a graph based search (Neo4j) in top of
> NiFi.
> > Would be also fantastic to have it within NiFi.
> >
> > There are plenty of examples
> >
> >
> https://blog.grandstack.io/using-neo4js-full-text-search-with-graphql-e3fa484de2ea
> > From the idea it could go in this direction - of course much more
> > rudimentary. Then one would have the possibility to have only the results
> > displayed as text or to find out exploratory connections (graph layout).
> > The built-in data lineage function of NiFi would also benefit from the
> > power of Neo4j.
> >
> > Simon Bence <simonbence.dev@gmail.com> schrieb am Fr., 21. Feb. 2020,
> > 19:00:
> >
> > > Dear Community,
> > >
> > > In my project, I do use relatively high number of processors and
> process
> > > groups. The current search function on the NiFi UI has no
> capabilitites
> > to
> > > narrow the results based on the group, which would make the results
> more
> > > relevant, so I would like to propose a possible solution. Please if you
> > > have any comment on this, do not hesitate to share it.
> > >
> > > The general approach would be to keep the current text box and extend
> the
> > > server side capabilities to process search query in the similar manner
> > for
> > > example the Google search behaves.This extensions I would call
> "filters".
> > > For now I am interested in the ones I will mention below, but I think,
> it
> > > is only a matter of small work for further extend the solution with
> > further
> > > ones.
> > >
> > > In order to distinguish the filters from the rest of the search query,
> I
> > > propose to put them at the beginning of the query and use the
> > > [a-zA-Z0-9\.]{1..n}\:[a-zA-Z0-9\.]{1..n} format. For example a filter
> > might
> > > look the following: lorem:ipsum
> > >
> > > Adding this, the search query should look like the following:
> > >
> > > filter1:value filter2:value rest of the query
> > >
> > > As for processing the filters, I suggest the following behaviour:
> > >
> > > - Without filters the current behaviour should be kept
> > > - Everything after the filters should be handled as the search term
> > > - After the first "non filter word", anything should be considered as
> > part
> > > of the search term (meaning: to keep the text parsing simple, I would
> not
> > > go in the direction to support filters at the end of the query, etc.)
> > > - The ordering of the filters should have no effect on the result
> > > - Filter duplications should be eliminated
> > > - In case a filter appears multiple times in the query, the first
> > occasion
> > > will be used
> > > - Unknown filters should be ignored
> > > - Only adding filters will not end up with result, at least one
> character
> > > must appear as search term
> > >
> > > Suggested filters:
> > >
> > > scope
> > > Narrows the search based on the user's currently active process group.
> > The
> > > allowed values are: "all" and "here". All produces the current
> behaviour,
> > > thus no filtering happens, but "here" should use the current process
> > group
> > > as "root" of the search, ignoring everything else (including parent
> > group).
> > > Note: This needs a minimal frontend change, because as I did see,
> > currently
> > > the current group is not sent with the search query.
> > >
> > > group
> > > Narrows the search for a given processing group, if it exists. The
> > > behaviour is recursive, thus the result will include the contained
> groups
> > > as well. If it is a non-existing group, the result list should be
> empty.
> > >
> > > properties
> > > Controls if properties values are included or not. If not provided, the
> > > property values will be included. This is because in a lot of cases
> there
> > > is a huge number of results come from property names.
> > >
> > > - Valid values for inclusion: yes, true, include, 1
> > > - Valid values for exclusion: no, none, false, exclude, 0
> > >
> > > It is possible that the range of possible values should be limited (and
> > not
> > > being ambiguous), but I see a merit of "permissiveness" here as it is
> > > simpler to remember.
> > >
> > > Also some example:
> > >
> > > 1.
> > > scope:here properties:exclude lorem ipsum
> > > This should search only in the current group (and it's children),
> > excluding
> > > properties and return with components containing the "lorem ipsum"
> > > expression.
> > >
> > > 2.
> > > group:myGroup someQuery
> > > This should result the finding of components with someQuery expression,
> > but
> > > only within the myGroup group, even if it is not the active one.
> > >
> > > 3.
> > > scope:all properties:include lorem
> > > This should behave the same as "lorem" without filters.
> > >
> > > Thanks for reading, I am interested to hear your opinion!
> > >
> > > Kind regards,
> > > Bence
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message