lucene-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Risden (Jira)" <>
Subject [jira] [Commented] (SOLR-13989) Move all hadoop related code to a contrib module
Date Tue, 03 Dec 2019 03:12:00 GMT


Kevin Risden commented on SOLR-13989:

I do think that Hadoop has some rough edges and would be good to isolate as much as possible.
Thanks [~rcmuir] for the great comments in the commit from SOLR-13986 where a lot of the pitfall
areas in Hadoop are. 

So I was looking at some of the new package/plugin management. I think Hadoop/HDFS could be
isolated into its own package/plugin. I don't know what is all involved with it yet. I think
moving to contrib could be a first step. It would be good to isolate it from dependencies
in Solr altogether. It looks like the plugins allow for separate classloader meaning different
versions of dependencies potentially. The one area that isn't in the plugin/package framework
looks like custom security manager policy - I've worked with ES on the HDFS integration and
custom security policy to open up permissions only when the HDFS integration was needed. Since
Solr hasn't had a security manager really, I don't think the package/plugin integration thought
about the security manager part yet.

Note there are 2 different Hadoop related integrations that I'm aware of in Solr. One is the
Hadoop authentication integration. This is pretty minimal as far as I know. Should be just
hadoop-common and hadoop-auth for the most part. The other is the Hadoop HDFS directory integration
which is more involved. There are a bunch of supporting classes in Solr for this (directory
impl, caches, etc). It might make sense as 2 contribs or two plugins. They don't have to be
used together and aren't really related (just happen to both use Hadoop dependencies underneath).

> Move all hadoop related code to a contrib module
> ------------------------------------------------
>                 Key: SOLR-13989
>                 URL:
>             Project: Solr
>          Issue Type: Task
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: Hadoop Integration
>            Reporter: Shalin Shekhar Mangar
>            Priority: Major
>             Fix For: master (9.0)
> Spin off from SOLR-13986:
> {quote}
> It seems really important to move or remove this hadoop shit out of the solr core: It
is really unreasonable that solr core depends on hadoop. that's gonna simply block any progress
improving its security, because solr code will get dragged down by hadoop's code.
> {quote}
> We should move all hadoop related dependencies to a separate contrib module

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message