lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shad Storhaug <s...@shadstorhaug.com>
Subject RE: Strong named assemblies for Lucene.Net 4.8
Date Tue, 12 Sep 2017 22:24:02 GMT
Aaron,

Thanks for your interest in Lucene.Net. Answers inline...

> Are there any plans to strong-name the Lucene.Net 4.8 assemblies in the beta Nuget packages?
I'm assuming that when the strong name is added to the Lucene.Net assembly it will use the
Lucene.Net.snk already present in the repo so we should go ahead and use that key as well?

I have no objections and understand this is a serious blocker for anyone who requires strong
naming, however since you are literally only the second person to ask for this I repeat what
I told the last person. Please, open an issue on JIRA about strong naming and encourage people
to vote on it if it is a blocker for them by sending an email to our user mailing list (or
provide any feedback why we shouldn't do it). Including information about your expected user
base would also be something for us to consider. "Because it is a blocker if someone requires
it" isn't quite enough to make a case for a feature that we are not sure anyone uses anymore,
but I'd be happy to advocate for it if we have more evidence this is still needed by some
people.

Do note that all of our dependencies are strong named AFAIK and I have setup the versioning
with strong naming in mind, so this shouldn't take a huge effort to do. 

If you are willing to help we would accept a contribution to enable strong naming *as an option*
during build (FEATURE_STRONG_NAME), which is something that is on my TODO list. And yes, we
will be using the Lucene.Net.snk in our repository for this.

> Is there some specific investigation needed that I could do?

See above.

> I'm curious though why 4.5.1 was targeted specifically for Lucene.Net since there weren't
many significant changes in that version?

This is a question that I have also asked, but the person/people who made that decision are
no longer actively involved in the project.

> Does anyone have more context on the problem? I don't recall any issues with unmanaged
references and strong names (we have both unmanaged references from C# DLLs and Managed C++
DLLs in our codebase) but might not fully understand the comment. I could look into this further
but wondering if anyone has more information before I try to start blind.

This comment is now out of date (actually, the first sentence has been removed from the codebase
since you looked). Basically, we were unable to strong name Lucene.Net.Analysis.Common previously
because of a reference to ICU4NET, which has since been changed to icu.net. ICU4NET was a
blocker for the SNK, but now it is no longer an issue. The dependency has also been consolidated
into a new project named Lucene.Net.ICU to prevent the majority of our users from having a
25 MB dependency that a small minority will actually need.

> Any timeline for the next beta release? I've seen mention recently that another beta
release is planned soon. Any timeline on this?

"Soon" is a relative term, which of course depends on how quickly we can get through all of
the tooling hurdles and bug fixes and how much help we have to do it. I just sent an email
to our dev list about some of the issues we are facing with failing tests (http://apache.markmail.org/message/3kgnc7bguo73kmbw?q=lucenenet+debugging+help+requested+for+beta),
and any help you could provide would also be welcome. 

Basically, there are 4 things I would like to include in the release that are not yet done:

1. We are currently getting random "file in use by another process" exceptions in the tests,
which I don't believe were happening in the last release. Need to confirm that these issues
were not present in the last release, find out which commit caused them to start, and decide
on a fix.
2. There is a new .NET Core 2.0 command line tool (lucene-cli) that needs to be published
and packaged in the build (ideally on Chocolatey, but just zipped and included in the Lucene.Net
distribution http://www.apache.org/dist/lucenenet/ failing that). 
3. Include API documentation generation as part of the build. See https://github.com/apache/lucenenet/pull/206.
4. Change from .NET Framework 4.5.1 to .NET Framework 4.5. This was also requested previously
and according to the requestor the 3 assemblies that would most likely be problematic all
compiled fine.

Ideally, we would also like to get all of the tests passing, too, but I am not going to hold
up the release if the ICU issues are not fixed.

If you know anything about packaging a .NET Core .DLL command line utility as a Chocolatey
package, we could use the help with that as well. It doesn't seem to be documented anywhere
how to make a package for a .NET Core .DLL (Chocolatey only considers .EXE files by default).

And if there is any strings you can pull at Microsoft to make the tooling run faster, that
would also help :). Currently it takes:

1. ~10 minutes to discover the tests in Visual Studio 2017 (in VS 2015 this was a minute or
two)
2. ~20 minutes to install the SDK/restore/build/pack on the command line (with the project.json
and preview tooling, this was about 5 minutes)
3. Since there is no way to switch target framework in VSTest, we are doing this manually
in a linked MSBuild file. But Visual Studio usually needs to be closed and reopened again
when it is changed, which means we are at least 10 minutes away from being able to run the
tests whenever it is switched. Previously, we had 2 different solution files. Now it is looking
like the most practical thing to do will be to make a solution file per assembly/test assembly
pair.

> Additionally, I came across this blog post<http://code972.com/blog/2016/07/98-lucene-net-4-8-is-in-beta-and-we-need-your-help>
which suggests that Lucene.Net 4.8 should be a good option even in its current "beta" form.

We are now quite a bit better off from when that blog post was written. I cannot comment on
whether we are more/less stable than Lucene 3.0.3, but our goal is to get it as stable as
Java Lucene before making the release official, and then work on closing the version gap.
We have a few tests that are newly failing after upgrading from project.json to .csproj, but
considering that we have nearly 8000 tests that pass I'd say we are on track to meet that
goal.

> The blog post referenced above also mentioned that a significant amount of contribution
to Lucene.Net 4.8 was done by Microsoft employees. Does anyone have names or contact info
for some of these people? I would love to reach out internally to understand these contributions
and how Lucene.Net 4.8 is being used elsewhere at Microsoft as this will help us coordinate
our plans and any potential contributions.

The only Microsoft employee that I have crossed paths with during the year or so I have been
involved with the project is Connie Yau, who primarily helped us get on .NET Standard and
helped out a lot with the International Components for Unicode (ICU) part, which still has
a bit more work left to complete. See:

1. https://github.com/sillsdev/icu-dotnet/pull/37
2. https://github.com/apache/lucenenet/tree/master/src/Lucene.Net.Analysis.ICU/Analysis/Icu


@Itamar, do you know of any others?


Thanks,
Shad Storhaug (NightOwl888)


-----Original Message-----
From: itamar.synhershko@gmail.com [mailto:itamar.synhershko@gmail.com] On Behalf Of Itamar
Syn-Hershko
Sent: Tuesday, September 12, 2017 2:37 AM
To: dev@lucenenet.apache.org
Subject: Re: Strong named assemblies for Lucene.Net 4.8

Hi Aaron,

With regards to strong naming, please see this:
http://code972.com/blog/2014/04/68-ditching-strong-naming-for-lucene-net

As Lucene.NET is an Apache project, all decisions have to go through the community and developers
so I can't really make a decision here. I do hope the reasoning above would make sense to
all involved. You can sign the assemblies yourself if needed.

Cheers,

--

Itamar Syn-Hershko
Freelance Developer & Consultant
Elasticsearch Partner
Microsoft MVP | Lucene.NET PMC
http://code972.com | @synhershko <https://twitter.com/synhershko> http://BigDataBoutique.co.il/

On Mon, Sep 11, 2017 at 10:33 PM, Aaron Meyers < Aaron.Meyers@microsoft.com.invalid>
wrote:

> tl;dr Are there any plans to strong-name the Lucene.Net 4.8 assemblies 
> in the beta Nuget packages? Is there some specific investigation 
> needed that I could do? Any timeline for the next beta release? And 
> does anyone know why
> 4.5.1 was chosen as the target framework for Lucene.Net 4.8?
>
> Quick intro: I'm an engineering lead for the language understanding 
> engine underlying the "Q&A" natural language query feature in 
> Microsoft's Power BI product (internally known as "Lucia"). We would 
> like to start using Lucene.Net to implement our index of the user's 
> database string values (replacing current internal solutions which 
> have some drawbacks). Our scenario is rather different from the 
> typical application of Lucene but still needs much of the same underlying algorithms/features.
>
> I have a working implementation over Lucene.Net 3.0.3 but recently 
> discovered that Lucene.Net 4.8 supports both store compression and 
> memory-mapped files which would be useful for our scenario. 
> Additionally, I came across this blog post<http://code972.com/blog/ 
> 2016/07/98-lucene-net-4-8-is-in-beta-and-we-need-your-help> which 
> suggests that Lucene.Net 4.8 should be a good option even in its 
> current "beta" form.
>
> As I've started working with Lucene.Net 4.8, I hit two early roadblocks:
>
>   1.  All of our assemblies are strong-named but the current 
> Lucene.Net
> 4.8 beta packages on Nuget are not strong-named
>   2.  Our codebase currently targets .NET 4.5 but Lucene.Net 4.8 
> targets
> 4.5.1
>
> For 1), I was able to build a strong-named version of Lucene.Net 4.8 
> by cloning the git repo and using the Lucene.Net.snk included in the 
> repo. I had to comment out these lines in the AssemblyInfo.cs for 
> Lucene.Net (I haven't tried to build any other assemblies yet - for 
> our usage, we'll probably only need the core Lucene.Net.dll as we'll 
> provide our own analyzers and we build queries directly):
> // LUCENENET NOTE: For now it is not possible to use a SNK because we 
> have unmanaged references in Analysis.Common.
> // However, we still need InternalsVisibleTo in order to prevent 
> making everything public just for the sake of testing.
> // This has broad implications, though because many methods are marked 
> "protected internal", which means other assemblies // must update 
> overridden methods to match.
> /*
> [assembly: InternalsVisibleTo("Lucene.Net.Tests")]
> [assembly: InternalsVisibleTo("Lucene.Net.TestFramework")]
> [assembly: InternalsVisibleTo("Lucene.Net.Highlighter")] // For 
> Automaton
> [assembly: InternalsVisibleTo("Lucene.Net.ICU")] // For Automaton
> [assembly: InternalsVisibleTo("Lucene.Net.Misc")]
> [assembly: InternalsVisibleTo("Lucene.Net.Suggest")] // For Automaton
> [assembly: InternalsVisibleTo("Lucene.Net.Tests.Analysis.Common")] // 
> For Automaton
> [assembly: InternalsVisibleTo("Lucene.Net.Tests.Highlighter")] // For 
> Automaton
> [assembly: InternalsVisibleTo("Lucene.Net.Tests.ICU")] // For 
> Analysis.Util.TestSegmentingTokenizerBase
> [assembly: InternalsVisibleTo("Lucene.Net.Tests.Misc")]
> [assembly: InternalsVisibleTo("Lucene.Net.Tests.QueryParser")]
> [assembly: InternalsVisibleTo("Lucene.Net.Tests.Cli")] // For 
> lucene-cli */
>
> I didn't see any issue in JIRA for this comment. Does anyone have more 
> context on the problem? I don't recall any issues with unmanaged 
> references and strong names (we have both unmanaged references from C# 
> DLLs and Managed C++ DLLs in our codebase) but might not fully 
> understand the comment. I could look into this further but wondering 
> if anyone has more information before I try to start blind.
>
> For 2), I also changed the target framework in the assembly I built to 
> temporarily unblock our development. We are planning to move to 4.5.2 
> target since both 4.5 and 4.5.1 haven't actually been supported since 2016.
> I'm curious though why 4.5.1 was targeted specifically for Lucene.Net 
> since there weren't many significant changes in that version? In any 
> case though this shouldn't be a blocker for us (while we'll do some 
> development on the
> 4.5 target at the moment, we're of course running 4.6 or later on our 
> development machines so I don't expect any differences between 4.5 and
> 4.5.1 target).
>
> Two other questions:
>
>   1.  I've seen mention recently that another beta release is planned 
> soon. Any timeline on this? We can ship a strong-named version of 
> Lucene.Net built ourselves but would prefer an official release if 
> possible. I'm assuming that when the strong name is added to the 
> Lucene.Net assembly it will use the Lucene.Net.snk already present in 
> the repo so we should go ahead and use that key as well?
>   2.  The blog post referenced above also mentioned that a significant 
> amount of contribution to Lucene.Net 4.8 was done by Microsoft employees.
> Does anyone have names or contact info for some of these people? I 
> would love to reach out internally to understand these contributions 
> and how Lucene.Net 4.8 is being used elsewhere at Microsoft as this 
> will help us coordinate our plans and any potential contributions.
>
> Thanks for your help and all the great work on Lucene.Net 4.8. We're 
> very excited to be able to leverage this in our product.
>
> Aaron Meyers
> Senior Software Engineer
> Microsoft Power BI
>
>
Mime
View raw message