oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John Mcgibbney <lewis.mcgibb...@gmail.com>
Subject Re: Upgrading Lucene and Solr
Date Thu, 14 Jul 2016 23:22:05 GMT
Dyno mate

On Mon, Jul 11, 2016 at 12:02 PM, Tom Barber <tom.barber@meteorite.bi>
wrote:

> [INFO] Installing /root/oodt/pom.xml to
>
> /root/.m2/repository/org/apache/oodt/oodt/1.1-SNAPSHOT/oodt-1.1-SNAPSHOT.pom
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] OODT Core .......................................... SUCCESS [
>  1.290 s]
> [INFO] Common Utilities ................................... SUCCESS [
> 15.731 s]
> [INFO] CAS Command Line Interface ......................... SUCCESS [
>  4.385 s]
> [INFO] Process Control System Input Data Package .......... SUCCESS [
>  1.186 s]
> [INFO] Catalog and Archive Service Generic Multi-valued Metadata Container
> SUCCESS [  2.357 s]
> [INFO] CAS Protocol ....................................... SUCCESS [
>  5.509 s]
> [INFO] CAS Protocol FTP Implementation .................... SUCCESS [
>  2.993 s]
> [INFO] CAS Protocol HTTP Implementation ................... SUCCESS [
> 10.829 s]
> [INFO] CAS Protocol IMAPS Implementation .................. SUCCESS [
>  3.236 s]
> [INFO] CAS Protocol SFTP Implementation ................... SUCCESS [
>  2.445 s]
> [INFO] Query Expression ................................... SUCCESS [
>  2.873 s]
> [INFO] OODT Single Sign On Security Package ............... SUCCESS [
>  1.121 s]
> [INFO] Catalog and Archive File Management Component ...... SUCCESS [03:36
> min]
> [INFO] OODT CAS Virtual Catalog and Integration Service. .. SUCCESS [
>  9.659 s]
> [INFO] Catalog and Archive Resource Management Component .. SUCCESS [
> 14.399 s]
> [INFO] Catalog and Archive Workflow Management Component .. SUCCESS [
> 34.506 s]
> [INFO] Catalog and Archive Crawling Framework ............. SUCCESS [
> 15.162 s]
> [INFO] CAS Curation Web Services .......................... SUCCESS [
> 10.538 s]
> [INFO] Process Control System Core Package ................ SUCCESS [
>  9.441 s]
> [INFO] OODT Wicket Web Components ......................... SUCCESS [
>  2.837 s]
> [INFO] CAS Curation Interface ............................. SUCCESS [
> 15.108 s]
> [INFO] CAS PGE Adaptor Framework .......................... SUCCESS [
> 10.969 s]
> [INFO] CAS Installer Maven Mojo ........................... SUCCESS [
>  1.550 s]
> [INFO] OODT :: Archetypes :: OpsUI ........................ SUCCESS [
>  0.738 s]
> [INFO] OODT :: Archetypes :: RADiX ........................ SUCCESS [
>  0.490 s]
> [INFO] OODT :: Archetypes ................................. SUCCESS [
>  0.014 s]
> [INFO] CAS Push-Pull-Framework ............................ SUCCESS [
> 10.270 s]
> [INFO] Product Service .................................... SUCCESS [
>  1.391 s]
> [INFO] Profile Service .................................... SUCCESS [
>  3.924 s]
> [INFO] OODT Web Grid ...................................... SUCCESS [
>  8.825 s]
> [INFO] XML-configured, DBMS-based Product and Profile Server SUCCESS [
>  5.605 s]
> [INFO] Apache OODT Configurable OPeNDAP Profile Server .... SUCCESS [
> 14.057 s]
> [INFO] CAS File Manager Browser Web App ................... SUCCESS [
> 12.046 s]
> [INFO] CAS Product Server Web Application ................. SUCCESS [
> 12.723 s]
> [INFO] CAS Workflow Manager Monitor Web App ............... SUCCESS [
>  4.691 s]
> [INFO] Catalog and Archive File Management Browser ........ SUCCESS [
>  5.015 s]
> [INFO] Catalog and Archive Workflow Management GUI Editor . SUCCESS [
>  7.072 s]
> [INFO] Process Control System Operator Interface Webapp ... SUCCESS [
> 10.877 s]
> [INFO] OODT Process Control System JAX-RS service layer ... SUCCESS [
>  5.047 s]
> [INFO] Apache OODT ........................................ SUCCESS [
>  0.221 s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 08:18 min
> [INFO] Finished at: 2016-07-11T19:00:29+00:00
> [INFO] Final Memory: 136M/2984M
> [INFO]
> ------------------------------------------------------------------------
> root@ubuntu:~/oodt# git status
> On branch 1.x-lucene
> Your branch is up-to-date with 'origin/1.x-lucene'.
>
>
> You should be good to go.
>
> On Mon, Jul 11, 2016 at 4:19 PM, Mattmann, Chris A (3980) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > I have a beautiful use case for BigTranslate that I can test this
> > in pretty immediately - so commit to 1.x soon and I’ll be your
> > guinea pig.
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Chief Architect
> > Instrument Software and Science Data Systems Section (398)
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 168-519, Mailstop: 168-527
> > Email: chris.a.mattmann@nasa.gov
> > WWW:  http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Director, Information Retrieval and Data Science Group (IRDS)
> > Adjunct Associate Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > WWW: http://irds.usc.edu/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On 7/11/16, 11:17 AM, "Tom Barber" <tom.barber@meteorite.bi> wrote:
> >
> > >INFO] Reactor Summary:
> > >[INFO]
> > >[INFO] OODT Core .......................................... SUCCESS [
> > >10.222 s]
> > >[INFO] Common Utilities ................................... SUCCESS [
> > >17.715 s]
> > >[INFO] CAS Command Line Interface ......................... SUCCESS [
> > > 9.428 s]
> > >[INFO] Process Control System Input Data Package .......... SUCCESS [
> > > 1.564 s]
> > >[INFO] Catalog and Archive Service Generic Multi-valued Metadata
> Container
> > >SUCCESS [  3.094 s]
> > >[INFO] CAS Protocol ....................................... SUCCESS [
> > > 9.456 s]
> > >[INFO] CAS Protocol FTP Implementation .................... SUCCESS [
> > > 4.550 s]
> > >[INFO] CAS Protocol HTTP Implementation ................... SUCCESS [
> > >16.927 s]
> > >[INFO] CAS Protocol IMAPS Implementation .................. SUCCESS [
> > > 4.857 s]
> > >[INFO] CAS Protocol SFTP Implementation ................... SUCCESS [
> > > 4.020 s]
> > >[INFO] Query Expression ................................... SUCCESS [
> > > 3.140 s]
> > >[INFO] OODT Single Sign On Security Package ............... SUCCESS [
> > > 2.458 s]
> > >[INFO] Catalog and Archive File Management Component ...... SUCCESS
> [04:32
> > >min]
> > >[INFO] OODT CAS Virtual Catalog and Integration Service. .. SUCCESS [
> > >15.820 s]
> > >[INFO] Catalog and Archive Resource Management Component .. SUCCESS [
> > >23.145 s]
> > >[INFO] Catalog and Archive Workflow Management Component .. SUCCESS [
> > >59.957 s]
> > >[INFO] Catalog and Archive Crawling Framework ............. SUCCESS [
> > >22.978 s]
> > >[INFO] CAS Curation Web Services .......................... SUCCESS [
> > > 9.949 s]
> > >[INFO] Process Control System Core Package ................ SUCCESS [
> > >13.890 s]
> > >[INFO] OODT Wicket Web Components ......................... SUCCESS [
> > > 4.355 s]
> > >[INFO] CAS Curation Interface ............................. SUCCESS [
> > > 9.994 s]
> > >[INFO] CAS PGE Adaptor Framework .......................... SUCCESS [
> > >18.437 s]
> > >[INFO] CAS Installer Maven Mojo ........................... SUCCESS [
> > > 6.322 s]
> > >[INFO] OODT :: Archetypes :: OpsUI ........................ SUCCESS [
> > > 9.322 s]
> > >[INFO] OODT :: Archetypes :: RADiX ........................ SUCCESS [
> > > 1.049 s]
> > >[INFO] OODT :: Archetypes ................................. SUCCESS [
> > > 0.014 s]
> > >[INFO] CAS Push-Pull-Framework ............................ SUCCESS [
> > >15.797 s]
> > >[INFO] Product Service .................................... SUCCESS [
> > > 1.704 s]
> > >[INFO] Profile Service .................................... SUCCESS [
> > > 7.741 s]
> > >[INFO] OODT Web Grid ...................................... SUCCESS [
> > >19.080 s]
> > >[INFO] XML-configured, DBMS-based Product and Profile Server SUCCESS [
> > >12.202 s]
> > >[INFO] Apache OODT Configurable OPeNDAP Profile Server .... SUCCESS [
> > >21.353 s]
> > >[INFO] CAS File Manager Browser Web App ................... SUCCESS [
> > > 9.061 s]
> > >[INFO] CAS Product Server Web Application ................. SUCCESS [
> > >12.943 s]
> > >[INFO] CAS Workflow Manager Monitor Web App ............... SUCCESS [
> > > 3.712 s]
> > >[INFO] Catalog and Archive File Management Browser ........ SUCCESS [
> > > 8.218 s]
> > >[INFO] Catalog and Archive Workflow Management GUI Editor . SUCCESS [
> > >10.785 s]
> > >[INFO] Process Control System Operator Interface Webapp ... SUCCESS [
> > >13.757 s]
> > >[INFO] OODT Process Control System JAX-RS service layer ... SUCCESS [
> > > 7.015 s]
> > >[INFO] Apache OODT ........................................ SUCCESS [
> > > 0.020 s]
> > >[INFO]
> > >------------------------------------------------------------------------
> > >[INFO] BUILD SUCCESS
> > >[INFO]
> > >------------------------------------------------------------------------
> > >[INFO] Total time: 11:40 min
> > >[INFO] Finished at: 2016-07-11T15:11:00+00:00
> > >[INFO] Final Memory: 167M/2913M
> > >[INFO]
> > >------------------------------------------------------------------------
> > >root@ubuntu:~/oodt#
> > >
> > >
> > >Okay I finished up the workflow changes last night and all the tests
> pass
> > >on 2.x so I'll lift those commits tonight and patch the 1.x branch.
> > >
> > >Just because the tests pass, don't expect it to be perfect, I don't know
> > >lucene I just manipulated what existed to work with the new API, someone
> > >who knows what they are doing needs to take a look, specifically at the
> > >todoc() methods in Filemanager and Workflow.
> > >
> > >Tom
> > >
> > >On Sat, Jul 9, 2016 at 7:18 AM, Mattmann, Chris A (3980) <
> > >chris.a.mattmann@jpl.nasa.gov> wrote:
> > >
> > >> Yeah we can back port the Lucene part to 1.x
> > >>
> > >> Sent from my iPhone
> > >>
> > >> > On Jul 8, 2016, at 11:15 PM, Tom Barber <tom.barber@meteorite.bi>
> > wrote:
> > >> >
> > >> > yeah there's no reason we can't diff the branch and merge it into
> 1.0
> > as
> > >> > well
> > >> >> On 9 Jul 2016 00:26, "Chris Mattmann" <mattmann@apache.org>
wrote:
> > >> >>
> > >> >> awesome!!!
> > >> >>
> > >> >> Please backport to 1.x :)
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>> On 7/8/16, 3:01 PM, "Tom Barber" <tom.barber@meteorite.bi>
wrote:
> > >> >>>
> > >> >>> There you go, running commentary over. The filemanager test
suite
> > now
> > >> >>> passes on Lucene 6, I'll fix up the rest next week.
> > >> >>>
> > >> >>> Tom
> > >> >>>
> > >> >>>> On Fri, Jul 8, 2016 at 10:31 PM, Tom Barber <
> > tom.barber@meteorite.bi>
> > >> >>> wrote:
> > >> >>>
> > >> >>>> Better than I thought actually:
> > >> >>>>
> > >> >>>> Tests in error:
> > >> >>>>  TestLuceneQueryCliAction.testClientTransTrueAndFlatProduct:78
»
> > >> >>>> CmdLineAction ...
> > >> >>>>  TestFileManagerCli.testLuceneQuery:350 » Runtime
> > >> >>>> org.apache.oodt.cas.cli.excep...
> > >> >>>>
> > >> >>>> Tests run: 221, Failures: 1, Errors: 2, Skipped: 0
> > >> >>>>
> > >> >>>> The failure is actually one Avro. So only 2 failing on
lucene 6
> in
> > the
> > >> >> FM.
> > >> >>>>
> > >> >>>> On Fri, Jul 8, 2016 at 9:48 PM, Tom Barber <
> > tom.barber@meteorite.bi>
> > >> >>>> wrote:
> > >> >>>>
> > >> >>>>> Just FYI:
> > >> >>>>>
> > >> >>>>> Doesn't work yet, but I've pushed a lucene branch
which is based
> > on
> > >> 2.0
> > >> >>>>> and upgrades lucene to 6.1.0. LuceneCatalog in filemanager
has
> > been
> > >> 75%
> > >> >>>>> (guess) converted and tests are half passing.
> > >> >>>>>
> > >> >>>>> On Wed, Jul 6, 2016 at 8:30 PM, Lewis John Mcgibbney
<
> > >> >>>>> lewis.mcgibbney@gmail.com> wrote:
> > >> >>>>>
> > >> >>>>>> ACK
> > >> >>>>>>
> > >> >>>>>> On Wed, Jul 6, 2016 at 8:55 AM, Tom Barber <
> > tom.barber@meteorite.bi
> > >> >
> > >> >>>>>> wrote:
> > >> >>>>>>
> > >> >>>>>>> Once avro_rpc is stable, I suggest we rename
the branch
> 2.0-dev
> > or
> > >> >>>>>> similar
> > >> >>>>>>> then branch the branch to uprade solr and
lucene outside of
> that
> > >> >> scope
> > >> >>>>>> so
> > >> >>>>>>> we don't destroy the semi working avro upgrades
;)
> > >> >>>>>>>
> > >> >>>>>>> On Wed, Jul 6, 2016 at 4:54 PM, Tom Barber
<
> > >> tom.barber@meteorite.bi
> > >> >>>
> > >> >>>>>>> wrote:
> > >> >>>>>>>
> > >> >>>>>>>> Coolio!
> > >> >>>>>>>>
> > >> >>>>>>>> On Wed, Jul 6, 2016 at 4:54 PM, Chris
Mattmann <
> > >> >> mattmann@apache.org>
> > >> >>>>>>>> wrote:
> > >> >>>>>>>>
> > >> >>>>>>>>> Yes I was just telling Paul R. the
other day we need to do
> > >> >>>>>>>>> this, or we’re DOA. I’ve first
hand seen the scalability
> > >> >>>>>>>>> issues with Big Translate and DRAT
using the old Lucene
> > >> >>>>>>>>> library. Let’s get it fixed in 2.x.
> > >> >>>>>>>>>
> > >> >>>>>>>>> Cheers,
> > >> >>>>>>>>> Chris
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>>>> On 7/6/16, 8:51 AM, "Tom Barber" <tom.barber@meteorite.bi>
> > >> >> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>>> Just throwing this one out there
as we're making a bunch of
> > >> >>>>>> breaking
> > >> >>>>>>>>>> changes in 2.0.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Is it worth upgrading Lucene and
Solr versions within OODT?
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> My feeling is that they are both
pretty old and whilst
> we're
> > >> >> making
> > >> >>>>>>> life
> > >> >>>>>>>>>> hard for everyone we might as
well go the whole hog and
> drag
> > >> >> them
> > >> >>>>>> into
> > >> >>>>>>>>> the
> > >> >>>>>>>>>> 21st century, but I'm interested
in others opinions.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Chris you have a much better understand
about those things
> > than
> > >> >>>>>> myself.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Tom
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>> --
> > >> >>>>>> *Lewis*
> > >> >>
> > >> >>
> > >>
> >
>



-- 
*Lewis*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message