manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erlend Garåsen <e.f.gara...@usit.uio.no>
Subject Re: Running ManifoldCF on Resin
Date Fri, 04 Mar 2011 14:28:47 GMT

Maybe I should create the PostgreSQL patch without any more testing. I 
have some problems to run the jobs inside the application, but I doubt 
the reason is my PostgreSQL code changes since the same happens with 
Apache Derby. The status of the job says "Starting up" and nothing shows 
up in the manifoldcf.log nor in the history logs available inside the 
application.

If I select "Simple History", the list is empty, but in the activities 
list (in the upper right corner) I see that "robots parse" is the last 
activity run.

This is not something Resin specific since the same happens under Tomcat.

My contribution is clean and I have only made a few changes in the 
DBInterfacePostgreSQL and ManifoldCF class. Nothing is changed in the 
Database super class.

I haven't run "ant test-pg" since it seems to need a local Postgresql 
server running. But "ant test" reports no errors.

I have done an svn up before I created the patch in order to include the 
latest changes from trunk.

The documentation needs to be updated and I can suggest what to add later.

BTW, it should be interesting to know what causes the problems described 
above since I really want to run my jobs.

PS: I do not have access to the Eclipse 3.2 codestyle.xml file, even if 
I'm logging in, so I had to turn of formatting in Eclipse in order to 
keep the existing formatting (I get the following error when I click on 
the link to the xml file: "You are not allowed to do AttachFile on this 
page.")

Erlend



I don't think it is something PG specific either,

On 25.02.11 16.15, Karl Wright wrote:
> The DBCreate command requires privs necessary to create the database
> instance in the first place.  It uses the admin credentials that you
> supply to it as command args.  These are NOT the same credentials used
> to access the database instance, after it has been created.
>
> Thus, you can skip the DBCreate step if you have already created the
> database by hand.
>
> It does not surprise me that PostgreSQL's JDBC driver uses the global
> Java trust store.  Since there is only one database active for any
> installation, it is perfectly OK for the global trust store to be used
> for this purpose.  It will, however, probably require the addition of
> some documentation as to how to set it up properly on various
> platforms.
>
> Thanks,
> Karl
>
> On Fri, Feb 25, 2011 at 9:46 AM, Erlend Garåsen<e.f.garasen@usit.uio.no>  wrote:
>>
>> I think we need another parameter as well:
>> org.apache.manifoldcf.postgresql.port
>>
>> I was going to test my code today, but figured out that the following
>> command class needs admin privileges to the PostgreSQL database, which I
>> don't and should never have:
>> org.apache.manifoldcf.core.DBCreate
>>
>> So I skipped that part. Anyway, the DBCreate method is also supposed to work
>> with my contribution if I had the sufficient privileges.
>>
>> I went on by using the Install method class instead, but ended up with a
>> Handshake exception:
>> javax.net.ssl.SSLHandshakeException:
>> sun.security.validator.ValidatorException: PKIX path building failed:
>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>> valid certification path to requested target.
>>
>> After I added our PG server's certificate to my local keystore, things began
>> to work as it should:
>> ->  Agent tables installed
>>
>> To be continued. I have a lot of things to do at the moment, and I cannot
>> test my contribution properly yet since I'm at home with a slower bandwidth.
>>
>> To be continued. :)
>>
>> Erlend
>>
>>
>> On 04.02.11 16.30, Karl Wright wrote:
>>>
>>> Sure, feel free to create a ticket and try to come up with a patch.
>>> Since the DBInterfacePostgreSQL class is responsible for any remote
>>> communication, I would make those parameters local to the postgresql
>>> implementation.  So the parameters would be something like:
>>>
>>> org.apache.manifoldcf.postgresql.hostname
>>> org.apache.manifoldcf.postgresql.ssl
>>>
>>> or some such.  Then, you'd want to load the necessary parameters in
>>> DBInterfacePostgreSQL.java itself.  (I wouldn't bother with MySQL
>>> since that's not officially supported at this time.)
>>>
>>> The only problem I see is that the constructor architecture of
>>> Database.java may make it a challenge to load the appropriate
>>> parameters in the DBInterfacePostgreSQL constructor.  But I think you
>>> could make it work, and if not, we should probably fix this anyway to
>>> make it easier.
>>>
>>> Thanks,
>>> Karl
>>>
>>> On Fri, Feb 4, 2011 at 10:17 AM, Erlend Garåsen<e.f.garasen@usit.uio.no>
>>>   wrote:
>>>>
>>>> You're right! I didn't read the command section in the documentation
>>>> properly. ManifoldCF does also run on Resin now. The only problem with
>>>> Resin
>>>> is that the necessary resources, such as CSS, are not loaded. All the
>>>> other
>>>> errors have disappeared, which means that it should be possible to run
>>>> the
>>>> application on Resin.
>>>>
>>>> Do you think I should create a ticket about running an external
>>>> PostgreSQL
>>>> database as well? I think I know which classes to modify. I see there are
>>>> classes for MySQL as well, so perhaps the issue should be generally, i.e.
>>>> not PG specific.
>>>>
>>>> I was thinking about extending the property list like this:
>>>> org.apache.manifoldcf.database.hostname (not required, defaults to
>>>> localhost)
>>>> org.apache.manifoldcf.database.ssl (not required, defaults to false)
>>>>
>>>> For PostgreSQL I think it should be quite easy to implement this, but I'm
>>>> not sure when it comes to MySQL. Haven't touched that database system for
>>>> years.
>>>>
>>>> Erlend
>>>>
>>>>
>>>> On 04.02.11 15.42, Karl Wright wrote:
>>>>>
>>>>> Ok, have you followed the instructions for a multi-process
>>>>> installation in how-to-build-and-deploy.html?
>>>>> You will need to create the database (using DBCreate), and install the
>>>>> schema (using Install), and register the crawler agent (using
>>>>> Register).  You will also need to create the properties.xml and
>>>>> logging.ini files by hand.
>>>>>
>>>>> Let me know if this helps.
>>>>> Karl
>>>>>
>>>>> On Fri, Feb 4, 2011 at 9:35 AM, Erlend Garåsen<e.f.garasen@usit.uio.no>
>>>>>   wrote:
>>>>>>
>>>>>> I guess the problem is that the database does not exist. When I
>>>>>> installed
>>>>>> it
>>>>>> on Tomcat, I got an error message telling me that the schema TEST
does
>>>>>> not
>>>>>> exist. I tried to build the database with the username and password
>>>>>> "test"
>>>>>> by the following command:
>>>>>> ./processes/script/executecommand.sh
>>>>>> org.apache.manifoldcf.core.DBCreate
>>>>>> test test
>>>>>>
>>>>>> And I also configured these two lines in my properties.xml file:
>>>>>>   <property name="org.apache.manifoldcf.database.username"
>>>>>> value="test"/>
>>>>>>   <property name="org.apache.manifoldcf.database.password"
>>>>>> value="test"/>
>>>>>>
>>>>>> I'm using the Derby database since I cannot see how I could contact
an
>>>>>> external PostgreSQL server on another host. PG is one of several
>>>>>> database
>>>>>> systems the university supports. I guess there should be an option
for
>>>>>> configuring external hosts and optionally turn on SSL encryption.
Maybe
>>>>>> another ticket to create.
>>>>>>
>>>>>> But first I want to run ManifoldCF on Tomcat using Derby, then deploy
>>>>>> it
>>>>>> on
>>>>>> Resin and then try to configure an external PostgreSQL server.
>>>>>>
>>>>>> Erlend
>>>>>>
>>>>>>
>>>>>> On 03.02.11 16.32, Karl Wright wrote:
>>>>>>>
>>>>>>> Sure, if you want, create a ticket, and see if you can figure
out what
>>>>>>> the problems are.  I'd start with not being able to reach the
>>>>>>> stylesheet, actually.  The "redirection" error sound like the
crawler
>>>>>>> UI is trying to present an error and resin is handling one of
the java
>>>>>>> standard tags in a non-standard way.
>>>>>>>
>>>>>>> Karl
>>>>>>>
>>>>>>> On Thu, Feb 3, 2011 at 10:28 AM, Erlend
>>>>>>> Garåsen<e.f.garasen@usit.uio.no>
>>>>>>>   wrote:
>>>>>>>>
>>>>>>>> Resin is de facto application server at the university, but
we also
>>>>>>>> support
>>>>>>>> Tomcat.
>>>>>>>>
>>>>>>>> I can try to deploy the war files on Tomcat instead. The
error
>>>>>>>> message
>>>>>>>> comes
>>>>>>>> from Resin, probably caused by a redirect after the http
header is
>>>>>>>> written.
>>>>>>>> I can notify you, eventually create a patch, if I manage
to run
>>>>>>>> ManifoldCF
>>>>>>>> on Resin.
>>>>>>>>
>>>>>>>> Erlend
>>>>>>>>
>>>>>>>> On 03.02.11 16.16, Karl Wright wrote:
>>>>>>>>>
>>>>>>>>> I have never tried this on Resin, just on Tomcat and
Jetty.  Clearly
>>>>>>>>> Resin as an application server is having difficulty doing
some very
>>>>>>>>> basic stuff.
>>>>>>>>>
>>>>>>>>> FWIW, when we first started the Jetty development, I
had to do
>>>>>>>>> nothing
>>>>>>>>> whatsoever to the crawler UI or the war packaging to
make it work.
>>>>>>>>>   It
>>>>>>>>> just worked.  Your experiences with Resin are making
me wonder if
>>>>>>>>> Resin is indeed working up to Java standards.
>>>>>>>>>
>>>>>>>>> Karl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Feb 3, 2011 at 9:31 AM, Erlend
>>>>>>>>> Garåsen<e.f.garasen@usit.uio.no>
>>>>>>>>>   wrote:
>>>>>>>>>>
>>>>>>>>>> I have placed the three war files into Resin's webapps
directory
>>>>>>>>>> and
>>>>>>>>>> set
>>>>>>>>>> the
>>>>>>>>>> following system variable inside resin.xml:
>>>>>>>>>>
>>>>>>>>>> <system-property
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.manifoldcf.configfile="/Users/erlendfg/apps/manifoldcf/properties.xml"
>>>>>>>>>> />
>>>>>>>>>>
>>>>>>>>>> I have also created my own home directory for ManifoldCF
and copied
>>>>>>>>>> the
>>>>>>>>>> necessary files from the example directory. When
I start Resin and
>>>>>>>>>> goes
>>>>>>>>>> to
>>>>>>>>>> ... mcf-crawler-ui/, the start page shows up, but
the page looks
>>>>>>>>>> ugly
>>>>>>>>>> (I
>>>>>>>>>> guess the path to CSS files etc. are broken). And
when I try to
>>>>>>>>>> click
>>>>>>>>>> on
>>>>>>>>>> a
>>>>>>>>>> link, e.g. "List Output Connections", I get the following
error:
>>>>>>>>>> java.lang.IllegalStateException: can't forward after
writing HTTP
>>>>>>>>>> headers
>>>>>>>>>>
>>>>>>>>>> I find nothing in manifoldcf.log.
>>>>>>>>>>
>>>>>>>>>> BTW: PostgreSQL is not configured. I will start using
Derby until
>>>>>>>>>> it
>>>>>>>>>> runs
>>>>>>>>>> stable on Resin.
>>>>>>>>>>
>>>>>>>>>> E
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Erlend Garåsen
>>>>>>>>>> Center for Information Technology Services
>>>>>>>>>> University of Oslo
>>>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile:
(+47) 91380968,
>>>>>>>>>> VIP:
>>>>>>>>>> 31050
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Erlend Garåsen
>>>>>>>> Center for Information Technology Services
>>>>>>>> University of Oslo
>>>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
VIP:
>>>>>>>> 31050
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Erlend Garåsen
>>>>>> Center for Information Technology Services
>>>>>> University of Oslo
>>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968,
VIP:
>>>>>> 31050
>>>>>>
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
>>>> 31050
>>>>
>>
>>
>> --
>> Erlend Garåsen
>> Center for Information Technology Services
>> University of Oslo
>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050
>>


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Mime
View raw message