manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erlend Garåsen <e.f.gara...@usit.uio.no>
Subject Re: Importing old configuration data
Date Tue, 24 Apr 2012 07:59:12 GMT

Thanks a lot, Karl. Yes, I messed with the file (unzipped, changed a 
password and zipped it again). I did not think the order of the content 
was relevant, but I was wrong. Here is the order after I changed it:

erlendfg$ jar tf crawler_solr-test02_config.zip

Which outputs:
authorities
connections
jobs
outputs

Thanks for your helo!

Erlend

On 24.04.12 07.34, Karl Wright wrote:
> Oh, wait - you messed with the file, right?  Probably you changed the
> order of the entries around by doing so...
>
> The current 0.5 code outputs the right order:
>
>              java.util.zip.ZipEntry outputEntry = new
> java.util.zip.ZipEntry("outputs");
>              zos.putNextEntry(outputEntry);
>              outputManager.exportConfiguration(zos);
>              zos.closeEntry();
>
>              java.util.zip.ZipEntry authEntry = new
> java.util.zip.ZipEntry("authorities");
>              zos.putNextEntry(authEntry);
>              authManager.exportConfiguration(zos);
>              zos.closeEntry();
>
>              java.util.zip.ZipEntry connEntry = new
> java.util.zip.ZipEntry("connections");
>              zos.putNextEntry(connEntry);
>              connManager.exportConfiguration(zos);
>              zos.closeEntry();
>
>              java.util.zip.ZipEntry jobsEntry = new
> java.util.zip.ZipEntry("jobs");
>              zos.putNextEntry(jobsEntry);
>              jobManager.exportConfiguration(zos);
>              zos.closeEntry();
>
>
> ... so there's no *current* problem with this.  If you've messed with
> the order though and still want to use the file, try this:
>
> - unpack
> - pack each individual entry in its own zip
> - import them, one by one, in the order above (outputs, authorities,
> connections, jobs)
>
> I bet that will work for you...
>
> Karl
>
>
> On Mon, Apr 23, 2012 at 1:33 PM, Karl Wright<daddywri@gmail.com>  wrote:
>> The problem is not with the current version; it's with the ManifoldCF
>> 0.3 version, which did not output things in the correct order.
>> Therefore the outputs do not get processed before the jobs do, leading
>> to this problem.
>>
>> There are two possible solutions.  The first is to hack up your copy
>> of 0.5 to do TWO passes over the data file, looking for "outputs" on
>> the first pass, and looking for everything on the second.  The second
>> is create the "Solr" output connection by hand; running the script
>> after that should populate everything.
>>
>> Thanks,
>> Karl
>>
>>
>> On Mon, Apr 23, 2012 at 9:25 AM, Karl Wright<daddywri@gmail.com>  wrote:
>>> Hmm.  Your output shows that the output connector is indeed registered:
>>>
>>>   ./processes/script/executecommand.sh
>>> org.apache.manifoldcf.agents.RegisterOutput
>>> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
>>>
>>> If you have no passwords in the config, you can send me the file and
>>> I'll debug it to see what's going on.
>>>
>>> Karl
>>>
>>> On Mon, Apr 23, 2012 at 9:04 AM, Karl Wright<daddywri@gmail.com>  wrote:
>>>> Hmm.  There have been no changes to the format or meaning of the
>>>> configuration file AFAIK for quite a while.
>>>>
>>>> This is complaining about there not being an output connection called
>>>> "Solr" in place at the time it tries to create a job.  In theory, on
>>>> import, the creation of connections takes place before the creation of
>>>> jobs.  Is there any possibility that the connectors have not been
>>>> registered?  I think they would need  to be before you attempt the
>>>> import.
>>>>
>>>> Karl
>>>>
>>>> On Mon, Apr 23, 2012 at 8:57 AM, Erlend Garåsen<e.f.garasen@usit.uio.no>
 wrote:
>>>>>
>>>>> I'm trying to import my old configuration files into my freshly installed
>>>>> MCF version, but I'm afraid that these configuration data is too old
(from
>>>>> version 0.3, I think). I get the following error: "Database exception:
>>>>> Exception doing query: ERROR: insert or update on table "jobs" violates
>>>>> foreign key constraint "jobs_outputname_fkey"
>>>>>   Detail: Key (outputname)=(Solr) is not present in table
>>>>> "outputconnections"".
>>>>>
>>>>> Here's what I tried to do. The last step fails:
>>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Install
>>>>>   ./processes/script/executecommand.sh
>>>>> org.apache.manifoldcf.agents.RegisterOutput
>>>>> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
>>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Register
>>>>> org.apache.manifoldcf.crawler.system.CrawlerAgent
>>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.crawler.Register
>>>>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector
>>>>> WebCrawler
>>>>> ./processes/script/executecommand.sh
>>>>> org.apache.manifoldcf.crawler.ImportConfiguration export_mcf.zip
>>>>>
>>>>> Complete stack trace:
>>>>>
>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
>>>>> exception: Exception doing query: ERROR: insert or update on table "jobs"
>>>>> violates foreign key constraint "jobs_outputname_fkey"
>>>>>   Detail: Key (outputname)=(Solr) is not present in table
>>>>> "outputconnections".
>>>>>         at
>>>>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:680)
>>>>>         at
>>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:708)
>>>>>         at
>>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1393)
>>>>>         at
>>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>>>>         at
>>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:185)
>>>>>         at
>>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>>>>>         at
>>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>>>>>         at
>>>>> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>>>>>         at org.apache.manifoldcf.crawler.jobs.Jobs.save(Jobs.java:756)
>>>>>         at
>>>>> org.apache.manifoldcf.crawler.jobs.JobManager.save(JobManager.java:576)
>>>>>         at
>>>>> org.apache.manifoldcf.crawler.jobs.JobManager.importConfiguration(JobManager.java:230)
>>>>>         at
>>>>> org.apache.manifoldcf.crawler.system.ManifoldCF.importConfiguration(ManifoldCF.java:1100)
>>>>>         at
>>>>> org.apache.manifoldcf.crawler.ImportConfiguration.doExecute(ImportConfiguration.java:40)
>>>>>         at
>>>>> org.apache.manifoldcf.crawler.BaseCrawlerInitializationCommand.execute(BaseCrawlerInitializationCommand.java:37)
>>>>>         at
>>>>> org.apache.manifoldcf.crawler.ImportConfiguration.main(ImportConfiguration.java:57)
>>>>> Caused by: org.postgresql.util.PSQLException: ERROR: insert or update
on
>>>>> table "jobs" violates foreign key constraint "jobs_outputname_fkey"
>>>>>   Detail: Key (outputname)=(Solr) is not present in table
>>>>> "outputconnections".
>>>>>         at
>>>>> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103)
>>>>>         at
>>>>> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)
>>>>>         at
>>>>> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>>>>>         at
>>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512)
>>>>>         at
>>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
>>>>>         at
>>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
>>>>>         at
>>>>> org.apache.manifoldcf.core.database.Database.execute(Database.java:839)
>>>>>         at
>>>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:640)
>>>>>
>>>>>
>>>>> --
>>>>> Erlend Garåsen
>>>>> Center for Information Technology Services
>>>>> University of Oslo
>>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP:
31050


-- 
Erlend Garåsen
Center for Information Technology Services
University of Oslo
P.O. Box 1086 Blindern, N-0317 OSLO, Norway
Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Mime
View raw message