manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Importing old configuration data
Date Tue, 24 Apr 2012 05:34:01 GMT
Oh, wait - you messed with the file, right?  Probably you changed the
order of the entries around by doing so...

The current 0.5 code outputs the right order:

            java.util.zip.ZipEntry outputEntry = new
java.util.zip.ZipEntry("outputs");
            zos.putNextEntry(outputEntry);
            outputManager.exportConfiguration(zos);
            zos.closeEntry();

            java.util.zip.ZipEntry authEntry = new
java.util.zip.ZipEntry("authorities");
            zos.putNextEntry(authEntry);
            authManager.exportConfiguration(zos);
            zos.closeEntry();

            java.util.zip.ZipEntry connEntry = new
java.util.zip.ZipEntry("connections");
            zos.putNextEntry(connEntry);
            connManager.exportConfiguration(zos);
            zos.closeEntry();

            java.util.zip.ZipEntry jobsEntry = new
java.util.zip.ZipEntry("jobs");
            zos.putNextEntry(jobsEntry);
            jobManager.exportConfiguration(zos);
            zos.closeEntry();


... so there's no *current* problem with this.  If you've messed with
the order though and still want to use the file, try this:

- unpack
- pack each individual entry in its own zip
- import them, one by one, in the order above (outputs, authorities,
connections, jobs)

I bet that will work for you...

Karl


On Mon, Apr 23, 2012 at 1:33 PM, Karl Wright <daddywri@gmail.com> wrote:
> The problem is not with the current version; it's with the ManifoldCF
> 0.3 version, which did not output things in the correct order.
> Therefore the outputs do not get processed before the jobs do, leading
> to this problem.
>
> There are two possible solutions.  The first is to hack up your copy
> of 0.5 to do TWO passes over the data file, looking for "outputs" on
> the first pass, and looking for everything on the second.  The second
> is create the "Solr" output connection by hand; running the script
> after that should populate everything.
>
> Thanks,
> Karl
>
>
> On Mon, Apr 23, 2012 at 9:25 AM, Karl Wright <daddywri@gmail.com> wrote:
>> Hmm.  Your output shows that the output connector is indeed registered:
>>
>>  ./processes/script/executecommand.sh
>> org.apache.manifoldcf.agents.RegisterOutput
>> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
>>
>> If you have no passwords in the config, you can send me the file and
>> I'll debug it to see what's going on.
>>
>> Karl
>>
>> On Mon, Apr 23, 2012 at 9:04 AM, Karl Wright <daddywri@gmail.com> wrote:
>>> Hmm.  There have been no changes to the format or meaning of the
>>> configuration file AFAIK for quite a while.
>>>
>>> This is complaining about there not being an output connection called
>>> "Solr" in place at the time it tries to create a job.  In theory, on
>>> import, the creation of connections takes place before the creation of
>>> jobs.  Is there any possibility that the connectors have not been
>>> registered?  I think they would need  to be before you attempt the
>>> import.
>>>
>>> Karl
>>>
>>> On Mon, Apr 23, 2012 at 8:57 AM, Erlend Garåsen <e.f.garasen@usit.uio.no>
wrote:
>>>>
>>>> I'm trying to import my old configuration files into my freshly installed
>>>> MCF version, but I'm afraid that these configuration data is too old (from
>>>> version 0.3, I think). I get the following error: "Database exception:
>>>> Exception doing query: ERROR: insert or update on table "jobs" violates
>>>> foreign key constraint "jobs_outputname_fkey"
>>>>  Detail: Key (outputname)=(Solr) is not present in table
>>>> "outputconnections"".
>>>>
>>>> Here's what I tried to do. The last step fails:
>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Install
>>>>  ./processes/script/executecommand.sh
>>>> org.apache.manifoldcf.agents.RegisterOutput
>>>> org.apache.manifoldcf.agents.output.solr.SolrConnector SolrConnector
>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.agents.Register
>>>> org.apache.manifoldcf.crawler.system.CrawlerAgent
>>>> ./processes/script/executecommand.sh org.apache.manifoldcf.crawler.Register
>>>> org.apache.manifoldcf.crawler.connectors.webcrawler.WebcrawlerConnector
>>>> WebCrawler
>>>> ./processes/script/executecommand.sh
>>>> org.apache.manifoldcf.crawler.ImportConfiguration export_mcf.zip
>>>>
>>>> Complete stack trace:
>>>>
>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
>>>> exception: Exception doing query: ERROR: insert or update on table "jobs"
>>>> violates foreign key constraint "jobs_outputname_fkey"
>>>>  Detail: Key (outputname)=(Solr) is not present in table
>>>> "outputconnections".
>>>>        at
>>>> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:680)
>>>>        at
>>>> org.apache.manifoldcf.core.database.Database.executeUncachedQuery(Database.java:708)
>>>>        at
>>>> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1393)
>>>>        at
>>>> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:144)
>>>>        at
>>>> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:185)
>>>>        at
>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performModification(DBInterfacePostgreSQL.java:631)
>>>>        at
>>>> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performInsert(DBInterfacePostgreSQL.java:182)
>>>>        at
>>>> org.apache.manifoldcf.core.database.BaseTable.performInsert(BaseTable.java:68)
>>>>        at org.apache.manifoldcf.crawler.jobs.Jobs.save(Jobs.java:756)
>>>>        at
>>>> org.apache.manifoldcf.crawler.jobs.JobManager.save(JobManager.java:576)
>>>>        at
>>>> org.apache.manifoldcf.crawler.jobs.JobManager.importConfiguration(JobManager.java:230)
>>>>        at
>>>> org.apache.manifoldcf.crawler.system.ManifoldCF.importConfiguration(ManifoldCF.java:1100)
>>>>        at
>>>> org.apache.manifoldcf.crawler.ImportConfiguration.doExecute(ImportConfiguration.java:40)
>>>>        at
>>>> org.apache.manifoldcf.crawler.BaseCrawlerInitializationCommand.execute(BaseCrawlerInitializationCommand.java:37)
>>>>        at
>>>> org.apache.manifoldcf.crawler.ImportConfiguration.main(ImportConfiguration.java:57)
>>>> Caused by: org.postgresql.util.PSQLException: ERROR: insert or update on
>>>> table "jobs" violates foreign key constraint "jobs_outputname_fkey"
>>>>  Detail: Key (outputname)=(Solr) is not present in table
>>>> "outputconnections".
>>>>        at
>>>> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103)
>>>>        at
>>>> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)
>>>>        at
>>>> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>>>>        at
>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512)
>>>>        at
>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
>>>>        at
>>>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
>>>>        at
>>>> org.apache.manifoldcf.core.database.Database.execute(Database.java:839)
>>>>        at
>>>> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:640)
>>>>
>>>>
>>>> --
>>>> Erlend Garåsen
>>>> Center for Information Technology Services
>>>> University of Oslo
>>>> P.O. Box 1086 Blindern, N-0317 OSLO, Norway
>>>> Ph: (+47) 22840193, Fax: (+47) 22852970, Mobile: (+47) 91380968, VIP: 31050

Mime
View raw message