manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: MCF / Postgres Issues
Date Mon, 29 Jun 2015 17:51:08 GMT
The configuration looks standard.

If postgresql is running on a unix machine, can you check whether there are
any disk errors?  It sure sounds like there may be.

Karl


On Mon, Jun 29, 2015 at 1:37 PM, Joyce, Colin <Colin.Joyce@fmr.com> wrote:

> Hi Guys,
>
>
>
> We are currently having some MCF / postgres issues and I was hoping to get
> some help.
>
>
>
> *Configuration:*
>
>
>
> MCF 2.1
>
> Postgres 8.4
>
>
>
> We have followed the postgres configuration best practices e.g. turned off
> autovaccum etc.
>
>
>
> We are carrying out db maintenance via cron
>
>
>
> -          Lazy vaccum once a day
>
> -          Full vacumm once a week
>
> -          DB reindex once a week
>
>
>
> *Jobs:*
>
>
>
> -          1 continuous job that manages 60,000+ docs
>
> -          3 small stand-alone jobs
>
>
>
> *Issue:*
>
>
>
> We are randomly getting the below error – in one case the whole db has
> become corrupted.
>
>
>
> *MCF Error:*
>
>
>
> 2015-06-27 07:12:04,352  THREAD="Seeding thread" SEVERITY=INFO
> logger=org.apache.manifoldcf.crawler.jobs.JobManager.noteJobSeeded(JobManager.java:6508)
> MESSAGE="Job 1435140141926 has been successfully reseeded"
>
> 2015-06-27 07:38:04,433  THREAD="Seeding thread" SEVERITY=INFO
> logger=org.apache.manifoldcf.crawler.jobs.JobManager.getJobsReadyForSeeding(JobManager.java:7051)
> MESSAGE="Marked job 1435140141926 for seeding"
>
> 2015-06-27 07:42:32,876  THREAD="Seeding thread" SEVERITY=INFO
> logger=org.apache.manifoldcf.crawler.jobs.JobManager.noteJobSeeded(JobManager.java:6508)
> MESSAGE="Job 1435140141926 has been successfully reseeded"
>
> 2015-06-27 07:45:30,176  THREAD="Assessment thread" SEVERITY=ERROR
> logger=org.apache.manifoldcf.crawler.system.AssessmentThread.run(AssessmentThread.java:77)
> MESSAGE="Assessment thread aborting and restarting due to database
> connection reset: Database exception: SQLException doing query (08006): An
> I/O error occured while sending to the backend."
>
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: SQLException doing query (08006): An I/O error occured while
> sending to the backend.
>
>                 at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:702)
>
>                 at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:728)
>
>                 at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.startATransaction(DBInterfacePostgreSQL.java:1259)
>
>                 at
> org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:262)
>
>                 at
> org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:242)
>
>                 at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1432)
>
>                 at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)
>
>                 at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:191)
>
>                 at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:835)
>
>                 at
> org.apache.manifoldcf.core.database.BaseTable.performQuery(BaseTable.java:221)
>
>                 at
> org.apache.manifoldcf.crawler.jobs.Jobs.assessMarkedJobs(Jobs.java:2052)
>
>                 at
> org.apache.manifoldcf.crawler.jobs.JobManager.assessMarkedJobs(JobManager.java:802)
>
>                 at
> org.apache.manifoldcf.crawler.system.AssessmentThread.run(AssessmentThread.java:65)
>
> Caused by: org.postgresql.util.PSQLException: An I/O error occured while
> sending to the backend.
>
>                 at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:283)
>
>                 at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
>
>                 at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
>
>                 at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:366)
>
>                 at
> org.apache.manifoldcf.core.database.Database.execute(Database.java:863)
>
>                 at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:683)
>
> Caused by: java.io.EOFException
>
>                 at
> org.postgresql.core.PGStream.ReceiveChar(PGStream.java:276)
>
>                 at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
>
>                 at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>
>                 ... 5 more
>
> 2015-06-27 07:45:33,453  THREAD="Set priority thread" SEVERITY=ERROR
> logger=org.apache.manifoldcf.crawler.system.SetPriorityThread.run(SetPriorityThread.java:161)
> MESSAGE="Set priority thread aborting and restarting due to database
> connection reset: Database exception: SQLException doing query (08006): An
> I/O error occured while sending to the backend."
>
> org.apache.manifoldcf.core.interfaces.ManifoldCFException: Database
> exception: SQLException doing query (08006): An I/O error occured while
> sending to the backend.
>
>                 at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.finishUp(Database.java:702)
>
>                 at
> org.apache.manifoldcf.core.database.Database.executeViaThread(Database.java:728)
>
>                 at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.startATransaction(DBInterfacePostgreSQL.java:1259)
>
>                 at
> org.apache.manifoldcf.core.database.Database.internalTransactionBegin(Database.java:262)
>
>                 at
> org.apache.manifoldcf.core.database.Database.synchronizeTransactions(Database.java:242)
>
>                 at
> org.apache.manifoldcf.core.database.Database$QueryCacheExecutor.create(Database.java:1432)
>
>                 at
> org.apache.manifoldcf.core.cachemanager.CacheManager.findObjectsAndExecute(CacheManager.java:146)
>
>                 at
> org.apache.manifoldcf.core.database.Database.executeQuery(Database.java:191)
>
>                 at
> org.apache.manifoldcf.core.database.DBInterfacePostgreSQL.performQuery(DBInterfacePostgreSQL.java:835)
>
>                 at
> org.apache.manifoldcf.crawler.jobs.JobManager.getNextNotYetProcessedReprioritizationDocuments(JobManager.java:2249)
>
>                 at
> org.apache.manifoldcf.crawler.system.SetPriorityThread.run(SetPriorityThread.java:138)
>
> Caused by: org.postgresql.util.PSQLException: An I/O error occured while
> sending to the backend.
>
>                 at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:283)
>
>                 at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
>
>                 at
> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
>
>                 at
> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:366)
>
>                 at
> org.apache.manifoldcf.core.database.Database.execute(Database.java:863)
>
>                 at
> org.apache.manifoldcf.core.database.Database$ExecuteQueryThread.run(Database.java:683)
>
> Caused by: java.io.EOFException
>
>                 at
> org.postgresql.core.PGStream.ReceiveChar(PGStream.java:276)
>
>                 at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1660)
>
>                 at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
>
>                 ... 5 more
>
> 2015-06-28 15:44:21,728 THREAD="tomcat-http--46" SEVERITY=INFO
> logger=org.apache.manifoldcf.ui.beans.AdminProfile
>
> The job hung.
>
>
>
>
>
> *Corresponding Postgres log:*
>
>
>
>
>
> ERROR:  could not write block 11569 of relation base/27977/28241:
> Input/output error
>
> CONTEXT:  writing block 11569 of relation base/27977/28241
>
> ERROR:  could not write block 3224 of relation base/27977/28241:
> Input/output error
>
> CONTEXT:  writing block 3224 of relation base/27977/28241
>
> CONTEXT:  writing block 3256 of relation base/27977/28241
>
> ERROR:  could not close file "base/27977/28161": Input/output error
>
> PANIC:  could not fdatasync log file 3, segment 157: Input/output error
>
> STATEMENT:  COMMIT
>
> LOG:  server process (PID 25059) was terminated by signal 6: Aborted
>
> LOG:  terminating any other active server processes
>
> WARNING:  terminating connection because of crash of another server process
>
> DETAIL:  The postmaster has commanded this server process to roll back the
> current transaction and exit, because another server process exited
> abnormally and possibly corrupted shared memory.
>
> HINT:  In a moment you should be able to reconnect to the database and
> repeat your command.
>
> WARNING:  terminating connection because of crash of another server process
>
> DETAIL:  The postmaster has commanded this server process to roll back the
> current transaction and exit, because another server process exited
> abnormally and possibly corrupted shared memory.
>
> HINT:  In a moment you should be able to reconnect to the database and
> repeat your command.
>
> LOG:  all server processes terminated; reinitializing
>
> LOG:  database system was interrupted; last known up at 2015-06-27
> 07:30:42 EDT
>
> LOG:  database system was not properly shut down; automatic recovery in
> progress
>
> LOG:  redo starts at 3/9AAADB60
>
> LOG:  record with zero length at 3/9DAD22E0
>
> LOG:  redo done at 3/9DAD22B0
>
> LOG:  last completed transaction was at log time 2015-06-27
> 07:45:18.768223-04
>
> LOG:  database system is ready to accept connections
>
> FATAL:  database "template0" is not currently accepting connections
>
> ERROR:  could not fsync segment 0 of relation base/27977/28019:
> Input/output error
>
>
>
> Has anybody had similar issues? Or does anything stand out from our
> configuration
>
>
>
> Thanks,
>
>
>
> Colin
>
>
>
>
>
>
>

Mime
View raw message