lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcin (JIRA)" <j...@apache.org>
Subject [jira] Commented: (SOLR-1842) DataImportHandler ODBC keeps lock on the source table while optimisatising is being run...
Date Mon, 05 Apr 2010 15:09:27 GMT

    [ https://issues.apache.org/jira/browse/SOLR-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853393#action_12853393
] 

Marcin commented on SOLR-1842:
------------------------------

The data-config is quite simple:

<dataConfig>
	<dataSource type="JdbcDataSource" batchSize="-1" driver="com.mysql.jdbc.Driver" readOnly="false"
url="jdbc:mysql://host:8080/solr" user="user" password="xxxxxx"/>
	<document name="data">
        <entity name="updates" 
		query="select SQL_BUFFER_RESULT * from updates where date_added > '${dataimporter.last_index_time}'"

		transformer="DateFormatTransformer,RegexTransformer">
		
            <field column="id" />
			
            <field column="source" />
			
            <field column="user_id" />
            <field column="user_name" />
			
			<field column="body" />
			<field column="body_html" />
			
			<field column="published" />
			<field column="date_added" />
			<field column="date_added_day" sourceColName="published" dateTimeFormat="yyyy-MM-dd"
/>
			<field column="date_added_hour" sourceColName="published" dateTimeFormat="yyyy-MM-dd
HH" />

			<field column="sh_li" splitBy="::" />
			<field column="sh_dm" splitBy="::" />

			<field column="duplicate" />
			<field column="lang" />

			<field column="country" />
			<field column="town" />
			<field column="lat" />
			<field column="lng" />
			<field column="woeid" />

		</entity>
    </document>
</dataConfig>

just realize that I am using JDBC not ODBC, sorry for mistake. I am using SQL_BUFFER_RESULT
in order to avoid table locks while optimization is being done.

hope it will help a bit.

> DataImportHandler ODBC keeps lock on the source table while optimisatising is being run...
> ------------------------------------------------------------------------------------------
>
>                 Key: SOLR-1842
>                 URL: https://issues.apache.org/jira/browse/SOLR-1842
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - DataImportHandler
>    Affects Versions: 1.5
>            Reporter: Marcin
>
> Hi Guys,
> I don't know if its really a bug but I think its quite good place for it.
> The problem is with dataImportHandler and DB queries.
> For example:
> Let's have a big table which keeps docs to being indexed, we are running query against
it on a datimporthandler and query locks table which is quite obvius and desire behaviour
from the SQL points of view but while optimisation is being done its should not allow to issue
query because in that case table is being locked till optimisation process will finish which
can take a time...
> As a workaround you can use "select SQL_BUFFER_RESULT..." statment which will move everything
into temp table and release all locks but still dataImportHandlerwill be waiting for optimisation
to finish. Which means you will be able to insert new docs into main table at least.
> cheers

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message