lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Khludnev (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-6110) DIH increase tcp connections so full-import failed
Date Thu, 22 May 2014 19:51:02 GMT

    [ https://issues.apache.org/jira/browse/SOLR-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14006340#comment-14006340
] 

Mikhail Khludnev edited comment on SOLR-6110 at 5/22/14 7:50 PM:
-----------------------------------------------------------------

# -I can't imagine if anything is changed in DIH recently, it seems really strange to me-
# -for me it seems like it open and closes jdbc connections, it's also strange and can happen
only if you use something like- ... WHERE P_ID={p.ID}
# -giving all of this I propose to configure jdbc datasource in your container (luckily solr
is j2ee...), and refer to it via jndi as it described an DIH wiki-
# -it doesn't seem like a jira, please close it and post the question to the mailing list-

oh'my.. you never know..


was (Author: mkhludnev):
# I can't imagine if anything is changed in DIH recently, it seems really strange to me
# for me it seems like it open and closes jdbc connections, it's also strange and can happen
only if you use something like ... WHERE P_ID={p.ID}
# giving all of this I propose to configure jdbc datasource in your container (luckily solr
is j2ee...), and refer to it via jndi as it described an DIH wiki
# it doesn't seem like a jira, please close it and post the question to the mailing list

> DIH increase tcp connections so full-import failed
> --------------------------------------------------
>
>                 Key: SOLR-6110
>                 URL: https://issues.apache.org/jira/browse/SOLR-6110
>             Project: Solr
>          Issue Type: Bug
>          Components: contrib - DataImportHandler
>    Affects Versions: 4.7.2
>         Environment: Debian Lenny
>            Reporter: Gambier
>            Priority: Blocker
>
> Before i used solr 4.3.1 and i had no problem. Since i use solr 4.7.2 i have a problem
with the DIH.
> When i do a full import with DIH the number of TCP connection increase.
> For example i have 240000 documents to index and the full-import stop randomly between
6000 and 9000 docs indexed with the error :
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
> With the command (while full-import is running) :
> netstat | grep localdo
> I saw that there are many connections in status "TIME_WAIT"
> The problem is that the OS don't free those connections quickly. So the tcp connections
queue increase to its limit and crash ---> full-import failed.
> To solve this problem i found a solution :
> If i change OS settings "/proc/sys/net/ipv4/tcp_tw_reuse" to 1 and "/proc/sys/net/ipv4/tcp_tw_recycle"
to 1 my full-import works to the end.
> So why connections are not closed in this new version ?
> And why the java socket option SO_REUSEADDR is not used ?
> I'm not a Java developer so maybe i miss something but the only way for me to do a success
full-import was to use the solution above.
> Thank you by advance for your answers to help me solve this problem with a better solution
than mine.
> Thomas



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message