lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-8575) Fix HDFSLogReader replay status numbers and a performance bug where we can reopen FSDataInputStream too often.
Date Wed, 20 Jan 2016 23:05:39 GMT

    [ https://issues.apache.org/jira/browse/SOLR-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15109686#comment-15109686
] 

Mark Miller edited comment on SOLR-8575 at 1/20/16 11:04 PM:
-------------------------------------------------------------

bq. Using an internally stored length is probably better anyway, though.

The problem is that our internal size does not correlate with what we can actually read, even
after an hflush. (unless we reopen inputstreams)

bq. updated size from the file then you have to use hsync with the length update flag[1] using
an HdfsOutputStream 

Ah, interesting, I'll poke around that a bit to see if we want to do anything different.


was (Author: markrmiller@gmail.com):
bq. Using an internally stored length is probably better anyway, though.

The problem is that our internal size does not correlate with what we can actually read, even
after an hflush.

bq. updated size from the file then you have to use hsync with the length update flag[1] using
an HdfsOutputStream 

Ah, interesting, I'll poke around that a bit to see if we want to do anything different.

> Fix HDFSLogReader replay status numbers and a performance bug where we can reopen FSDataInputStream
too often.
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-8575
>                 URL: https://issues.apache.org/jira/browse/SOLR-8575
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Mark Miller
>            Assignee: Mark Miller
>         Attachments: SOLR-8575.patch
>
>
> [~pdvorak@cloudera.com] noticed some funny transaction log replay status logging a while
back:
> active=true starting pos=444978 current pos=2855956 current size=16262 % read=17562
> active=true starting pos=444978 current pos=5748869 current size=16262 % read=35352
> 17562% read? Current size does not change as expected in this case?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message