cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuki Morishita (JIRA)" <>
Subject [jira] [Updated] (CASSANDRA-3668) Performance of sstableloader is affected in 1.0.x
Date Fri, 27 Apr 2012 21:26:52 GMT


Yuki Morishita updated CASSANDRA-3668:

    Attachment: 3668-1.1.txt

I rebased and attached patch. (also pushed to

bq. I don't think the 'retries' field should be moved from StreamInSession to IncomingStreamReader.
The ISR is attached to the streaming of one file, so when the node on the other side does
a retry, a new ISR will be created (while the StreamInSession persists until closed), and
so retries field will be reseted and we will retry infinitely. But it probably does mean it'll
make more sense to have one retry counter per-file (changing StreamInSession.currentFiles
to a Map<String, Integer> maybe?)

I modified to have per session retries, since we are closing session after retries exceed
the limit.

bq. Currently, every messages related to the streaming goes through the ISR socket except
for the message of session failure (in StreamInSession.closeInternal).


bq. StreamOutSession.begin() shouldn't assume the queue has at least one element (at least
the current code don't expect it)


bq. MessagingService: Looking at ThreadPoolExecutor code it seems that this is harmless but
bumping the executor core pool size would make it exceed the max pool size.

Agree. I changed max pool size to Integer.MAX_VALUE. I also changed the name of variable streamingThreadsPerNode
to streamExecutorCoreSize and set its default value to 0, in order to preserve current behavior.

bq. IncomingTcpConnection: sets the name when starting streaming but don't unset it. Probably
not worth changing the name, the thread name is probably not the best place anyway to log
activity (but it's a good idea to set a meaningful name in the first place).

For now I reverted this change. I just felt weird to see the thread name of IncomingTCPConn
as default (like 'Thread-10') on the other hand OutboundTCPConn gets name like 'WRITE-'.

bq. In FileStreamTask.runMayThrow, instead of logging an error, we could wrap the IOException
in a new one with the message. That way we wouldn't have 2 messages in the log for the same

You are right. Fixed.

bq. Would feel more coherent to me to move the sending of the SESSION_FINISHED message in
ISR (for instance SIS.closeIfFinished could return a boolean if the session is indeed finished
(and we would just have to pass the file name to this method instead of the ISR instance)).

Moved to ISR.

bq. Not this patch fault but in SIS.getIncomingFiles(), adding the currentFiles is useless

This was actually necessary because pending files in 'files' field do not get updated. But
having separate field for currently streaming files seems redundant, so in this version I
modified to replace pending file in 'files' with actually streaming one.

bq. In StreamOutSession.begin() there is a commented debug line (could be removed)

> Performance of sstableloader is affected in 1.0.x
> -------------------------------------------------
>                 Key: CASSANDRA-3668
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>    Affects Versions: 1.0.0
>            Reporter: Manish Zope
>            Assignee: Yuki Morishita
>            Priority: Minor
>              Labels: streaming
>             Fix For: 1.1.1
>         Attachments: 3668-1.1.txt, 3688-reply_before_closing_writer.txt, sstable-loader
>   Original Estimate: 48h
>  Remaining Estimate: 48h
> One of my colleague had reported the bug regarding the degraded performance of the sstable
generator and sstable loader.
> ISSUE :- 
> As stated in above issue generator performance is rectified but performance of the sstableloader
is still an issue.
> 3589 is marked as duplicate of 3618.Both issues shows resolved status.But the problem
with sstableloader still exists.
> So opening other issue so that sstbleloader problem should not go unnoticed.
> FYI : We have tested the generator part with the patch given in 3589.Its Working fine.
> Please let us know if you guys require further inputs from our side.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message