cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuki Morishita <mor.y...@gmail.com>
Subject Re: Stream sstables hosted on a node from client using streaming protocol
Date Mon, 11 May 2015 14:04:41 GMT
Yeah, at least you need to set up Schema for loading SSTable.
If you find anything that we can improve, please file JIRA.

On Sun, May 10, 2015 at 5:27 AM, Pierre Devops <pierredevops@gmail.com> wrote:
> OK so I know a little more now, it's not doable in client mode ATM because
> it rely to much on server side stuff.
>
> It needs to initialize ColumnFamilyStore and use an instance of it
> afterwards, which will require to much server-side configuration
> initialization.
>
> Secondly the way it streams is inefficient because it will deserialize the
> streamed sstable to rebuild a new sstable in SSTableWriter.appendFromStream
> (needed to rebuild index & other compoment)  while I just need to copy the
> -Data- file on the disk.
>
> So I think I'm going to provide my own IncomingFileMessage and its own
> deserializer.
>
>
>
> 2015-05-09 23:32 GMT+02:00 Pierre Devops <pierredevops@gmail.com>:
>
>> Thanks yuki, copying SSLTableLoader was the first thing I try, but without
>> success.
>>
>> I checked BulkLoadConnectionFactory (
>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/tools/BulkLoadConnectionFactory.java)
>> and I don't see what it provide over the DefaultConnectionFactory that can
>> help me more in this case.
>>
>> Without setting up a custom connection factory, it manages already to
>> connect to the node, and send a streaming request (I see it in cassandra
>> logs).
>>
>> INFO  21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899 ID#0]
>>> Creating new streaming plan for SST Import
>>> INFO  21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899, ID#0]
>>> Received streaming plan for SST Import
>>> INFO  21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899, ID#0]
>>> Received streaming plan for SST Import
>>> INFO  21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899 ID#0]
>>> Prepare completed. Receiving 0 files(0 bytes), sending 2 files(4083518
>>> bytes)
>>> INFO  21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899] Session
>>> with /127.0.0.1 is complete
>>> WARN  21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899] Stream
>>> failed
>>> ERROR 21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899] Streaming
>>> error occurred
>>
>>
>>
>> So it looks like my client is receiving two message in its
>> ConnectionHandler loop (
>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/streaming/ConnectionHandler.java#L251)
>> , the first one is a PREPARE_MESSAGE type with a StreamSummary indicating
>> the good number of files.
>>
>> But the second message it receives, it fails to deserialize. So I debugged
>> and streamed what was coming from this socket, and it was the sstables. but
>> I don't know why it fails deseriliazion of message type.
>>
>>



-- 
Yuki Morishita
 t:yukim (http://twitter.com/yukim)

Mime
View raw message