cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre Devops <pierredev...@gmail.com>
Subject Re: Stream sstables hosted on a node from client using streaming protocol
Date Sun, 10 May 2015 10:27:29 GMT
OK so I know a little more now, it's not doable in client mode ATM because
it rely to much on server side stuff.

It needs to initialize ColumnFamilyStore and use an instance of it
afterwards, which will require to much server-side configuration
initialization.

Secondly the way it streams is inefficient because it will deserialize the
streamed sstable to rebuild a new sstable in SSTableWriter.appendFromStream
(needed to rebuild index & other compoment)  while I just need to copy the
-Data- file on the disk.

So I think I'm going to provide my own IncomingFileMessage and its own
deserializer.



2015-05-09 23:32 GMT+02:00 Pierre Devops <pierredevops@gmail.com>:

> Thanks yuki, copying SSLTableLoader was the first thing I try, but without
> success.
>
> I checked BulkLoadConnectionFactory (
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/tools/BulkLoadConnectionFactory.java)
> and I don't see what it provide over the DefaultConnectionFactory that can
> help me more in this case.
>
> Without setting up a custom connection factory, it manages already to
> connect to the node, and send a streaming request (I see it in cassandra
> logs).
>
> INFO  21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899 ID#0]
>> Creating new streaming plan for SST Import
>> INFO  21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899, ID#0]
>> Received streaming plan for SST Import
>> INFO  21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899, ID#0]
>> Received streaming plan for SST Import
>> INFO  21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899 ID#0]
>> Prepare completed. Receiving 0 files(0 bytes), sending 2 files(4083518
>> bytes)
>> INFO  21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899] Session
>> with /127.0.0.1 is complete
>> WARN  21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899] Stream
>> failed
>> ERROR 21:16:25 [Stream #a630d860-f690-11e4-a2d0-adca0d5ee899] Streaming
>> error occurred
>
>
>
> So it looks like my client is receiving two message in its
> ConnectionHandler loop (
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/streaming/ConnectionHandler.java#L251)
> , the first one is a PREPARE_MESSAGE type with a StreamSummary indicating
> the good number of files.
>
> But the second message it receives, it fails to deserialize. So I debugged
> and streamed what was coming from this socket, and it was the sstables. but
> I don't know why it fails deseriliazion of message type.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message