crail-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonas Pfefferle" <peppe...@japf.ch>
Subject Re: Question on crail type-1 with terasort shuffle data
Date Wed, 26 Jun 2019 16:24:12 GMT
I'm talking subsystem NQN. Host NQN should not be relevant. Per default Host 
NQN of the clients is generated randomly (you can set it via the 
configuration file).


  On Wed, 26 Jun 2019 16:22:18 +0000
  David Crespi <david.crespi@storedgesystems.com> wrote:
> We’re talking host nqn right, not subsystem nqn?
> 
> David
> 714-476-2692
> 
>> On Jun 26, 2019, at 09:17, Jonas Pfefferle <pepperjo@japf.ch> wrote:
>> 
>> There can only be 1 NQN in the whole system. For multiple storage 
>>classes just use different ip/port combinations with the same NQN. 
>>e.g. start a NVMf server on node 1 with storage class 1 with NQN X 
>>and start a server on node 2 with storage class 2 with NQN X. You can 
>>start another server on node 1 by choosing a different port but still 
>>use the same NQN X.
>> I know that this is cumbersome to use. We are working on it, we just 
>>have not found a solution that makes everyone happy :-)
>> 
>> 
>> Regards,
>> Jonas
>> 
>> On Wed, 26 Jun 2019 16:09:06 +0000
>> David Crespi <david.crespi@storedgesystems.com> wrote:
>>> Sorry Jonas but I’m still confused.
>>> I started off doing this by having each storage class 1 connected to 
>>>it’s own subsystem,
>>> with a single namespace.  This is where I was seeing the correct IP 
>>>address of that subsystem,
>>> but the wrong controller name.  It was the controller name of the 
>>>storage class 2, that was also
>>> registered with the namenode.  So this is the correct way of using 
>>>it, but it’s having a problem
>>> determining which controller to identify?
>>> Regards,
>>>          David
>>> From: Jonas Pfefferle<mailto:pepperjo@japf.ch>
>>> Sent: Wednesday, June 26, 2019 9:01 AM
>>> To: dev@crail.apache.org<mailto:dev@crail.apache.org>; David 
>>>Crespi<mailto:david.crespi@storedgesystems.com>
>>> Subject: Re: Question on crail type-1 with terasort shuffle data
>>> I'm sure you already figured this out but just to make sure we are 
>>>on the
>>> same page. The NVMf storage tier (or datanode, we use them 
>>>interchangeably)
>>> is just another NVMf client that connects to the NVMf server and 
>>>retrieves
>>> all necessary information like capacity etc. to register the 
>>>datanode to the
>>> namenode. When starting a NVMf datanode you can specify via the 
>>>commandline
>>> what storage class the datanode should be in. It then registers the 
>>>ip/port
>>> combination with the storage class information at the namenode. Now 
>>>whenever
>>> a block is requested from the namenode by a client, the namenode 
>>>sends the
>>> ip/port information and the storage class to the client. With this
>>> information the client connects to the NVMf server (not the 
>>>datanode)
>>> directly with the NQN provided in the config file, namespace 1 and 
>>>the
>>> ip/port information it got from the namenode.
>>> I hope this makes it clearer.
>>> Regards,
>>> Jonas
>>> On Wed, 26 Jun 2019 14:55:05 +0000
>>> David Crespi <david.crespi@storedgesystems.com> wrote:
>>>> Yep.  I wanted to use crail-store for temp results storage, storage
>>>> class 2, AND the shuffle data, with
>>>> 
>>>> storage class 1.
>>>> 
>>>> 
>>>> Regards,
>>>> 
>>>> 
>>>>          David
>>>> 
>>>> 
>>>> ________________________________
>>>> From: Jonas Pfefferle <pepperjo@japf.ch>
>>>> Sent: Wednesday, June 26, 2019 7:51:43 AM
>>>> To: dev@crail.apache.org; David Crespi
>>>> Subject: Re: Question on crail type-1 with terasort shuffle data
>>>> 
>>>> David,
>>>> 
>>>> 
>>>> Let's make sure we talk about the same thing, I assume with type 1
>>>> and 2 you
>>>> mean storage classes?
>>>> 
>>>> Regards,
>>>> Jonas
>>>> 
>>>> On Wed, 26 Jun 2019 14:25:52 +0000
>>>> David Crespi <david.crespi@storedgesystems.com> wrote:
>>>>> I’m a bit confused now. If it only ever handles 1 namespace, then
>>>>> multiple datanodes are useless.
>>>>> 
>>>>> Or are you saying only 1 namespace for type-1?  If you’re not saying
>>>>> only one for type-1, how do you
>>>>> 
>>>>> mix types if limited to 1 namespace?
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> 
>>>>>          David
>>>>> 
>>>>> 
>>>>> ________________________________
>>>>> From: Jonas Pfefferle <pepperjo@japf.ch>
>>>>> Sent: Wednesday, June 26, 2019 12:25:38 AM
>>>>> To: dev@crail.apache.org; David Crespi
>>>>> Subject: Re: Question on crail type-1 with terasort shuffle data
>>>>> 
>>>>> I forgot to add: Namespace has to be 1 for the moment. This is a
>>>>> limitation
>>>>> in how the namenode handles storage tier metadata (essentially it
>>>>> only
>>>>> stores ip/port). You can set it on the server side however it will
>>>>> not be
>>>>> respected by the client, i.e. the client will always use namespace
>>>>> id 1.
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> Jonas
>>>>> 
>>>>> On Wed, 26 Jun 2019 09:17:40 +0200
>>>>> "Jonas Pfefferle" <pepperjo@japf.ch> wrote:
>>>>>> Hi David,
>>>>>> 
>>>>>> 
>>>>>> This is currently a limitation of the NVMf storage tier. You need
to
>>>>>> use the same NQN on all datanodes. The port can be different 
>>>>>>however.
>>>>>> I will clarify this in the documentation.
>>>>>> 
>>>>>> Regards,
>>>>>> Jonas
>>>>>> 
>>>>>> On Tue, 25 Jun 2019 19:43:04 +0000
>>>>>> David Crespi <david.crespi@storedgesystems.com> wrote:
>>>>>>> It occurred to me that perhaps since I’m using the NVMf, that
the
>>>>>>> namenode can only handle one
>>>>>>> 
>>>>>>> Subsystem.  And for the type-1, these resources would need to
be
>>>>>>> added under the same subsystem,
>>>>>>> 
>>>>>>> but as different namespace’s.  Is this correct?  I did try
this and
>>>>>>> it appears to be working. If this IS the
>>>>>>> 
>>>>>>> correct way of utilizing these, how do I know which namespace(s)
are
>>>>>>> being accessed?  I don’t see
>>>>>>> 
>>>>>>> anything in the logs that identify which is being addressed.
>>>>>>> 
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>> 
>>>>>>>          David
>>>>>>> 
>>>>>>> 
>>>>>>> ________________________________
>>>>>>> From: David Crespi <david.crespi@storedgesystems.com>
>>>>>>> Sent: Tuesday, June 25, 2019 11:09:56 AM
>>>>>>> To: dev@crail.apache.org
>>>>>>> Subject: Question on crail type-1 with terasort shuffle data
>>>>>>> 
>>>>>>> I’m starting to run terasort now with the shuffle data going
to
>>>>>>> crail type-1.
>>>>>>> I’ve got the type-2 set to nqn:
>>>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13 @192.168.3.100,
>>>>>>> which appears to be working okay for the result data.  When I
>>>>>>> introduce type-1 into my config,
>>>>>>> it looks like the namenode gets confused and picks the nqn of
the
>>>>>>> type-2 crailstore, instead of
>>>>>>> the one it is assigned (bolded in red below).
>>>>>>> 
>>>>>>> In my test run, I have 5 type-2 nodes (192.168.3.100, ns=1-5)
and
>>>>>>> then 5 type-1 added to the
>>>>>>> spark worker nodes (192.168.3.101-105).  All are running the
same
>>>>>>> image as the namenode,
>>>>>>> which is the spark master.
>>>>>>> 
>>>>>>> Is there some additional setting I need to include for this?
>>>>>>> These are the variables that get set for each container (added
via
>>>>>>> environment)
>>>>>>> # NVMf storage
>>>>>>> crail.storage.nvmf.ip                     $NVMF_IP
>>>>>>> crail.storage.nvmf.port                 $NVMF_PORT
>>>>>>> crail.storage.nvmf.nqn                  $NVMF_NQN
>>>>>>> crail.storage.nvmf.hostnqn           $NVMF_HOSTNQN
>>>>>>> crail.storage.nvmf.namespace      $NVMF_NAMESPACE
>>>>>>> 
>>>>>>> 19/06/25 10:46:43 INFO crail: creating singleton crail file system
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.version 3101
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.directorydepth 16
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.tokenexpiration 10
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.blocksize 1048576
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.cachelimit 0
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.cachepath /dev/hugepages/cache
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.user crail
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.shadowreplication 1
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.debug true
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.statistics true
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.rpctimeout 1000
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.datatimeout 1000
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.buffersize 1048576
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.slicesize 65536
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.singleton true
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.regionsize 1073741824
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrecord 512
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.directoryrandomize true
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.cacheimpl
>>>>>>> org.apache.crail.memory.MappedBufferCache
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.locationmap
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.address
>>>>>>> crail://192.168.1.164:9060
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.blockselection
>>>>>>> roundrobin
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.fileblocks 16
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.rpctype
>>>>>>> org.apache.crail.namenode.rpc.tcp.TcpNameNode
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.log
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.types
>>>>>>> org.apache.crail.storage.nvmf.NvmfStorageTier
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.classes 2
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.rootclass 0
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.keepalive 2
>>>>>>> 19/06/25 10:46:43 INFO crail: buffer cache, allocationCount 0,
>>>>>>> bufferCount 1024
>>>>>>> 19/06/25 10:46:43 INFO crail: Initialize Nvmf storage client
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.ip 192.168.3.100
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.port 4420
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.nqn
>>>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.hostnqn
>>>>>>> nqn.2014-08.org.nvmexpress:uuid:1b4e28ba-2fa1-11d2-883f-0016d3cca420
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.allocationsize
>>>>>>> 1073741824
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.storage.nvmf.queueSize 64
>>>>>>> 19/06/25 10:46:43 INFO narpc: new NaRPC server group v1.0,
>>>>>>> queueDepth 32, messageSize 512, nodealy true
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.queueDepth 32
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.messageSize
512
>>>>>>> 19/06/25 10:46:43 INFO crail: crail.namenode.tcp.cores 1
>>>>>>> 19/06/25 10:46:43 INFO crail: connected to namenode(s)
>>>>>>> /192.168.1.164:9060
>>>>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>>>>>> 19/06/25 10:46:43 INFO crail: lookupDirectory: path /spark
>>>>>>> 19/06/25 10:46:43 INFO crail: lookup: name /spark, success, fd
2
>>>>>>> 19/06/25 10:46:43 INFO CrailDispatcher: creating main dir /spark
>>>>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive
true
>>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /,
fd 0,
>>>>>>> streamId 1, isDir true, writeHint 0
>>>>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>>>>> address = /192.168.3.100:4420, subsystem NQN =
>>>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13
>>>>>>> 19/06/25 10:46:43 INFO disni: creating  RdmaProvider of type
'nat'
>>>>>>> 19/06/25 10:46:43 INFO disni: jverbs jni version 32
>>>>>>> 19/06/25 10:46:43 INFO disni: sock_addr_in size mismatch, jverbs
>>>>>>> size 28, native size 16
>>>>>>> 19/06/25 10:46:43 INFO disni: IbvRecvWR size match, jverbs size
32,
>>>>>>> native size 32
>>>>>>> 19/06/25 10:46:43 INFO disni: IbvSendWR size mismatch, jverbs
size
>>>>>>> 72, native size 128
>>>>>>> 19/06/25 10:46:43 INFO disni: IbvWC size match, jverbs size 48,
>>>>>>> native size 48
>>>>>>> 19/06/25 10:46:43 INFO disni: IbvSge size match, jverbs size
16,
>>>>>>> native size 16
>>>>>>> 19/06/25 10:46:43 INFO disni: Remote addr offset match, jverbs
size
>>>>>>> 40, native size 40
>>>>>>> 19/06/25 10:46:43 INFO disni: Rkey offset match, jverbs size
48,
>>>>>>> native size 48
>>>>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>>>>> 139964194703408
>>>>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel
0
>>>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964194793600
>>>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 0, idPriv
0
>>>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>>>> /192.168.3.100:4420
>>>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>>>>> 19/06/25 10:46:43 INFO disni: allocPd, objId 139964194800080
>>>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>>>> 461, pd 1
>>>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>>>> 139962246807056
>>>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964194801488,
ncqe
>>>>>>> 64
>>>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964194803176,
>>>>>>> send_wr size 32, recv_wr_size 32
>>>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>>>> RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:45059, 
>>>>>>>dstAddress
>>>>>>> /192.168.3.100:4420
>>>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195036000
>>>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 1, idPriv
0
>>>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>>>> /192.168.3.100:4420
>>>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>>>> 461, pd 1
>>>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>>>> 139962246807056
>>>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195036752,
ncqe
>>>>>>> 128
>>>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195037304,
>>>>>>> send_wr size 64, recv_wr_size 64
>>>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>>>> RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:57619, 
>>>>>>>dstAddress
>>>>>>> /192.168.3.100:4420
>>>>>>> 19/06/25 10:46:43 INFO crail: EndpointCache miss
>>>>>>> /192.168.3.100:4420, fsId 0, cache size 1
>>>>>>> 19/06/25 10:46:43 INFO crail: delete: name /spark, recursive
true,
>>>>>>> success
>>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /,
fd 0,
>>>>>>> streamId 1, capacity 262656
>>>>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark, type
>>>>>>> DIRECTORY, storageAffinity 0, locationAffinity 0
>>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /,
fd 0,
>>>>>>> streamId 2, isDir true, writeHint 0
>>>>>>> 19/06/25 10:46:43 INFO crail: EndpointCache hit /192.168.3.100:4420,
>>>>>>> fsId 0
>>>>>>> 19/06/25 10:46:43 INFO crail: createFile: name /spark, success,
fd
>>>>>>> 4, token 0
>>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, close, path /,
fd 0,
>>>>>>> streamId 2, capacity 524800
>>>>>>> 19/06/25 10:46:43 INFO crail: createNode: name /spark/broadcast,
>>>>>>> type DIRECTORY, storageAffinity 0, locationAffinity 0
>>>>>>> 19/06/25 10:46:43 INFO crail: CoreOutputStream, open, path /spark,
>>>>>>> fd 4, streamId 3, isDir true, writeHint 0
>>>>>>> 19/06/25 10:46:43 INFO crail: Connecting to NVMf target at Transport
>>>>>>> address = /192.168.3.104:4420, subsystem NQN =
>>>>>>> nqn.2018-12.com.StorEdgeSystems:cntlr13  (<-- should be
>>>>>>> nqn.2018-12.com.StorEdgeSystems:worker-4)
>>>>>>> 19/06/25 10:46:43 INFO disni: createEventChannel, objId
>>>>>>> 139964195131168
>>>>>>> 19/06/25 10:46:43 INFO disni: createId, id 139964195141520
>>>>>>> 19/06/25 10:46:43 INFO disni: new client endpoint, id 2, idPriv
0
>>>>>>> 19/06/25 10:46:43 INFO disni: launching cm processor, cmChannel
0
>>>>>>> 19/06/25 10:46:43 INFO disni: resolveAddr, addres
>>>>>>> /192.168.3.104:4420
>>>>>>> 19/06/25 10:46:43 INFO disni: resolveRoute, id 0
>>>>>>> 19/06/25 10:46:43 INFO disni: setting up protection domain, context
>>>>>>> 461, pd 1
>>>>>>> 19/06/25 10:46:43 INFO disni: new endpoint CQ processor
>>>>>>> 19/06/25 10:46:43 INFO disni: createCompChannel, context
>>>>>>> 139962246807056
>>>>>>> 19/06/25 10:46:43 INFO disni: createCQ, objId 139964195142224,
ncqe
>>>>>>> 64
>>>>>>> 19/06/25 10:46:43 INFO disni: createQP, objId 139964195151896,
>>>>>>> send_wr size 32, recv_wr_size 32
>>>>>>> 19/06/25 10:46:43 INFO disni: connect, id 0
>>>>>>> 19/06/25 10:46:43 INFO disni: got event type +
>>>>>>> RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.3.13:35873, 
>>>>>>>dstAddress
>>>>>>> /192.168.3.104:4420
>>>>>>> 19/06/25 10:46:43 INFO crail: ERROR: failed data operation
>>>>>>> com.ibm.jnvmf.UnsuccessfulComandException: Command was not
>>>>>>> successful. {StatusCodeType: 1 - Command Specific, SatusCode:
132 -
>>>>>>> The host is not allowed to establish an association to any 
>>>>>>>controller
>>>>>>> in the NVM subsystem or the host is not allowed to establish
an
>>>>>>> association to the specified controller., CID: 0, Do_not_retry:
>>>>>>> false, More: false, SQHD: 0}
>>>>>>>       at com.ibm.jnvmf.QueuePair.connect(QueuePair.java:128)
>>>>>>>       at
>>>>>>> com.ibm.jnvmf.AdminQueuePair.connect(AdminQueuePair.java:36)
>>>>>>> 
>>>>>>> 
>>>>>>> Regards,
>>>>>>> 
>>>>>>>          David
>>>>>>> 
>>>>>> 


Mime
View raw message