Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 26169 invoked from network); 8 Nov 2010 16:47:19 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 8 Nov 2010 16:47:19 -0000 Received: (qmail 57084 invoked by uid 500); 8 Nov 2010 16:47:46 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 56271 invoked by uid 500); 8 Nov 2010 16:47:45 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 56262 invoked by uid 99); 8 Nov 2010 16:47:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 16:47:44 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mcanaleta@gmail.com designates 209.85.210.44 as permitted sender) Received: from [209.85.210.44] (HELO mail-pz0-f44.google.com) (209.85.210.44) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 16:47:37 +0000 Received: by pzk4 with SMTP id 4so1200644pzk.31 for ; Mon, 08 Nov 2010 08:47:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=QGAYras+klgNoeNUEuLhVS1ns08jpxl4NqmHWN2xmP4=; b=M7vtzP6a9KbSOQnRwrKqeNOjnTr1RYE0z8bsOHjJfhv2ljKqQ3NPtSeogH/7gT2jT+ mabV1Nw17d302bUs4IUsxrixaNE9E9upgK8RRfublXIzcUAvE/drDdEYEcaXrvWY5s24 CPFKi7ehVyG6o5mL/PT8h8fMXE0Tu+7ZQijl8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=pGRc+jl2GT49bGUrtPQ9ea/iqxq3E0f1gec5ZutUvSujy3qiL/mhG6jEf5sFMhnrDs 5nSm+5iIeAkaAUqevMRLwSTEX+5/Wb6JmXPzdntJEWMuTCxn4b77jAomusY1kOptNBpv TSeBc7EAkm0LoLn18vydnzyL/HLnhp4V7z7yE= MIME-Version: 1.0 Received: by 10.42.225.131 with SMTP id is3mr3881201icb.2.1289234834594; Mon, 08 Nov 2010 08:47:14 -0800 (PST) Received: by 10.231.12.1 with HTTP; Mon, 8 Nov 2010 08:47:14 -0800 (PST) In-Reply-To: References: <67a5913d-169e-6daa-8483-bac1f184dbf2@me.com> Date: Mon, 8 Nov 2010 17:47:14 +0100 Message-ID: Subject: Re: New nodes won't bootstrap on .66 From: Marc Canaleta To: user Content-Type: multipart/alternative; boundary=20cf3054ab2f591d5504948d6012 X-Virus-Checked: Checked by ClamAV on apache.org --20cf3054ab2f591d5504948d6012 Content-Type: text/plain; charset=ISO-8859-1 I have just solved the problem removing the second keyspace (manually moving its column families to the first). So it seems the problem appears when having multiple keyspaces. 2010/11/8 Thibaut Britz > Hi, > > No I didn't solve the problem. I reinitialized the cluster and gave each > node manually a token before adding data. There are a few messages in > multiple threads related to this, so I suspect it's very common and I hope > it's gone with 0.7. > > Thibaut > > > > > > On Sun, Nov 7, 2010 at 6:57 PM, Marc Canaleta wrote: > >> Hi, >> >> Did you solve this problem? I'm having the same poblem. I'm trying to >> bootstrap a third node in a 0.66 cluster. It has two keyspaces: Keyspace1 >> and KeyspaceLogs, both with replication factor 2. >> >> It starts bootstrapping, receives some streams but it keeps waiting for >> streams. I enabled the debug mode. This lines may be useful: >> >> DEBUG [main] 2010-11-07 17:39:50,052 BootStrapper.java (line 70) Beginning >> bootstrap process >> DEBUG [main] 2010-11-07 17:39:50,082 StorageService.java (line 160) Added >> /10.204.93.16/Keyspace1 as a bootstrap source >> ... >> DEBUG [main] 2010-11-07 17:39:50,090 StorageService.java (line 160) Added >> /10.204.93.16/KeyspaceLogs as a bootstrap source >> ... (streaming mesages) >> DEBUG [Thread-56] 2010-11-07 17:45:51,706 StorageService.java (line 171) >> Removed /10.204.93.16/Keyspace1 as a bootstrap source; remaining is [/ >> 10.204.93.16] >> ... >> (and never ends). >> >> It seems it is waiting for [/10.204.93.16] when it should be waiting for >> /10.204.93.16/KeyspaceLogs. >> >> The third node is 64 bits, while the two existing nodes are 32 bits. Can >> this be a problem? >> >> Thank you. >> >> >> 2010/10/28 Dimitry Lvovsky >> >> Maybe your 7000 is being blocked by iptables >>> or some firewall or maybe you have it bound ( tag ) to >>> localhost instead an ip address. >>> >>> Hope this helps, >>> Dimitry. >>> >>> >>> >>> On Thu, Oct 28, 2010 at 5:35 PM, Thibaut Britz < >>> thibaut.britz@trendiction.com> wrote: >>> >>>> Hi, >>>> >>>> I have the same problem with 0.6.5 >>>> >>>> New nodes will hang forever in bootstrap mode (no streams are being >>>> opened) and the receiver thread just waits for data forever: >>>> >>>> >>>> INFO [Thread-53] 2010-10-27 20:33:37,399 SSTableReader.java (line 120) >>>> Sampling index for /hd2/cassandra/data/table_xyz/ >>>> table_xyz-3-Data.db >>>> INFO [Thread-53] 2010-10-27 20:33:37,444 StreamCompletionHandler.java >>>> (line 64) Streaming added /hd2/cassandra/data/table_xyz/table_xyz-3-Data.db >>>> >>>> Stacktracke: >>>> >>>> "pool-1-thread-53" prio=10 tid=0x00000000412f2800 nid=0x215c runnable >>>> [0x00007fd7cf217000] >>>> java.lang.Thread.State: RUNNABLE >>>> at java.net.SocketInputStream.socketRead0(Native Method) >>>> at java.net.SocketInputStream.read(SocketInputStream.java:129) >>>> at >>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:218) >>>> at >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:258) >>>> at >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:317) >>>> - locked <0x00007fd7e77e0520> (a java.io.BufferedInputStream) >>>> at >>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:126) >>>> at >>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >>>> at >>>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:314) >>>> at >>>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:262) >>>> at >>>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:192) >>>> at >>>> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1154) >>>> at >>>> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:662) >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Oct 28, 2010 at 12:44 PM, aaron morton >>> > wrote: >>>> >>>>> The best approach is to manually select the tokens, see the Load >>>>> Balancing section http://wiki.apache.org/cassandra/Operations Also >>>>> >>>>> Are there any log messages in the existing nodes or the new one which >>>>> mention each other? >>>>> >>>>> Is this a production system? Is it still running ? >>>>> >>>>> Sorry there is not a lot to go on, it sounds like you've done the right >>>>> thing. I'm assuming things like the Cluster Name, seed list and port numbers >>>>> are set correct as the new node got some data. >>>>> >>>>> You'll need to dig through the logs a bit more to see that the boot >>>>> strapping started and what was the last message it logged. >>>>> >>>>> Good Luck. >>>>> Aaron >>>>> >>>>> On 27 Oct 2010, at 22:40, Dimitry Lvovsky wrote: >>>>> >>>>> Hi Aaron, >>>>> Thanks for your reply. >>>>> >>>>> We still haven't solved this unfortunately. >>>>> >>>>> How did you start the bootstrap for the .18 node ? >>>>> >>>>> >>>>> Standard way: we set "AutoBootstrap" to true and added all the servers >>>>> from the working ring as seeds. >>>>> >>>>> >>>>>> Was it the .18 or the .17 node you tried to add >>>>> >>>>> >>>>> We first tried adding .17, it streamed for a while, took on a 50GB of >>>>> load, stopped streaming but then didn't enter into the ring. We left it for >>>>> a few days to see if it would come in, but no luck. After that we did >>>>> decommission and removeToken ( in that order) operations. >>>>> Since we couldn't get .17 in we tried again with .18. Before doing so >>>>> we increased the RpcTimeoutInMillis from 1000, to 10000 having read that >>>>> this may cause the problem of nodes not entering into the ring. It's been >>>>> going since friday and still, like .17, won't come into the ring. >>>>> >>>>> Does it have a token in the config or did you use nodetool move to set >>>>>> it >>>>> >>>>> No we didn't manually set the token in the config, rather we were >>>>> relaying on the token to be assigned durring bootstrap from the >>>>> RandomPartitioner. >>>>> >>>>> Again thanks for the help. >>>>> >>>>> Dimitry. >>>>> >>>>> >>>>> >>>>> On Tue, Oct 26, 2010 at 10:14 PM, Aaron Morton < >>>>> aaron@thelastpickle.com> wrote: >>>>> >>>>>> Dimitry, Did you get anywhere with this ? >>>>>> >>>>>> Was it the .18 or the .17 node you tried to add ? How did you start >>>>>> the bootstrap for the .18 node ? Does it have a token in the config or did >>>>>> you use nodetool move to set it? >>>>>> >>>>>> I had a quick look at the code AKAIK the message about removing the >>>>>> fat client is logged when the node does not have a record of the token the >>>>>> other node as. >>>>>> >>>>>> Aaron >>>>>> >>>>>> On 26 Oct, 2010,at 10:42 PM, Dimitry Lvovsky >>>>>> wrote: >>>>>> >>>>>> Hi All, >>>>>> We recently upgraded from .65 to .66 after which we tried adding a new >>>>>> node to our cluster. We left it bootstrapping and after 3 days, it still >>>>>> refused to join the ring. The strange thing is that nodetool info shows 50GB >>>>>> of load and nodetool ring shows that it sees the rest of ring, which it is >>>>>> not part of. We tried the process again with another server -- again the >>>>>> same thing as before: >>>>>> >>>>>> >>>>>> //from machine 192.168.218 >>>>>> >>>>>> >>>>>> /opt/cassandra/bin/nodetool -h localhost -p 8999 info >>>>>> 131373516047318302934572185119435768941 >>>>>> Load : 52.85 GB >>>>>> Generation No : 1287761987 >>>>>> Uptime (seconds) : 323157 >>>>>> Heap Memory (MB) : 795.42 / 1945.63 >>>>>> >>>>>> >>>>>> /opt/cassandra/bin/nodetool -h localhost -p 8999 ring >>>>>> Address Status Load Range Ring >>>>>> 158573510920250391466717289405976537674 >>>>>> 192.168.2.22 Up 59.45 GB 28203205416427384773583427414698832202 |<--| >>>>>> 192.168.2.23 Up 44.95 GB 60562227403709245514637766500430120055 | | >>>>>> 192.168.2.20 Up 47.15 GB 104160057322065544623939416372654814065 | | >>>>>> 192.168.2.21 Up 61.04 GB 158573510920250391466717289405976537674 |-->| >>>>>> >>>>>> opt/cassandra/bin/nodetool -h localhost -p 8999 streams >>>>>> Mode: Bootstrapping >>>>>> Not sending any streams. >>>>>> Not receiving any streams. >>>>>> >>>>>> >>>>>> Whats more, while looking at the log of one of the nodes I see gossip >>>>>> messages from 192.168.1.17 -- the first node we tried to add to the cluster >>>>>> but which is not running at the the time of the log message: >>>>>> INFO [Timer-0] 2010-10-26 02:13:20,340 Gossiper.java (line 406) >>>>>> FatClient /192.168.2.17 has been silent for 3600000ms, removing from >>>>>> gossip >>>>>> INFO [GMFD:1] 2010-10-26 02:13:51,398 Gossiper.java (line 591) Node / >>>>>> 192.168.2.17 is now part of the cluster >>>>>> >>>>>> >>>>>> Thanks in advance for the help, >>>>>> Dimitry >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Dimitry Lvovsky >>>>> Director of Engineering >>>>> ReviewPro >>>>> www.reviewpro.com >>>>> +34 616 337 103 >>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> Dimitry Lvovsky >>> Director of Engineering >>> ReviewPro >>> www.reviewpro.com >>> +34 616 337 103 >>> >> >> > --20cf3054ab2f591d5504948d6012 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I have just solved the problem removing the second keyspace (manually movin= g its column families to the first). So it seems the problem appears when h= aving multiple keyspaces.

2010/11/8 Thiba= ut Britz <thibaut.britz@trendiction.com>
Hi,

No I didn't solve the proble= m. I reinitialized the cluster and gave each node manually a token before a= dding data. There are a few messages in multiple threads related to this, s= o I suspect it's very common and I hope it's gone with 0.7.

Thibaut





On Sun, Nov 7, 2010 at 6:57 PM, Marc Canaleta <m= canaleta@gmail.com> wrote:
Hi,

Did you solve this problem? I'm having the same = poblem. I'm trying to bootstrap a third node in a 0.66 cluster. It has = two keyspaces: Keyspace1 and KeyspaceLogs, both with replication factor 2.= =A0

It starts bootstrapping, receives some streams but it k= eeps waiting for streams. I enabled the debug mode. This lines may be usefu= l:

DEBUG [main] 2010-11-07 17:39:50,052 BootS= trapper.java (line 70) Beginning bootstrap process
DEBUG [main] 2010-11-07 17:39:50,082 StorageService.java (line 16= 0) Added /10.20= 4.93.16/Keyspace1 as a bootstrap source
...
DEBUG [main] 2010-11-07 17:39:50,090 StorageService.java (line 160) Ad= ded /10.204.= 93.16/KeyspaceLogs as a bootstrap source
... (streaming mesages)
DEBUG [Thread-56] 2010-11= -07 17:45:51,706 StorageService.java (line 171) Removed /10.204.93.16/Keyspace1 as a b= ootstrap source; remaining is [/10.204.93.16]
...
(and never ends).

It seem= s it is waiting for=A0=A0[/10.204.93.16] when it should be waiting for /10.204.93.16/KeyspaceLogs.

The third node is 64 bits, while the two existing nodes= are 32 bits. Can this be a problem?

Thank you.


2010/10/28 Dimitry Lvovsky= <dimitry@reviewpro.com>

Maybe your =A0=A0=A0<= ;StoragePort>7000</StoragePort> is being blocked by iptables or so= me firewall or maybe you have it bound (<ListenAddress> tag ) =A0to l= ocalhost instead an ip address.

Hope this helps,
Dimitry.



On Thu, Oct 28, 2010 at 5:35 PM, Thibaut Britz <= thibaut.britz@trendiction.com> wrote:
Hi,

I have the s= ame problem with 0.6.5

New nodes will hang forever in bootstrap mode= (no streams are being opened) and the receiver thread just waits for data = forever:


=A0INFO [Thread-53] 2010-10-27 20:33:37,399 SSTableReader.java (lin= e 120) Sampling index for /hd2/cassandra/data/table_xyz/
table_xyz-3-Data.db
=A0INFO [Thread-53] 2010-10-27 20:33:37,444 StreamCompletionHandler.java (l= ine 64) Streaming added /hd2/cassandra/data/table_xyz/table_xyz-3-Data.db
Stacktracke:

"pool-1-thread-53" prio=3D10 tid=3D0x00= 000000412f2800 nid=3D0x215c runnable [0x00007fd7cf217000]
=A0=A0 java.lang.Thread.State: RUNNABLE
=A0=A0=A0=A0=A0=A0=A0 at java.ne= t.SocketInputStream.socketRead0(Native Method)
=A0=A0=A0=A0=A0=A0=A0 at = java.net.SocketInputStream.read(SocketInputStream.java:129)
=A0=A0=A0=A0= =A0=A0=A0 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)=
=A0=A0=A0=A0=A0=A0=A0 at java.io.BufferedInputStream.read1(BufferedInputStr= eam.java:258)
=A0=A0=A0=A0=A0=A0=A0 at java.io.BufferedInputStream.read(= BufferedInputStream.java:317)
=A0=A0=A0=A0=A0=A0=A0 - locked <0x00007= fd7e77e0520> (a java.io.BufferedInputStream)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrift.transport.TIOStreamTransport.rea= d(TIOStreamTransport.java:126)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrif= t.transport.TTransport.readAll(TTransport.java:84)
=A0=A0=A0=A0=A0=A0=A0= at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java= :314)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrift.protocol.TBinaryProtocol.readI32= (TBinaryProtocol.java:262)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.thrift.pr= otocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:192)
=A0=A0= =A0=A0=A0=A0=A0 at org.apache.cassandra.thrift.Cassandra$Processor.process(= Cassandra.java:1154)
=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.thrift.CustomTThreadPoolServe= r$WorkerProcess.run(CustomTThreadPoolServer.java:167)
=A0=A0=A0=A0=A0=A0= =A0 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExe= cutor.java:886)
=A0=A0=A0=A0=A0=A0=A0 at java.util.concurrent.ThreadPool= Executor$Worker.run(ThreadPoolExecutor.java:908)
=A0=A0=A0=A0=A0=A0=A0 at java.lang.Thread.run(Thread.java:662)


<= /div>










On Thu, Oct 28, 2010 at 12:44 PM, aaron morton <a= aron@thelastpickle.com> wrote:
The best approach is to manually select the tokens, see the Lo= ad Balancing section=A0http://wiki.apache.org/cassandra/Operations Also= =A0

Are there any log messages in the existing nodes or the new = one which mention each other?=A0

Is this a product= ion system? Is it still running ?=A0

Sorry there i= s not a lot to go on, it sounds like you've done the right thing. I'= ;m assuming things like the Cluster Name, seed list and port numbers are se= t correct as the new node got some data.

You'll need to dig through the logs a bit more to s= ee that the boot strapping started and what was the last message it logged.= =A0

Good Luck.=A0
Aaron

On 27 Oct 2010, at 22:40, Dimitry Lvovsky wrote:

Hi Aaron,
Thanks for your reply.

We still=A0haven't=A0solved this=A0unfortunately.=A0

How did you start the bootstrap for the .18 node ?=A0

=
Standard way:=A0we set "AutoBootstrap" to true and add= ed all the servers from the working ring as seeds. =A0
=A0
Was it the .18 or the .17 node you tried to add

=
We first tried adding .17, it streamed for a while, took on a 50GB of = load, stopped streaming but then didn't enter into the ring. =A0We left= it for a few days to see if it would come in, but no luck. =A0After that w= e did =A0decommission and =A0removeToken ( in that order) operations.=A0
Since we couldn't get .17 in we tried again with .18. =A0Before do= ing so we increased the RpcTimeoutInMillis from 1000, to 10000 having read = that this may cause the problem of nodes not entering into the ring. =A0=A0= It's been going since friday and still, like .17, won't come into t= he ring.

Does it have a token in the config or did you use nodetool move to set it
No we didn't manually set the token in the config, rath= er we were relaying on the token to be assigned durring bootstrap from the= =A0RandomPartiti= oner.

Again thanks for the help.

Dim= itry.
=A0=A0

=A0
On Tue, Oct 26, 2010 at 10:14 PM, Aaron Morton &= lt;aaron@thela= stpickle.com> wrote:
Dimitry, Did = you get anywhere with this ?=A0

Was it the .18 or = the .17 node you tried to add ? How did you start the bootstrap for the .18= node ? Does it have a token in the config or did you use nodetool move to = set it?=A0

I had a quick look at the code AKAIK =A0the message abo= ut removing the fat client is logged when the node does not have a record o= f the token the other node as.=A0

Aaron

On 26 Oct, 2010,at 10:42 PM, Dimitry Lvovsky <dimitry@reviewpro.com> wro= te:

Hi All,
We recently upgraded from .65 to .66 after which we tried adding a new node= to our cluster. We left it bootstrapping and after 3 days, it still refus= ed to join the ring. The strange thing is that nodetool info shows 50GB = of load and nodetool ring shows that it sees the rest of ring, which it is= not part of. We tried the process again with another server -- again the = same thing as before:


//from machine 192.168.218


/opt/cassandra/bin/nodetool -h localhost -p 8999 info
131373516047318302934572185119435768941
Load : 52.85 GB
Generation No : 1287761987
Uptime (seconds) : 323157
Heap Memory (MB) : 795.42 / 1945.63


/opt/cassandra/bin/nodetool -h localhost -p 8999 ring
Address Status Load Range = Ring
158573510920250391466717289405976537= 674
192.168.2.22 Up 59.45 GB 282032054164273847735834274146988322= 02 |<--|
192.168.2.23 Up 44.95 GB 605622274037092455146377665004301200= 55 | |
192.168.2.20 Up 47.15 GB 104160057322065544623939416372654814= 065 | |
192.168.2.21 Up 61.04 GB 158573510920250391466717289405976537= 674 |-->|

opt/cassandra/bin/nodetool -h localhost -p 8999 streams
Mode: Bootstrapping
Not sending any streams.
Not receiving any streams.


Whats more, while looking at the log of one of the nodes I see gossip messa= ges from 192.168.1.17 -- the first node we tried to add to the cluster but = which is not running at the the time of the log message:
INFO [Timer-0] 2010-10-26 02:13:20,340 Gossiper.java (line 406) FatClient = /192.168.2.17 has be= en silent for 3600000ms, removing from gossip
INFO [GMFD:1] 2010-10-26 02:13:51,398 Gossiper.java (line 591) Node /192.168.2.17 is now part = of the cluster


Thanks in advance for the help,
Dimitry

=

--
Dimitry Lvovsky
Director of Engineering
= ReviewPro
www.re= viewpro.com
+34 616 337 103





--
Dimitry Lvo= vsky
Director of Engineering
ReviewPro
www.reviewpro.com
+34 616 337 103



--20cf3054ab2f591d5504948d6012--