cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam <adam.scarce...@gmail.com>
Subject Re: How to Upgrade Virtual Router from 4.3 to 4.4.1
Date Mon, 03 Nov 2014 22:42:25 GMT
ok, so I've rolled back and now none of my hosts are connecting. My system
VMs are not running, and the only thing I can see in the logs is that my
zone is not ready for either the Console Proxy or the Secondary Storage VM
to start because none of my hosts are reachable, even though in the UI it
says they're all up and running.

I'm really losing all patience and confidence with CloudStack. I currently
have a 10 person team who cannot access any of their VMs because CloudStack
can't seem to find itself.

I stopped all services and erased all 4.4.1 packages on the management
server as well as all the KVM hosts.

# STOP ALL CLOUDSTACK SERVICES ON MANAGEMENT HOST
service cloudstack-agent stop
service cloudstack-usage stop
service cloudstack-management stop

# STOP CLOUDSTACK AGENT ON KVM HOST #N
service cloudstack-agent stop

# DROP ALL DATABASES
# THIS SCRIPT DROPS ALL DATABASES IN THE CLOUDSTACK MySQL DB (cloud,
cloud_usage, cloudbridge)
/root/cronScripts/mysql_DropAllDatabases.sh

# MODIFIED/etc/yum.repos.d/cloudstack.repo to use 4.3 repository
[root@cs-east-prod1 /]# cat /etc/yum.repos.d/cloudstack.repo
[cloudstack]
name=cloudstack
baseurl=http://cloudstack.apt-get.eu/rhel/4.3/
enabled=1
gpgcheck=0
[root@cs-east-prod1 /]#

# YUM ERASE/FS CLEAN/YUM INSTALL ON MANAGEMENT HOST
yum -y erase cloudstack\*
rm -Rf /var/log/cloudstack/management
rm -Rf /usr/share/cloudstack*
rm -Rf /usr/share/doc/cloudstack*
rm -Rf /var/log/cloudstack
rm -Rf /etc/cloudstack
rm -Rf /var/cache/cloudstack
rm -Rf /var/cloudstack
yum clean all
yum -y install cloudstack\*

# YUM ERASE/FS CLEAN/YUM INSTALL ON KVM HOST(S)
yum -y erase cloudstack\*
rm -Rf /usr/share/cloudstack*
rm -Rf /usr/share/doc/cloudstack*
rm -Rf /var/log/cloudstack
rm -Rf /etc/cloudstack
yum clean all
yum -y install cloudstack-agent

# RAN cloudstack-setup-databases
cloudstack-setup-databases cloud:<PASSWORD> --deploy-as=root:<PASSWORD> -i
10.97.38.10

# RESTORED MySQL BACKUPS
mysql -u root -p<PASSWORD> cloud <
/var/www/html/mysql-backups/cloud_11-01-2014.sql
mysql -u root -p<PASSWORD> cloud <
/var/www/html/mysql-backups/cloud_usage_11-01-2014.sql
mysql -u root -p<PASSWORD> cloud <
/var/www/html/mysql-backups/cloud_meta_11-01-2014.sql
mysql -u root -p<PASSWORD> cloud <
/var/www/html/mysql-backups/cloudbridge_11-01-2014.sql

# START ALL CLOUDSTACK SERVICES ON MANAGEMENT HOST
service cloudstack-management start
service cloudstack-usage start
service cloudstack-agent start

# START ALL CLOUDSTACK SERVICES ON KVM HOST(S)
service cloudstack-agent start

Where do I go from here?

Best Regards,



Adam Scarcella
On Nov 3, 2014 2:06 PM, "Nux!" <nux@li.nux.ro> wrote:

> Adam,
>
> I take it replacing the VR's disk did not work? Well, I'm hoping 4.4.2
> will come with a fixed template.
>
> Re downgrading this is how I did it:
> 1 - stop all services (mgmt and agents) and "yum downgrade cloudstack-\*"
> 2 - drop the DBs and import backups
> 3 - restart the downgraded services
> 4 - you'll also need to rebuild your VR, possibly "virsh destroy" the
> 4.4.1 one if it's still around
>
> Good luck
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Adam" <adam.scarcella@gmail.com>
> > To: users@cloudstack.apache.org
> > Sent: Monday, 3 November, 2014 18:51:13
> > Subject: Re: How to Upgrade Virtual Router from 4.3 to 4.4.1
>
> > I've decided to revert back to 4.3.1 since I cannot wait any loner to get
> > 4.4.1 working.
> >
> > I have a complete backup of all three databases just before I attempted
> to
> > upgrade so I'm assuming all I would have to do is remove 4.4.1, reinstall
> > 4.3.1, restore the db and then start everything back up correct?
> >
> > Are there any official steps to roll back?
> >
> > Best Regards,
> >
> >
> >
> > Adam Scarcella
> >
> > On Mon, Nov 3, 2014 at 12:13 PM, Nux! <nux@li.nux.ro> wrote:
> >
> >> What I am saying - and excuse my poor explanations - is that the HV may
> >> have already cached the old template, so the VR still uses the old
> broken
> >> one.
> >>
> >> What I was suggesting is find out which
> /var/lib/libvirt/images/blah.qcow2
> >> the VR is using and replace it with my template.
> >>
> >> --
> >> Sent from the Delta quadrant using Borg technology!
> >>
> >> Nux!
> >> www.nux.ro
> >>
> >> ----- Original Message -----
> >> > From: "Adam" <adam.scarcella@gmail.com>
> >> > To: users@cloudstack.apache.org
> >> > Sent: Monday, 3 November, 2014 17:03:01
> >> > Subject: Re: How to Upgrade Virtual Router from 4.3 to 4.4.1
> >>
> >> >>
> >> >> Not sure if this operation that you performed replaces the current
> >> >> systemvm template and *especially if it replaces it also on the
> >> >> hypervisor where it is running.*
> >> >>
> >> >
> >> > That procedure is the one documented in the official CloudStack
> >> > documentation so I hope it actually replaces the systemvm template.
> The
> >> > final output from that command tells it was successful:
> >> >
> >> > Successfully installed system VM template  to
> >> > /export/secondary/template/tmpl/1/274/
> >> >
> >> > Also, as stated before, the VR is not running on any hypervisor / host
> >> > because I cannot get it to run, so I'm unsure what you mean when you
> >> > say "*especially
> >> > if it replaces it also on the hypervisor where it is running.*"
> >> >
> >> > One more thing I would try is to make sure the VR on the HV is also
> using
> >> >> this image. You can try to compare md5sums of the image on the
> secondary
> >> >> storage and also the backend file for the VR's disk image (qemu-img
> info
> >> >> will show it to you) on the HV.
> >> >> If the sums differ then I would try to replace the VR disk image
> with my
> >> >> template directly on the HV and see if that does it.
> >> >
> >> >
> >> > I really don't understand what you're asking me to do.
> >> >
> >> > Best Regards,
> >> >
> >> >
> >> >
> >> > Adam Scarcella
> >> >
> >> > On Mon, Nov 3, 2014 at 11:50 AM, Nux! <nux@li.nux.ro> wrote:
> >> >
> >> >> Adam,
> >> >>
> >> >> Not sure if this operation that you performed replaces the current
> >> >> systemvm template and especially if it replaces it also on the
> >> hypervisor
> >> >> where it is running.
> >> >>
> >> >> One more thing I would try is to make sure the VR on the HV is also
> >> using
> >> >> this image. You can try to compare md5sums of the image on the
> secondary
> >> >> storage and also the backend file for the VR's disk image (qemu-img
> info
> >> >> will show it to you) on the HV.
> >> >> If the sums differ then I would try to replace the VR disk image
> with my
> >> >> template directly on the HV and see if that does it.
> >> >>
> >> >> --
> >> >> Sent from the Delta quadrant using Borg technology!
> >> >>
> >> >> Nux!
> >> >> www.nux.ro
> >> >>
> >> >> ----- Original Message -----
> >> >> > From: "Adam" <adam.scarcella@gmail.com>
> >> >> > To: users@cloudstack.apache.org
> >> >> > Sent: Monday, 3 November, 2014 16:08:05
> >> >> > Subject: Re: How to Upgrade Virtual Router from 4.3 to 4.4.1
> >> >>
> >> >> > Thanks HTH,
> >> >> >
> >> >> > I decided to try to use your already modified qcow2 file:
> >> >> >
> >> >> > [root@cs-east-prod1 /]#
> >> >> >
> >> >>
> >>
> /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
> >> >> > -m /export/secondary -u
> >> >> http://tmp.nux.ro/systemvm-4.4.1_jira7781.qcow2.bz2
> >> >> > -h kvm -F
> >> >> > --2014-11-03 10:11:32--
> >> >> http://tmp.nux.ro/systemvm-4.4.1_jira7781.qcow2.bz2
> >> >> > Resolving tmp.nux.ro... 217.19.15.108
> >> >> > Connecting to tmp.nux.ro|217.19.15.108|:80... connected.
> >> >> > HTTP request sent, awaiting response... 200 OK
> >> >> > Length: 305498799 (291M) [application/x-bzip2]
> >> >> > Saving to:
> >> >> >
> >> >>
> >>
> “/usr/share/cloudstack-common/scripts/storage/secondary/00ccf4fc-fa94-4e82-8415-f8ac89fbbfaa.qcow2”
> >> >> >
> >> >> >
> >> >>
> >>
> 100%[===================================================================================================================>]
> >> >> > 305,498,799 9.02M/s   in 37s
> >> >> >
> >> >> > 2014-11-03 10:12:10 (7.82 MB/s) -
> >> >> >
> >> >>
> >>
> “/usr/share/cloudstack-common/scripts/storage/secondary/00ccf4fc-fa94-4e82-8415-f8ac89fbbfaa.qcow2”
> >> >> > saved [305498799/305498799]
> >> >> >
> >> >> > Uncompressing to
> >> >> >
> >> >>
> >>
> /usr/share/cloudstack-common/scripts/storage/secondary/00ccf4fc-fa94-4e82-8415-f8ac89fbbfaa.qcow2.tmp
> >> >> > (type bz2)...could take a long time
> >> >> > Moving to
> >> >> >
> >> >>
> >>
> /export/secondary/template/tmpl/1/274///00ccf4fc-fa94-4e82-8415-f8ac89fbbfaa.qcow2...could
> >> >> > take a while
> >> >> > Successfully installed system VM template  to
> >> >> > /export/secondary/template/tmpl/1/274/
> >> >> > [root@cs-east-prod1 /]#
> >> >> >
> >> >> > Then I went into the Admin GUI and tried to start the VR, but it
> >> failed
> >> >> > with the exact same message:
> >> >> >
> >> >> > [image: Inline image 1]
> >> >> >
> >> >> > So I destroyed the Virtual Router and stopped and started the
> >> following
> >> >> > services:
> >> >> >
> >> >> > [root@cs-east-prod1 /]# service iptables stop
> >> >> > iptables: Setting chains to policy ACCEPT: nat filter      [  OK  ]
> >> >> > iptables: Flushing firewall rules:                         [  OK  ]
> >> >> > iptables: Unloading modules:                               [  OK  ]
> >> >> > [root@cs-east-prod1 /]# service cloudstack-management stop
> >> >> > Stopping cloudstack-management:                            [  OK  ]
> >> >> > [root@cs-east-prod1 /]# service cloudstack-agent stop
> >> >> > Stopping Cloud Agent:
> >> >> > [root@cs-east-prod1 /]# service cloudstack-usage stop
> >> >> > Stopping CloudStack Usage Monitor cloudstack-usage         [  OK  ]
> >> >> > [root@cs-east-prod1 /]# service cloudstack-management start
> >> >> > Starting cloudstack-management:                            [  OK  ]
> >> >> > [root@cs-east-prod1 /]# service cloudstack-usage start
> >> >> > Starting CloudStack Usage Monitor cloudstack-usage         [  OK  ]
> >> >> > [root@cs-east-prod1 /]# service cloudstack-agent start
> >> >> > Starting Cloud Agent:
> >> >> > [root@cs-east-prod1 /]#
> >> >> >
> >> >> > I then tried starting any VM instance to have CS automatically
> >> re-create
> >> >> > the VR, but that failed as well.
> >> >> >
> >> >> > Motty, I also looked at trying to increase the RAM and CPU for the
> >> router
> >> >> > but only saw one option
> >> >> >
> >> >> > [image: Inline image 2]
> >> >> >
> >> >> > So I created a new Service Offering with twice as much resources:
> >> >> >
> >> >> > [image: Inline image 3]
> >> >> >
> >> >> > Then upgraded the service offering for the VR
> >> >> >
> >> >> > [image: Inline image 4]
> >> >> >
> >> >> > [image: Inline image 6]
> >> >> >
> >> >> > But it failed still with the exact same error:
> >> >> >
> >> >> > [image: Inline image 7]
> >> >> >
> >> >> > Best Regards,
> >> >> >
> >> >> >
> >> >> >
> >> >> > Adam Scarcella
> >> >> >
> >> >> > On Mon, Nov 3, 2014 at 10:23 AM, motty cruz <motty.cruz@gmail.com>
> >> >> wrote:
> >> >> >
> >> >> >> Adam,
> >> >> >> what are the specifications of the VR? can you upgrade the RAM and
> >> CPU
> >> >> and
> >> >> >> try to restart that VR? I had similar issue and giving the VR more
> >> RAM
> >> >> and
> >> >> >> CPU will start the VR.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> Motty
> >> >> >>
> >> >> >> On Mon, Nov 3, 2014 at 7:04 AM, Nux! <nux@li.nux.ro> wrote:
> >> >> >>
> >> >> >> > Adam,
> >> >> >> >
> >> >> >> > You can modify the qcow2 file that is in your secondary storage,
> >> i.e.
> >> >> you
> >> >> >> > should mount that storage somewhere and go to
> >> >> >> > /mount/point/template/tmpl/274 and modify the qcow2 file.
> >> >> >> >
> >> >> >> > Alternatively you could use my already modified copy to
> overwrite
> >> your
> >> >> >> > uuid.qcow2 with (at your own risk of course):
> >> >> >> > http://tmp.nux.ro/systemvm-4.4.1_jira7781.qcow2.bz2
> >> >> >> >
> >> >> >> > HTH
> >> >> >> >
> >> >> >> > --
> >> >> >> > Sent from the Delta quadrant using Borg technology!
> >> >> >> >
> >> >> >> > Nux!
> >> >> >> > www.nux.ro
> >> >> >> >
> >> >> >> > ----- Original Message -----
> >> >> >> > > From: "Adam" <adam.scarcella@gmail.com>
> >> >> >> > > To: users@cloudstack.apache.org
> >> >> >> > > Sent: Monday, 3 November, 2014 14:58:58
> >> >> >> > > Subject: Re: How to Upgrade Virtual Router from 4.3 to 4.4.1
> >> >> >> >
> >> >> >> > > [root@cs-east-prod1 management]# netstat -a|grep 9090
> >> >> >> > > unix  3      [ ]         STREAM     CONNECTED     29090
> >> >> >> > > [root@cs-east-prod1 management]# nmap localhost
> >> >> >> > >
> >> >> >> > > Starting Nmap 5.51 ( http://nmap.org ) at 2014-11-03 08:47
> EST
> >> >> >> > > Nmap scan report for localhost (127.0.0.1)
> >> >> >> > > Host is up (0.000015s latency).
> >> >> >> > > Other addresses for localhost (not scanned): 127.0.0.1
> >> >> >> > > Not shown: 990 closed ports
> >> >> >> > > PORT     STATE SERVICE
> >> >> >> > > 22/tcp   open  ssh
> >> >> >> > > 25/tcp   open  smtp
> >> >> >> > > 80/tcp   open  http
> >> >> >> > > 111/tcp  open  rpcbind
> >> >> >> > > 199/tcp  open  smux
> >> >> >> > > 631/tcp  open  ipp
> >> >> >> > > 2049/tcp open  nfs
> >> >> >> > > 3306/tcp open  mysql
> >> >> >> > > 8080/tcp open  http-proxy
> >> >> >> > > 9090/tcp open  zeus-admin
> >> >> >> > >
> >> >> >> > > Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds
> >> >> >> > > [root@cs-east-prod1 management]# telnet localhost 9090
> >> >> >> > > Trying ::1...
> >> >> >> > > Connected to localhost.
> >> >> >> > > Escape character is '^]'.
> >> >> >> > > Connection closed by foreign host.
> >> >> >> > > [root@cs-east-prod1 management]# nmap 10.97.38.10
> >> >> >> > >
> >> >> >> > > Starting Nmap 5.51 ( http://nmap.org ) at 2014-11-03 08:49
> EST
> >> >> >> > > Nmap scan report for cs-east-prod1.na.tibco.com (10.97.38.10)
> >> >> >> > > Host is up (0.000015s latency).
> >> >> >> > > Not shown: 993 closed ports
> >> >> >> > > PORT     STATE SERVICE
> >> >> >> > > 22/tcp   open  ssh
> >> >> >> > > 80/tcp   open  http
> >> >> >> > > 111/tcp  open  rpcbind
> >> >> >> > > 2049/tcp open  nfs
> >> >> >> > > 3306/tcp open  mysql
> >> >> >> > > 8080/tcp open  http-proxy
> >> >> >> > > 9090/tcp open  zeus-admin
> >> >> >> > >
> >> >> >> > > Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds
> >> >> >> > > [root@cs-east-prod1 management]# telnet 10.97.38.10 9090
> >> >> >> > > Trying 10.97.38.10...
> >> >> >> > > Connected to 10.97.38.10.
> >> >> >> > > Escape character is '^]'.
> >> >> >> > > Connection closed by foreign host.
> >> >> >> > > [root@cs-east-prod1 management]#
> >> >> >> > >
> >> >> >> > > Also, I checked for the /opt/cloud/bin/ directory on my
> >> Management
> >> >> >> Host,
> >> >> >> > > but it's not there:
> >> >> >> > >
> >> >> >> > > [root@cs-east-prod1 management]# cd /opt/cloud/bin/
> >> >> >> > > -bash: cd: /opt/cloud/bin/: No such file or directory
> >> >> >> > > [root@cs-east-prod1 management]# cd /
> >> >> >> > > [root@cs-east-prod1 /]# find . -name "createipAlias.sh"
> >> >> >> > > [root@cs-east-prod1 /]# find . -name "createIpAlias.sh"
> >> >> >> > > [root@cs-east-prod1 /]#
> >> >> >> > >
> >> >> >> > > Are you saying that the /opt/cloud/bin/createipAlias.sh
> script is
> >> >> >> inside
> >> >> >> > > the Virtual Router VM? If so, how could I possibly change that
> >> >> since it
> >> >> >> > > won't start?
> >> >> >> > >
> >> >> >> > > Best Regards,
> >> >> >> > >
> >> >> >> > >
> >> >> >> > >
> >> >> >> > > Adam Scarcella
> >> >> >> > >
> >> >> >> > > On Mon, Nov 3, 2014 at 9:07 AM, Pierre-Luc Dion <
> >> >> pdion891@apache.org>
> >> >> >> > wrote:
> >> >> >> > >
> >> >> >> > >> Could it be possible the firewall is blocking port 9090 from
> the
> >> >> >> > >> management-server?
> >> >> >> > >>
> >> >> >> > >> 2014-11-03 08:14:36,813 ERROR [c.c.c.ClusterManagerImpl]
> >> >> (main:null)
> >> >> >> > Unable
> >> >> >> > >> to ping management server at 10.97.38.10:9090 due to
> >> >> ConnectException
> >> >> >> > >> java.net.ConnectException: Connection refused
> >> >> >> > >>
> >> >> >> > >> On Mon, Nov 3, 2014 at 9:02 AM, Nux! <nux@li.nux.ro> wrote:
> >> >> >> > >>
> >> >> >> > >> > BTW the agent logs should help clarify this.
> >> >> >> > >> > The aforementioned changes must be done either in the
> current
> >> >> >> > (broken) VR
> >> >> >> > >> > or in the systemvm (after which you should redeploy the VR
> to
> >> >> pick
> >> >> >> up
> >> >> >> > the
> >> >> >> > >> > change).
> >> >> >> > >> >
> >> >> >> > >> > HTH
> >> >> >> > >> > Lucian
> >> >> >> > >> >
> >> >> >> > >> > --
> >> >> >> > >> > Sent from the Delta quadrant using Borg technology!
> >> >> >> > >> >
> >> >> >> > >> > Nux!
> >> >> >> > >> > www.nux.ro
> >> >> >> > >> >
> >> >> >> > >> > ----- Original Message -----
> >> >> >> > >> > > From: "Nux!" <nux@li.nux.ro>
> >> >> >> > >> > > To: users@cloudstack.apache.org
> >> >> >> > >> > > Sent: Monday, 3 November, 2014 13:53:10
> >> >> >> > >> > > Subject: Re: How to Upgrade Virtual Router from 4.3 to
> 4.4.1
> >> >> >> > >> >
> >> >> >> > >> > > Hello,
> >> >> >> > >> > >
> >> >> >> > >> > > I think you are hitting a problem I also had:
> >> >> >> > >> > > "2014-11-03 08:16:17,024 ERROR
> >> >> [c.c.v.VirtualMachineManagerImpl]
> >> >> >> > >> > > (Work-Job-Executor-1:ctx-5c33d547 job-24555/job-24556
> >> >> >> ctx-fcb072b2)
> >> >> >> > >> > Failed to
> >> >> >> > >> > > start instance VM[DomainRouter|r-4758-VM]
> >> >> >> > >> > > com.cloud.utils.exception.ExecutionException: Unable to
> >> start
> >> >> >> > >> > > VM[DomainRouter|r-4758-VM] due to error in finalizeStart,
> >> not
> >> >> >> > retrying"
> >> >> >> > >> > >
> >> >> >> > >> > > It's
> https://issues.apache.org/jira/browse/CLOUDSTACK-7781
> >> >> >> > >> > >
> >> >> >> > >> > > Basically the systemvm template has some problems, you
> need
> >> to
> >> >> >> > somehow
> >> >> >> > >> > copy
> >> >> >> > >> > > /opt/cloud/bin/createIpAlias.sh to
> >> >> /opt/cloud/bin/createipAlias.sh
> >> >> >> > to
> >> >> >> > >> > get it
> >> >> >> > >> > > working.
> >> >> >> > >> > >
> >> >> >> > >> > > --
> >> >> >> > >> > > Sent from the Delta quadrant using Borg technology!
> >> >> >> > >> > >
> >> >> >> > >> > > Nux!
> >> >> >> > >> > > www.nux.ro
> >> >> >> > >> > >
> >> >> >> > >> > > ----- Original Message -----
> >> >> >> > >> > >> From: "Adam" <adam.scarcella@gmail.com>
> >> >> >> > >> > >> To: users@cloudstack.apache.org
> >> >> >> > >> > >> Sent: Monday, 3 November, 2014 13:32:12
> >> >> >> > >> > >> Subject: Re: How to Upgrade Virtual Router from 4.3 to
> >> 4.4.1
> >> >> >> > >> > >
> >> >> >> > >> > >> [root@cs-east-prod1 ~]# cat /etc/redhat-release
> >> >> >> > >> > >> CentOS release 6.6 (Final)
> >> >> >> > >> > >> [root@cs-east-prod1 ~]# uname -a
> >> >> >> > >> > >> Linux cs-east-prod1 2.6.32-504.el6.x86_64 #1 SMP Wed
> Oct 15
> >> >> >> > 04:27:16
> >> >> >> > >> UTC
> >> >> >> > >> > >> 2014 x86_64 x86_64 x86_64 GNU/Linux
> >> >> >> > >> > >> [root@cs-east-prod1 ~]# virsh --version
> >> >> >> > >> > >> 0.10.2
> >> >> >> > >> > >> [root@cs-east-prod1 ~]# sed -i 's/INFO/DEBUG/g'
> >> >> >> > >> > >> /etc/cloudstack/agent/log4j-cloud.xml
> >> >> >> > >> > >> [root@cs-east-prod1 ~]# service cloudstack-management
> stop
> >> >> >> > >> > >> Stopping cloudstack-management:
> >> [
> >> >> >> OK  ]
> >> >> >> > >> > >> [root@cs-east-prod1 ~]# mv
> >> >> >> > >> > >> /var/log/cloudstack/management/management-server.log
> >> >> >> > >> > >> /var/log/cloudstack/management/management-server.bak
> >> >> >> > >> > >> [root@cs-east-prod1 ~]# service cloudstack-management
> >> start
> >> >> >> > >> > >> Starting cloudstack-management:
> >> [
> >> >> >> OK  ]
> >> >> >> > >> > >> [root@cs-east-prod1 ~]#
> >> >> >> > >> > >>
> >> >> >> > >> > >> Please see attached ZIP of clean management-server.log
> in
> >> >> DEBUG
> >> >> >> > mode.
> >> >> >> > >> At
> >> >> >> > >> > >> exactly (*2014-11-03 08:15:35,700 on line 3025 of the
> >> attached
> >> >> >> > >> > >> management-server.log*) I tried starting the Virtual
> Router
> >> >> from
> >> >> >> > the
> >> >> >> > >> > >> Management UI and received this "Status" message in the
> UI.
> >> >> >> > >> > >>
> >> >> >> > >> > >> [image: Inline image 1]
> >> >> >> > >> > >>
> >> >> >> > >> > >> I can see in the management-server.log that while the
> >> >> >> > >> > cloudstack-management
> >> >> >> > >> > >> service is initializing, it throws the following ERROR
> at
> >> >> exactly
> >> >> >> > >> > (*2014-11-03
> >> >> >> > >> > >> 08:14:36,813 on line 342 of the attached
> >> >> management-server.log*):
> >> >> >> > >> > >>
> >> >> >> > >> > >> 2014-11-03 08:14:36,813 ERROR [c.c.c.ClusterManagerImpl]
> >> >> >> > (main:null)
> >> >> >> > >> > Unable
> >> >> >> > >> > >> to ping management server at 10.97.38.10:9090 due to
> >> >> >> > ConnectException
> >> >> >> > >> > >> java.net.ConnectException: Connection refused
> >> >> >> > >> > >>    at sun.nio.ch.Net.connect0(Native Method)
> >> >> >> > >> > >>    at sun.nio.ch.Net.connect(Net.java:465)
> >> >> >> > >> > >>    at sun.nio.ch.Net.connect(Net.java:457)
> >> >> >> > >> > >>    at
> >> >> >> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1123)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> com.cloud.cluster.ClusterManagerImpl.pingManagementNode(ClusterManagerImpl.java:1092)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> com.cloud.cluster.ClusterManagerImpl.checkConflicts(ClusterManagerImpl.java:1170)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> com.cloud.cluster.ClusterManagerImpl.configure(ClusterManagerImpl.java:1057)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$3.with(CloudStackExtendedLifeCycle.java:114)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:153)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.configure(CloudStackExtendedLifeCycle.java:110)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.start(CloudStackExtendedLifeCycle.java:56)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:167)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:51)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:339)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:143)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:108)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:945)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:145)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:122)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:117)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:70)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:57)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:61)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4210)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> >
> >> >>
> >>
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4709)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >>
> >> >> >>
> >> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
> >> >> >> > >> > >>    at
> >> >> >> > >> >
> >> >> >>
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1041)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:964)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> >
> >> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> >> >> >> > >> > >>    at
> >> >> >> > >>
> >> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> >> >> >> > >> > >>    at
> >> >> >> > >> >
> >> >> >>
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
> >> >> >> > >> > >>    at
> >> >> >> > >>
> >> org.apache.catalina.core.StandardHost.start(StandardHost.java:722)
> >> >> >> > >> > >>    at
> >> >> >> > >> >
> >> >> >>
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> >
> >> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >>
> >> >> >>
> >> org.apache.catalina.core.StandardService.start(StandardService.java:516)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> >
> >> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
> >> >> >> > >> > >>    at
> >> >> >> org.apache.catalina.startup.Catalina.start(Catalina.java:593)
> >> >> >> > >> > >>    at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >> Method)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> >> >> > >> > >>    at
> >> >> >> > >> > >>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> >> >> > >> > >>    at java.lang.reflect.Method.invoke(Method.java:606)
> >> >> >> > >> > >>    at
> >> >> >> > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> >> >> >> > >> > >>    at
> >> >> >> > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> >> >> >> > >> > >> 2014-11-03 08:14:36,816 INFO  [c.c.c.ClusterManagerImpl]
> >> >> >> > (main:null)
> >> >> >> > >> > >> Detected that another management node with the same IP
> >> >> >> 10.97.38.10
> >> >> >> > is
> >> >> >> > >> > >> considered as running in DB, however it is not
> pingable, we
> >> >> will
> >> >> >> > >> > continue
> >> >> >> > >> > >> cluster initialization with this management server node
> >> >> >> > >> > >>
> >> >> >> > >> > >> I'm not sure what that error means, but more than likely
> >> >> that's
> >> >> >> the
> >> >> >> > >> > reason
> >> >> >> > >> > >> why the Virtual Router will not start.
> >> >> >> > >> > >>
> >> >> >> > >> > >> * PLEASE HELP !  *
> >> >> >> > >> > >>
> >> >> >> > >> > >> Best Regards,
> >> >> >> > >> > >>
> >> >> >> > >> > >>
> >> >> >> > >> > >>
> >> >> >> > >> > >> Adam Scarcella
> >> >> >> > >> > >>
> >> >> >> > >> > >> On Sun, Nov 2, 2014 at 8:20 PM, Nux! <nux@li.nux.ro>
> >> wrote:
> >> >> >> > >> > >>
> >> >> >> > >> > >>> Hi,
> >> >> >> > >> > >>>
> >> >> >> > >> > >>> More agent and management logs please, make sure they
> are
> >> in
> >> >> >> > DEBUG[1]
> >> >> >> > >> > mode.
> >> >> >> > >> > >>>
> >> >> >> > >> > >>> [1] -
> >> >> >> > >> > >>>
> >> >> >> > >>
> >> >> >>
> >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
> >> >> >> > >> > >>>
> >> >> >> > >> > >>> --
> >> >> >> > >> > >>> Sent from the Delta quadrant using Borg technology!
> >> >> >> > >> > >>>
> >> >> >> > >> > >>> Nux!
> >> >> >> > >> > >>> www.nux.ro
> >> >> >> > >> > >>>
> >> >> >> > >> > >>> ----- Original Message -----
> >> >> >> > >> > >>> > From: "Adam" <adam.scarcella@gmail.com>
> >> >> >> > >> > >>> > To: users@cloudstack.apache.org
> >> >> >> > >> > >>> > Sent: Sunday, 2 November, 2014 23:33:22
> >> >> >> > >> > >>> > Subject: How to Upgrade Virtual Router from 4.3 to
> 4.4.1
> >> >> >> > >> > >>>
> >> >> >> > >> > >>> > I've performed a complete upgrade from 4.3 to 4.4.1
> and
> >> >> >> followed
> >> >> >> > >> the
> >> >> >> > >> > >>> > instructions perfectly, but my Virtual Router will
> not
> >> >> start,
> >> >> >> > nor
> >> >> >> > >> am
> >> >> >> > >> > I
> >> >> >> > >> > >>> able
> >> >> >> > >> > >>> > to "upgrade" it.
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > I tried destroying the Virtual Router, starting any
> VM
> >> >> >> Instance,
> >> >> >> > >> and
> >> >> >> > >> > >>> > allowing CS to recreate the VR. However that didn't
> >> help.
> >> >> The
> >> >> >> VR
> >> >> >> > >> > still
> >> >> >> > >> > >>> did
> >> >> >> > >> > >>> > not start and it was still saying that it required
> to be
> >> >> >> > upgraded.
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > [image: Inline image 1]
> >> >> >> > >> > >>> > [image: Inline image 2]
> >> >> >> > >> > >>> > [image: Inline image 3]
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > I know there are manual steps to finalize the
> systemvm
> >> >> >> upgrade,
> >> >> >> > but
> >> >> >> > >> > I did
> >> >> >> > >> > >>> > those steps:
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>>
> >> >> >> > >> >
> >> >> >> > >>
> >> >> >> >
> >> >> >>
> >> >>
> >>
> http://cloudstack-release-notes.readthedocs.org/en/latest/upgrade/upgrade-4.3.html#manual-hofix
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > [root@cs-east-prod1 cloudstack-logs]# mysql -u root
> -p
> >> >> >> > >> > >>> > Enter password:
> >> >> >> > >> > >>> > Welcome to the MySQL monitor.  Commands end with ; or
> >> \g.
> >> >> >> > >> > >>> > Your MySQL connection id is 174
> >> >> >> > >> > >>> > Server version: 5.1.73-log Source distribution
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > Copyright (c) 2000, 2013, Oracle and/or its
> affiliates.
> >> All
> >> >> >> > rights
> >> >> >> > >> > >>> reserved.
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > Oracle is a registered trademark of Oracle
> Corporation
> >> >> and/or
> >> >> >> > its
> >> >> >> > >> > >>> > affiliates. Other names may be trademarks of their
> >> >> respective
> >> >> >> > >> > >>> > owners.
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > Type 'help;' or '\h' for help. Type '\c' to clear the
> >> >> current
> >> >> >> > input
> >> >> >> > >> > >>> > statement.
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > mysql> select id,name from cloud.vm_template where
> name
> >> =
> >> >> >> > >> > >>> > 'systemvm-kvm-4.4';
> >> >> >> > >> > >>> > +-----+------------------+
> >> >> >> > >> > >>> >| id  | name             |
> >> >> >> > >> > >>> > +-----+------------------+
> >> >> >> > >> > >>> >| 274 | systemvm-kvm-4.4 |
> >> >> >> > >> > >>> > +-----+------------------+
> >> >> >> > >> > >>> > 1 row in set (0.00 sec)
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > mysql> select type from cloud.vm_template where
> >> id='274';
> >> >> >> > >> > >>> > +--------+
> >> >> >> > >> > >>> >| type   |
> >> >> >> > >> > >>> > +--------+
> >> >> >> > >> > >>> >| SYSTEM |
> >> >> >> > >> > >>> > +--------+
> >> >> >> > >> > >>> > 1 row in set (0.00 sec)
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > mysql> select distinct(vm_template_id) from
> >> >> cloud.vm_instance
> >> >> >> > where
> >> >> >> > >> > >>> > type='ConsoleProxy' and hypervisor_type = 'KVM';
> >> >> >> > >> > >>> > +----------------+
> >> >> >> > >> > >>> >| vm_template_id |
> >> >> >> > >> > >>> > +----------------+
> >> >> >> > >> > >>> >|            274 |
> >> >> >> > >> > >>> > +----------------+
> >> >> >> > >> > >>> > 1 row in set (0.00 sec)
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > mysql> select distinct(vm_template_id) from
> >> >> cloud.vm_instance
> >> >> >> > where
> >> >> >> > >> > >>> > type='SecondaryStorageVm' and hypervisor_type =
> 'KVM';
> >> >> >> > >> > >>> > +----------------+
> >> >> >> > >> > >>> >| vm_template_id |
> >> >> >> > >> > >>> > +----------------+
> >> >> >> > >> > >>> >|            274 |
> >> >> >> > >> > >>> > +----------------+
> >> >> >> > >> > >>> > 1 row in set (0.03 sec)
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > mysql> select value from cloud.configuration where
> name
> >> =
> >> >> >> > >> > >>> > 'router.template.kvm';
> >> >> >> > >> > >>> > +------------------+
> >> >> >> > >> > >>> >| value            |
> >> >> >> > >> > >>> > +------------------+
> >> >> >> > >> > >>> >| systemvm-kvm-4.4 |
> >> >> >> > >> > >>> > +------------------+
> >> >> >> > >> > >>> > 1 row in set (0.00 sec)
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > mysql>
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > * PLEASE HELP !  *
> >> >> >> > >> > >>> >
> >> >> >> > >> > >>> > Best Regards,
> >> >> >> > >> > >>> >
> >> >> >> > >> > > >> > Adam Scarcella
> >> >> >> > >> >
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Thanks for your support,
> >> >> >> Motty
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message