cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Darren Shepherd <darren.s.sheph...@gmail.com>
Subject Re: conflicting dependencies between CloudStack and Whirr
Date Mon, 23 Sep 2013 20:41:45 GMT
I think I agree, but I hate to shoot down a GSoC project.  Where is
the code for this currently?  I don't really know much about this
enhancement.  Regardless, I still want jackson.

Darren

On Mon, Sep 23, 2013 at 11:54 AM, Chiradeep Vittal
<Chiradeep.Vittal@citrix.com> wrote:
> I still think this is the wrong way to implement the Whirr integration.
>
> On 9/23/13 6:45 AM, "sebgoa" <runseb@gmail.com> wrote:
>
>>
>>On Sep 23, 2013, at 7:12 AM, Darren Shepherd
>><darren.s.shepherd@gmail.com> wrote:
>>
>>> The only way this will get committed is if we either move to latest
>>>gson or
>>> jackson.  Regardless, the outcome will be that Meng can assume this gson
>>> conflict won't happen.  The problem is just how fast can we move all of
>>>ACS
>>> to a new json library.  Is it possible for Meng to commit to a branch
>>>that
>>> just blindly updates gson to the latest (in /pom.xml).
>>
>>Yes I will talk to Meng about this. thanks
>>
>>> When we get ACS on
>>> a newer json library, we can merge in that branch.  Tomorrow (9/23), I
>>>can
>>> attempt to put together a patch to move to jackson.  I think I ironed
>>>out
>>> most of the technical issues so I just have to globally apply the
>>>changes.
>>>
>>> Darren
>>>
>>>
>>> On Fri, Sep 20, 2013 at 1:09 AM, Sebastien Goasguen
>>><runseb@gmail.com>wrote:
>>>
>>>>
>>>> On Sep 20, 2013, at 12:45 AM, Hugo Trippaers <hugo@trippaers.nl> wrote:
>>>>
>>>>>
>>>>> The Nicira NVP plugin is also depending on gson. If we make any
>>>>>changes
>>>> we need to validate against their api to ensure that the interface
>>>>still
>>>> works.
>>>>>
>>>>> If we put the changes in a separate branch i'd be happy to run the
>>>> changes against the api.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Hugo
>>>>
>>>> So how can Meng get out of this bind, she is trying to complete her
>>>>google
>>>> summer of code project.
>>>>
>>>> thanks,
>>>>
>>>> -Sebastien
>>>>
>>>>>
>>>>>
>>>>> On Sep 20, 2013, at 12:08 PM, Darren Shepherd <
>>>> darren.s.shepherd@gmail.com> wrote:
>>>>>
>>>>>> After much searching I found the two lines of code I needed to get
it
>>>> to serialize right.  I'll keep looking and see if I find other snags.
>>>> Maybe we can just move to jackson.... we'll see, haven't tried to
>>>> deserialize yet.
>>>>>>
>>>>>> Darren
>>>>>>
>>>>>>> On Sep 19, 2013, at 4:25 PM, Alex Huang <Alex.Huang@citrix.com>
>>>>>>>wrote:
>>>>>>>
>>>>>>> Darren,
>>>>>>>
>>>>>>> When I looked into updating to the latest gson, the problem,
IIRC,
>>>>>>>is
>>>> things that weren't considered cyclical dependencies are suddenly
>>>> considered cyclical with the latest.  I don't recall where though.
>>>>>>>
>>>>>>> --Alex
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>>>>> Sent: Thursday, September 19, 2013 1:33 PM
>>>>>>>> To: dev@cloudstack.apache.org
>>>>>>>> Subject: Re: conflicting dependencies between CloudStack
and Whirr
>>>>>>>>
>>>>>>>> Alright, I looked into this and it will take a bit more work
to
>>>> switch to Jackson.
>>>>>>>> The snag I hit was how we serialize commands for the agents.
 We
>>>>>>>>use a
>>>>>>>> customer typeadatper in gson to produce a format like {
>>>>>>>>"org.MyClass"
>>>>>>>> : {...} }.  In jackson I don't see producing a similar format
as
>>>> being so straight
>>>>>>>> forward.  I'm sure there's an elegant solution, but I didn't
>>>> immediately find it.
>>>>>>>> The problem is if you use a custom serializer in jackson
and do
>>>>>>>> writeObjectField(x.getClass().getName(), x), you'll get stuck
in a
>>>> loop
>>>>>>>> because it will call your serializer again.  So if somebody
can
>>>> figure out how to
>>>>>>>> do this format easily in jackson, the rest looks simple enough.
>>>>>>>>
>>>>>>>> So, being that it is not an obvious switch to using jackson,
what
>>>>>>>>is
>>>> the impact
>>>>>>>> of moving to the latest gson?  What were the issue encountered
when
>>>>>>>> people tried it before?
>>>>>>>>
>>>>>>>> Darren
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Sep 18, 2013 at 4:42 PM, Darren Shepherd <
>>>>>>>> darren.s.shepherd@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I can do some analysis on this.  I'm always up for a
terribly
>>>>>>>>>painful
>>>>>>>>> refactor :)
>>>>>>>>>
>>>>>>>>> Darren
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Wed, Sep 18, 2013 at 4:06 PM, Alex Huang
>>>>>>>>>><Alex.Huang@citrix.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I actually did a quick try to update cloudstack to
use newest
>>>>>>>>>>gson
>>>>>>>>>> version about 3 months back.  Had to roll it back
but I didn't
>>>>>>>>>>try
>>>>>>>>>> very hard though.  Part of the reason why I decided
to rollback
>>>>>>>>>>is
>>>>>>>>>> due to gson is used differently by various components
in
>>>>>>>>>>CloudStack
>>>>>>>>>> and I didn't have time to get into each component
to see if I
>>>>>>>>>>break
>>>>>>>>>> anything.  The other is also I thought using Jackson
would be
>>>>>>>>>>much
>>>>>>>>>> better and thought our next attempt should be with
Jackson.
>>>>>>>>>>
>>>>>>>>>> Not that any of these helps you.  Just know updating
CloudStack
>>>>>>>>>>to
>>>>>>>>>> latest gson is not simple.
>>>>>>>>>>
>>>>>>>>>> --Alex
>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Andrei Savu [mailto:savu.andrei@gmail.com]
>>>>>>>>>>> Sent: Wednesday, September 18, 2013 10:53 AM
>>>>>>>>>>> To: dev@whirr.apache.org
>>>>>>>>>>> Cc: dev@cloudstack.apache.org
>>>>>>>>>>> Subject: Re: conflicting dependencies between
CloudStack and
>>>>>>>>>>>Whirr
>>>>>>>>>>>
>>>>>>>>>>> It's easy to usr jclouds and whirr inside an
OSGi container -
>>>>>>>>>>>just
>>>>>>>>>>> add
>>>>>>>>>> the
>>>>>>>>>>> feature url. Bonus: you can also use jclouds
shell interface
>>>>>>>>>>>(part
>>>>>>>>>>> of
>>>>>>>>>> jclouds cli).
>>>>>>>>>>>
>>>>>>>>>>> Another option is to upgrade the CloudStack API
to use the new
>>>> version.
>>>>>>>>>>>> On Sep 18, 2013 5:14 AM, "Han,Meng" <menghan@ufl.edu>
wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>
>>>>>>>>>>>> I am adding an API to CloudStack which utilizes
Whirr to launch
>>>>>>>>>>>> various clusters on CloudStack. Now I am
facing a dependency
>>>>>>>>>> conflicting
>>>>>>>>>>> issue.
>>>>>>>>>>>>
>>>>>>>>>>>> Whirr 0.8.2 requires gson 2.2.2 while CloudStack
API requires
>>>>>>>>>>>> gson
>>>>>>>>>> 1.7.1.
>>>>>>>>>>>> If I use gson 1.7.1 for Whirr, the following
error will happen:
>>>>>>>>>>>>
>>>>>>>>>>>> com.google.common.util.**concurrent.ExecutionError:
>>>>>>>>>>> java.lang.**NoClassDefFoundError:
>>>>>>>>>>>> com/google/gson/TypeAdapter
>>>>>>>>>>>>
>>>>>>>>>>>> This TypeAdapter class can be found inside
gson-2.2.2.jar.
>>>>>>>>>>>> However if I modify CloudStack to use gson
2.2.2 CloudStack
>>>>>>>>>>>>will
>>>>>>>>>>>> not build successfully due to the following
test error.
>>>>>>>>>>>>
>>>>>>>>>>>> [INFO] Building Apache CloudStack Core 4.1.1
[INFO]
>>>>>>>>>>>>
>>>>>>>>>>>>------------------------------**------------------------------**
>>>>>>>>>>>> ------------
>>>>>>>>>>>> [INFO]
>>>>>>>>>>>> [INFO] --- maven-clean-plugin:2.5:clean (default-clean)
@
>>>>>>>>>>>> cloud-core
>>>>>>>>>>>> --- [INFO] Deleting /home/meng/cloudstack/core/**target
>>>>>>>>>>>> [INFO]
>>>>>>>>>>>> [INFO] --- maven-remote-resources-plugin:**1.3:process
>>>>>>>>>>>>(default)
>>>>>>>>>>>> @ cloud-core --- [INFO] [INFO] ---
>>>>>>>>>>>> maven-resources-plugin:2.5:**resources (default-resources)
@
>>>>>>>>>>>> cloud-core --- [debug] execute contextualize
[INFO] Using
>>>>>>>>>>>>'UTF-8'
>>>>>>>>>>>> encoding to copy filtered resources.
>>>>>>>>>>>> [INFO] skip non existing resourceDirectory
>>>>>>>>>>>> /home/meng/cloudstack/core/** src/main/resources
[INFO] Copying
>>>>>>>> 3
>>>>>>>>>>>> resources [INFO] [INFO] ---
>>>>>>>>>>>>maven-compiler-plugin:2.5.1:**compile
>>>>>>>>>>>> (default-compile) @ cloud-core --- [INFO]
Compiling 156 source
>>>>>>>>>>>> files to /home/meng/cloudstack/core/** target/classes
[INFO]
>>>>>>>>>>>> [INFO] --- maven-resources-plugin:2.5:**testResources
>>>>>>>>>>>> (default-testResources) @ cloud-core ---
[debug] execute
>>>>>>>>>>>> contextualize [INFO] Using 'UTF-8' encoding
to copy filtered
>>>> resources.
>>>>>>>>>>>> [INFO] skip non existing resourceDirectory
>>>>>>>>>>>> /home/meng/cloudstack/core/** src/test/resources
[INFO] Copying
>>>>>>>> 3
>>>>>>>>>>>> resources [INFO] [INFO] ---
>>>>>>>>>>>> maven-compiler-plugin:2.5.1:**testCompile
>>>>>>>>>>>> (default-testCompile) @ cloud-core --- [INFO]
Compiling 1
>>>>>>>>>>>>source
>>>>>>>>>>>> file to /home/meng/cloudstack/core/** target/test-classes
>>>>>>>>>>>>[INFO]
>>>>>>>>>>>> [INFO] --- maven-surefire-plugin:2.12:**test
(default-test) @
>>>>>>>>>>>> cloud-core
>>>>>>>>>>>> ---
>>>>>>>>>>>> [INFO] Surefire report directory: /home/meng/cloudstack/core/**
>>>>>>>>>>>> target/surefire-reports
>>>>>>>>>>>>
>>>>>>>>>>>> ------------------------------**-------------------------
>>>>>>>>>>>> T E S T S
>>>>>>>>>>>> ------------------------------**-------------------------
>>>>>>>>>>>> Running com.cloud.agent.transport.**RequestTest
>>>>>>>>>>>> log4j:WARN No appenders could be found for
logger
>>>>>>>>>>>> (com.cloud.agent.transport.**RequestTest).
>>>>>>>>>>>> log4j:WARN Please initialize the log4j system
properly.
>>>>>>>>>>>> log4j:WARN See
>>>>>>>>>>> http://logging.apache.org/**log4j/1.2/faq.html#noconfig<
>>>>>>>>>> http://logging.apa
>>>>>>>>>>> che.org/log4j/1.2/faq.html#noconfig>for more
info.
>>>>>>>>>>>> Tests run: 4, Failures: 1, Errors: 1, Skipped:
0, Time elapsed:
>>>>>>>>>>>> 3.054 sec <<< FAILURE!
>>>>>>>>>>>>
>>>>>>>>>>>> Results :
>>>>>>>>>>>>
>>>>>>>>>>>> Failed tests:
>>>> testSerDeser(com.cloud.agent.**transport.RequestTest)
>>>>>>>>>>>>
>>>>>>>>>>>> Tests in error:
>>>>>>>>>>>> testLogging(com.cloud.agent.**transport.RequestTest)
>>>>>>>>>>>>
>>>>>>>>>>>> Tests run: 4, Failures: 1, Errors: 1, Skipped:
0
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I ran "mvn -P developer clean install -DskipTests"
, the
>>>>>>>>>>>> CloudStack building process went through.
But when I issued an
>>>>>>>>>>>> API --launchCluster to CloudStack, the following
error related
>>>>>>>>>>>>to
>>>>>>>>>>>> gson popped up on CloudStack Management Server:
>>>>>>>>>>>>
>>>>>>>>>>>> ERROR [agent.transport.Request] (AgentManager-Handler-2:)
>>>>>>>>>>>>Caught
>>>>>>>>>>>> problem with
>>>>>>>>>>>> [{"StartupRoutingCommand":{"**cpus":2,"speed":3000,"memory":**
>>>>>>>>>>>> 3844370432,"dom0MinMemory":**384437043,"poolSync":false,"**
>>>>>>>>>>>> vms":{"v-2-VM":{"state":"**Running"},"s-1-VM":{"state":"**
>>>>>>>>>>>> Running"},"r-4-VM":{"state":"**Running"}},"caps":"hvm,**
>>>>>>>>>>>> snapshot","pool":"/root","**hypervisorType":"KVM","**
>>>>>>>>
>>>>>>>>hostDetails":{"com.cloud.**network.Networks.**RouterPrivateIpStrateg
>>>>>>>>y"
>>>>>>>>>>>> :"**
>>>>>>>>>>>>
>>>>>>>>>>>>HostLocal","Host.OS":"CentOS",**"Host.OS.Kernel.Version":"2.6.**
>>>>>>>>>>>> 32-358.el6.x86_64","Host.OS.**Version":"6.4"},"type":"**
>>>>>>>>>>>> Routing","dataCenter":"1","**pod":"1","cluster":"1","guid":**
>>>>>>>>>>>> "a7320748-6346-3c9a-975e-**90ac4ae4a986-
>>>>>>>>>>> **LibvirtComputingResource","*
>>>>>>>>>>>> *
>>>>>>>>>>>> name":"meng.acis.ufl.edu","id"**:2,"version":"4.1.1","**
>>>>>>>>>>>> publicIpAddress":"10.244.18.**55","publicNetmask":"255.0.0.**
>>>>>>>>>>>> 0","publicMacAddress":"00:23:**ae:94:f7:22","**
>>>>>>>>>>>> privateIpAddress":"10.244.18.**55","privateMacAddress":"00:**
>>>>>>>>>>>> 23:ae:94:f7:22","**privateNetmask":"255.0.0.0","**
>>>>>>>>>>>> storageIpAddress":"10.244.18.**55","storageNetmask":"255.0.0.**
>>>>>>>>>>>> 0","storageMacAddress":"00:23:**ae:94:f7:22","resourceName":"**
>>>>>>>>>>>> LibvirtComputingResource","**gatewayIpAddress":"!
>>>>>>>>>>>> 10.244.18
>>>>>>>>>>>> .1","contextMap":{},"wait":0}}**,{"StartupStorageCommand":{"**
>>>>>>>>>>>> totalSize":0,"poolInfo":{"**uuid":"9447c0b1-cc3f-439f-**
>>>>>>>>>>>> 85f6-13d35539a9ed","host":"10.**244.18.55","localPath":"/var/**
>>>>>>>>>>>> lib/libvirt/images/","**hostPath":"/var/lib/libvirt/**
>>>>>>>>>>>> images/","poolType":"**Filesystem","capacityBytes":**
>>>>>>>>>>>> 52844687360,"availableBytes":**41535332352},"resourceType":"**
>>>>>>>>>>>>
>>>>>>>>>>>>STORAGE_POOL","hostDetails":{}**,"type":"Storage","dataCenter"**
>>>>>>>>>>>>
>>>>>>>>>>>>:"1","pod":"1","guid":"**a7320748-6346-3c9a-975e-**90ac4ae4a986-
>>>>>>>>>>>>*
>>>>>>>>>>>> *
>>>>>>>>>>>> LibvirtComputingResource","**name":"meng.acis.ufl.edu","id"**
>>>>>>>>>>>>
>>>>>>>>>>>>:2,"version":"4.1.1","**resourceName":"**LibvirtComputingResourc
>>>>>>>>>>>>e
>>>>>>>>>>>> ","**
>>>>>>>>>>>> contextMap":{},"wait":0}}]
>>>>>>>>>>>> java.lang.ClassCastException:
>>>>>>>>>>>> com.google.gson.internal.$**Gson$Types$**GenericArrayTypeImpl
>>>>>>>>>>>> cannot be cast to java.lang.Class
>>>>>>>>>>>>     at
>>>>>>>>>>>> com.cloud.agent.transport.**ArrayTypeAdaptor.deserialize(**
>>>>>>>>>>>> ArrayTypeAdaptor.java:84)
>>>>>>>>>>>>     at
>>>>>>>>>>>> com.cloud.agent.transport.**ArrayTypeAdaptor.deserialize(**
>>>>>>>>>>>> ArrayTypeAdaptor.java:37)
>>>>>>>>>>>>     at com.google.gson.**TreeTypeAdapter.read(**
>>>>>>>>>>>> TreeTypeAdapter.java:58)
>>>>>>>>>>>>     at com.google.gson.Gson.fromJson(**Gson.java:795)
>>>>>>>>>>>>     at
>>>>>>>>>>>> com.cloud.agent.transport.**Request.getCommands(Request.**
>>>>>>>>>>>> java:235)
>>>>>>>>>>>>     at
>>>>>>>>>>>> com.cloud.agent.manager.**AgentManagerImpl$AgentHandler.**
>>>>>>>>>>>> processRequest(**AgentManagerImpl.java:1221)
>>>>>>>>>>>>     at
>>>>>>>>>>>> com.cloud.agent.manager.**AgentManagerImpl$AgentHandler.**
>>>>>>>>>>>> doTask(AgentManagerImpl.java:**1374)
>>>>>>>>>>>>     at com.cloud.agent.manager.**ClusteredAgentManagerImpl$**
>>>>>>>> ClusteredAgentHandler.doTask(**ClusteredAgentManagerImpl.**java:659
>>>>>>>>>>> )
>>>>>>>>>>>>     at com.cloud.utils.nio.Task.run(**Task.java:83)
>>>>>>>>>>>>     at java.util.concurrent.**ThreadPoolExecutor.runWorker(**
>>>>>>>>>>>> ThreadPoolExecutor.java:1110)
>>>>>>>>>>>>     at
>>>>>>>>>>>> java.util.concurrent.**ThreadPoolExecutor$Worker.run(**
>>>>>>>>>>>> ThreadPoolExecutor.java:603)
>>>>>>>>>>>>     at java.lang.Thread.run(Thread.**java:722)
>>>>>>>>>>>> WARN  [utils.nio.Task] (AgentManager-Handler-2:)
Caught the
>>>>>>>>>>>> following exception but pushing on
>>>>>>>>>>>>
>>>>>>>>>>>> On the CS Managment console I can see the
cluster was started
>>>>>>>>>>>> then quickly died.
>>>>>>>>>>>> Running on provider cloudstack using identity
>>>>>>>> h3DKHC9AVlhKnUhpyThMuLhC119QfN**QQ8xhyjbf_**rnu5ZL1QeOWdw7a
>>>>>>>>>>> ZRGXVO1VApG
>>>>>>>>>>>> 6q0a
>>>>>>>>>>>> **K-A-tQRQsZFwnOXQ
>>>>>>>>>>>> INFO  [whirr.actions.**BootstrapClusterAction]
>>>>>>>>>>>> (729061754@qtp-385354117-0:) Bootstrapping
cluster INFO
>>>>>>>>>>>> [whirr.compute.**BootstrapTemplate]
>>>>>>>>>>>>(729061754@qtp-385354117-0:)
>>>>>>>>>>>> Configuring template for
>>>>>>>>>>>> bootstrap-hadoop-datanode_**hadoop-tasktracker
>>>>>>>>>>>> INFO  [whirr.compute.**BootstrapTemplate]
>>>>>>>>>>>> (729061754@qtp-385354117-
>>>>>>>>>>> 0:)
>>>>>>>>>>>> Configuring template for bootstrap-hadoop-namenode_**hadoop-
>>>>>>>>>>> jobtracker
>>>>>>>>>>>> INFO  [whirr.compute.NodeStarter] (pool-4-thread-2:)
Starting 1
>>>>>>>>>>>> node(s) with roles [hadoop-datanode, hadoop-tasktracker]
INFO
>>>>>>>>>>>> [whirr.compute.NodeStarter] (pool-4-thread-4:)
Starting 1
>>>>>>>>>>>>node(s)
>>>>>>>>>>>> with roles [hadoop-namenode, hadoop-jobtracker]
...
>>>>>>>>>>>> INFO  [whirr.actions.**DestroyClusterAction]
>>>>>>>>>>>> (729061754@qtp-385354117-0:) Cluster hadoop
destroyed
>>>>>>>>>>>>
>>>>>>>>>>>> I attached the debuging log to this email.
>>>>>>>>>>>>
>>>>>>>>>>>> The launchCluster API is simple, it calls
the
>>>>>>>>>>>> LaunchClusterCommand in Whirr to launch a
cluster.
>>>>>>>>>>>>
>>>>>>>>>>>> LaunchClusterResponse response = new LaunchClusterResponse();
>>>>>>>>>>>>     response.setObjectName("**launchCluster");
>>>>>>>>>>>>     LaunchClusterCommand command = null;
>>>>>>>>>>>>     try {
>>>>>>>>>>>>         command = new LaunchClusterCommand();
>>>>>>>>>>>>     } catch (Exception ex) {
>>>>>>>> Logger.getLogger(**LaunchClusterCmd.class.**getName()).log(Level.SE
>>>>>>>>>>> VER
>>>>>>>>>>>> E,
>>>>>>>>>>>> null, ex);
>>>>>>>>>>>>     }
>>>>>>>>>>>>     String[] args = new String[2];
>>>>>>>>>>>>     args[0] = "--config";
>>>>>>>>>>>>     args[1] = config;
>>>>>>>>>>>>
>>>>>>>>>>>>     try {
>>>>>>>>>>>>         command.run(System.in, System.out,
System.err,
>>>>>>>>>>>> Arrays.asList(args));
>>>>>>>>>>>>     } catch (Exception ex) {
>>>>>>>> Logger.getLogger(**LaunchClusterCmd.class.**getName()).log(Level.SE
>>>>>>>>>>> VER
>>>>>>>>>>>> E,
>>>>>>>>>>>> null, ex);
>>>>>>>>>>>>     }
>>>>>>>>>>>>      response.setResponseName(**getCommandName());
>>>>>>>>>>>>      output = "successfully launched the
cluster.";
>>>>>>>>>>>>      response.setOutPut(output);
>>>>>>>>>>>>      response.setAsync(Boolean.**FALSE);
>>>>>>>>>>>>      this.setResponseObject(**response);
>>>>>>>>>>>>
>>>>>>>>>>>> Could someone help me out here? What would
be a proper gson
>>>>>>>>>>>> version for CloudStack and Whirr to run at
the same time?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks loads.
>>>>>>>>>>>>
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Meng
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>
>>>>
>>
>

Mime
View raw message