airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saminda Wijeratne <samin...@gmail.com>
Subject Re: Suggestions on maintaining the server/client configuration files?
Date Sat, 05 Apr 2014 00:56:20 GMT
If thats the case then we are good to go with the current model. I will
remove the configuration files from src/test/resources, add the
server/client configuration module as a maven dependency in test scope and
run the tests. If everything works fine I will commit this change.

myproxy credentials are read through our ServerSettings class. If these
details need to be overridden from values set at system properties, I can
update the ServerSettings class to check the system properties first before
checking the server/client property file. wdyt?


On Fri, Apr 4, 2014 at 11:14 AM, Lahiru Gunathilake <glahiru@gmail.com>wrote:

> Hi saminda,
>
> For unit tests we are only going to change credential related parameters
> to check certain funcationality which we cannot test with localhost (ex:
> pbs related stuff, slurm related stuff etc). In that case we only need to
> change credentials which can be read through system properties (these
> properties can be given in the maven command). In this case we can unify
> all the test cases to read same properties and all the grid tests should
> run with one configuration change (in command).
>
> If we are only changing credentials during unit tests, we should be good
> with current model I guess.
>
>
> On Fri, Apr 4, 2014 at 1:02 PM, Saminda Wijeratne <samindaw@gmail.com>wrote:
>
>>
>>
>> A new properties file (say "custom.properties") would be introduced
>> containing only the settings that are different from the shipped version of
>> the server or client properties file. At runtime the Settings class will
>> load the server/client properties file as usual and afterwards merge with
>> the settings defined in custom.properties file. Each module will have its
>> own custom.properties file defined for unit tests if they have settings
>> that would different compared to the default valued. Thus the
>> custom.properties file will act overloading and overriding the default
>> server/client properties file in the classpath. wdyt?
>>
> if we have multiple modules, before running tests we have to modify every
> place (because we cannot commit credentials to the repo, and at this point
> we only need to change credentials).
>
> WDYT ?
>
>>
>>
>>
>> On Fri, Apr 4, 2014 at 2:18 AM, Saminda Wijeratne <samindaw@gmail.com>wrote:
>>
>>> hi devs,
>>>
>>> I refactored the trunk code to link airavata-server.properties and
>>> airavata-client.properties from 2 maven modules now included in the trunk.
>>>  modules/configurations/server
>>> modules/configurations/client
>>>
>>> airavata-server.properties is used by the "airavata-standalone-server"
>>> module (@ modules/server) and in the server distribution.
>>> airavata-client.properties was used in,
>>> modules/orchestrator/orchestrator-client-sdks
>>> modules/xbaya-gui
>>> modules/distribution/xbaya-gui
>>> modules/distribution/airavata-client
>>> modules/airavata-client
>>> airavata-api/airavata-client-sdks/java-client-samples
>>>
>>> However I was not able to carry out similar merge for the properties
>>> files used in the tests since I kept hitting on broken unit tests. I will
>>> try to fix them first along with the issue reported by lahiru [2].
>>>
>>> Saminda
>>>
>>> 1. https://issues.apache.org/jira/browse/AIRAVATA-1121
>>> 2. https://issues.apache.org/jira/browse/AIRAVATA-1119
>>>
>>>
>>>
>>> On Mon, Mar 31, 2014 at 6:40 AM, Lahiru Gunathilake <glahiru@gmail.com>wrote:
>>>
>>>> I think we already have it in gfac-schemas and using tool like this[1],
>>>> we can quickly generate forms without worrying about changing them when the
>>>> schema changes.
>>>>
>>>> But I think we are going to do a new Application Catalog and once its
>>>> done we can use some tool to do the same thing.
>>>>
>>>> [1]https://code.google.com/p/xsd-web-forms/
>>>> Regards
>>>> Lahiru
>>>>
>>>>
>>>> On Mon, Mar 31, 2014 at 9:31 AM, Sachith Withana <swsachith@gmail.com>wrote:
>>>>
>>>>> Why don't we have some kind of Util module to store all these config
>>>>> files and have a rest-kind of class which acts as the middle layer between
>>>>> airavata modules and config files?
>>>>> All the config files are in one place, have a programmatic method of
>>>>> accessing them.
>>>>>
>>>>> IMO we should have a different set of config files for the Integration
>>>>> tests,
>>>>> 1. It would test the system using the thrift 'client' using external
>>>>> configurations files
>>>>> 2. since it is a main entry point for a gateway/airavata dev, it will
>>>>> show a clearer image on  which configurations we should use when we are
>>>>> using the client (it would only be the URLs and the ports for now).
>>>>>  3. It would eliminate any airavata-internal dependencies for the
>>>>> integration tests.
>>>>>
>>>>>
>>>>> On Mon, Mar 31, 2014 at 1:16 AM, Saminda Wijeratne <samindaw@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> It depends upon the test. If someone adds or updates configuration
>>>>>> data related to the correct functioning of the system or a component
it is
>>>>>> very likely that those configuration needs to be present for unit
tests as
>>>>>> well in case the unit tests depends upon those other components.
Thus it
>>>>>> becomes necessary to update all the test configuration files (to
be on the
>>>>>> safe side).
>>>>>>  On Mar 30, 2014 8:29 PM, "Lahiru Gunathilake" <glahiru@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Saminda,
>>>>>>>
>>>>>>> Ideally even at this point we should be able to remove all the
>>>>>>> configuration files from src/main/resources but for tests we
might need the
>>>>>>> files in src/test/resources.
>>>>>>>
>>>>>>> good thing about this is we have all the configuration in one
place
>>>>>>> (airavata-server.properties) but bad thing is we have to keep
it in each
>>>>>>> module's test configuration.
>>>>>>>
>>>>>>> Is this really a problem ?
>>>>>>>
>>>>>>> Regards
>>>>>>> Lahiru
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Mar 30, 2014 at 9:59 PM, Saminda Wijeratne <
>>>>>>> samindaw@gmail.com> wrote:
>>>>>>>
>>>>>>>> In another related note I found few places which has additional
>>>>>>>> property file types present in our trunk, some examples are,
>>>>>>>>
>>>>>>>> tools/job-monitor/src/main/resources/gsissh.properties
>>>>>>>> tools/job-monitor/src/test/resources/monitor.properties
>>>>>>>> tools/phoebus-integration/src/main/resources/service.properties
>>>>>>>> modules/commons/gfac-schema/src/main/resources/datatype.properties
>>>>>>>>
>>>>>>>> modules/commons/workflow-tracking/src/main/resources/log4j.properties
>>>>>>>>
>>>>>>>> modules/orchestrator/airavata-orchestrator-service/src/test/resources/orchestrator.properties
>>>>>>>>
>>>>>>>> modules/ws-messenger/messagebroker/src/test/resources/unit_tests.properties
>>>>>>>> modules/test-suite/src/test/resources/unit_test.properties
>>>>>>>> modules/xbaya-gui/src/main/resources/airavata-client.properties
>>>>>>>> modules/xbaya-gui/src/test/resources/airavata-server.properties
>>>>>>>> modules/gfac/gfac-core/src/main/resources/errors.properties
>>>>>>>> modules/gfac/gfac-core/src/main/resources/service.properties
>>>>>>>> modules/gfac/gfac-core/src/test/resources/logging.properties
>>>>>>>> modules/gfac/gfac-core/src/test/resources/airavata-client.properties
>>>>>>>>
>>>>>>>> I remember we raised an issue relating to having component
property
>>>>>>>> files but not coming to a conclusion regarding it. Any thoughts
about it?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 27, 2014 at 9:23 PM, Saminda Wijeratne <
>>>>>>>> samindaw@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> One way which so far seems to work is to define 2 maven
modules
>>>>>>>>> airavata-server-configuration and airavata-client-configurations
as jar
>>>>>>>>> artifacts where each will have the respective configuration
files. Then use
>>>>>>>>> them as dependencies for the modules which require the
relevant
>>>>>>>>> configurations. For the binary distributions we will
have to use the
>>>>>>>>> assembly-plugin and/or the dependency-plugin to copy
the configuration
>>>>>>>>> files in to the correct location in the distribution
directory.
>>>>>>>>>
>>>>>>>>> I'm still working on how this solution can be adhered
when running
>>>>>>>>> unit tests. At the moment the the duplicated configuration
files in
>>>>>>>>> src/test/resources only has a few changes compared to
the distributed
>>>>>>>>> configurations. Any thoughts?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Mar 26, 2014 at 10:22 PM, Saminda Wijeratne <
>>>>>>>>> samindaw@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> A little update on the efforts on this...
>>>>>>>>>>
>>>>>>>>>>    - Like Amila suggested, ServerSettings now by
default
>>>>>>>>>>    supports parameterization
>>>>>>>>>>    - settings can be added/updated by passing the
new settings
>>>>>>>>>>    via command line (and incidentally passing them
to the static func.
>>>>>>>>>>    ServerMain.main(...)).
>>>>>>>>>>       - $ ./airavata-server.sh --myproxy.user=saminda
>>>>>>>>>>       --myproxy.pass=pass123
>>>>>>>>>>       - Introduced ServerSettings.mergeSettings...(...)
>>>>>>>>>>    functions to help add/update settings through
Maps, command line args,
>>>>>>>>>>    additional prop files.
>>>>>>>>>>
>>>>>>>>>> The issue that I'm still facing is the duplication
of the
>>>>>>>>>> airavata-server.properties configuration file in
the code base. Last time I
>>>>>>>>>> checked copies of it is spread out at 14 locations
in the trunk (for
>>>>>>>>>> running junit tests, in standalone server dist, in
integration tests, in
>>>>>>>>>> sample gateway).
>>>>>>>>>>
>>>>>>>>>>    - config files in a single location in the trunk
and refer to
>>>>>>>>>>    them using relative paths:
>>>>>>>>>>       - done using the maven-builder-helper plugin.
>>>>>>>>>>       - But it seems the IDEs doesn't recognize resources
>>>>>>>>>>       outside the project folder.
>>>>>>>>>>    - "git subtrees" (which allows two way code sharing)
similar
>>>>>>>>>>    to "svn externals":
>>>>>>>>>>    - the limitations in it (configs needs to be in
a different
>>>>>>>>>>       repo, subtree will be merged on the root, cannot
specifically select which
>>>>>>>>>>       code to share?), it seems its not a viable
option.
>>>>>>>>>>
>>>>>>>>>> any ideas?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 6, 2014 at 1:55 PM, Amila Jayasekara
<
>>>>>>>>>> thejaka.amila@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Saminda,
>>>>>>>>>>>
>>>>>>>>>>> I guess the best thing is to create template
configuration files
>>>>>>>>>>> in a central location and replace needed variables
within the test
>>>>>>>>>>> initialization and place generated config files
within the target directory
>>>>>>>>>>> of the test case.
>>>>>>>>>>>
>>>>>>>>>>> E.g :- registry.url = ${registry.url}
>>>>>>>>>>>
>>>>>>>>>>> ${registry.url} is replaced within the test case
with
>>>>>>>>>>> appropriate value. Then we dont need to duplicate
configuration files in
>>>>>>>>>>> everywhere. Also when a configuration is updated
we only need to change in
>>>>>>>>>>> a single place.
>>>>>>>>>>> You may encapsulate template parameter replacing
logic into a
>>>>>>>>>>> common util class so that each test case can
just invoke the logic.
>>>>>>>>>>>
>>>>>>>>>>> Further we should have a single place to read
all configurations
>>>>>>>>>>> and all subcomponents must go through this configuration
component to get
>>>>>>>>>>> config values. Otherwise it will be hard to maintain
the configuration
>>>>>>>>>>> reading code.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Amila
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 6, 2014 at 1:38 PM, Saminda Wijeratne
<
>>>>>>>>>>> samindaw@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> There are several places which we need the
>>>>>>>>>>>> airavata-server.properties and airavata-client.properties
files to be
>>>>>>>>>>>> present in order to run tests and standalone
servers resulting in
>>>>>>>>>>>> replication. Any idea how we should handle
this?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> System Analyst Programmer
>>>>>>> PTI Lab
>>>>>>> Indiana University
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Sachith Withana
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> System Analyst Programmer
>>>> PTI Lab
>>>> Indiana University
>>>>
>>>
>>>
>>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
>

Mime
View raw message