manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: MCF 2.x binary package directory structure
Date Fri, 26 Sep 2014 11:04:20 GMT
Hi Abe-san,

For any of the three tickets related to your original post
(CONNECTORS-1048, CONNECTORS-345, CONNECTORS-1049), if you want to assign
the ticket to yourself, and propose an official patch for just that issue,
I'll be happy to review it and test it out.  Otherwise, I'll try to get to
these sometime soon.

Karl


On Fri, Sep 26, 2014 at 6:33 AM, Karl Wright <daddywri@gmail.com> wrote:

> Created CONNECTORS-1049 to describe relocating plugins and connector
> processes.
> Karl
>
> On Fri, Sep 26, 2014 at 5:28 AM, Karl Wright <daddywri@gmail.com> wrote:
>
>> I've replied to the jetty vs. tomcat issue in the CONNECTORS-345 comment
>> thread.
>> Thanks!
>> Karl
>>
>> On Fri, Sep 26, 2014 at 5:22 AM, Karl Wright <daddywri@gmail.com> wrote:
>>
>>> I've moved CONNECTORS-345 to "Fix in 2.0" and attached your comments.
>>> Let's talk about the jetty deployment issues in that ticket thread.
>>>
>>> Karl
>>>
>>>
>>> On Fri, Sep 26, 2014 at 5:16 AM, Karl Wright <daddywri@gmail.com> wrote:
>>>
>>>> CONNECTORS-1048 created for including jar versions.
>>>> Karl
>>>>
>>>> On Fri, Sep 26, 2014 at 5:03 AM, Karl Wright <daddywri@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Abe-san,
>>>>>
>>>>> It seems to me that we are discussing some distinct and independent
>>>>> improvements, so maybe I will open specific tickets for each one and
we can
>>>>> work on things one at a time.  But, first:
>>>>>
>>>>> bq. I just simplified directories since other Apache projects look
>>>>> like much simpler.
>>>>>
>>>>> That's true, but I don't know of any other Apache project which
>>>>> attempts to handle conditional compilation to the degree that we have
to in
>>>>> ManifoldCF.  So I think we should expect things to be more complicated.
:-)
>>>>>
>>>>> bq. BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
>>>>> other),  to create *-proprietary directories is awkward, isn't  it?
>>>>>
>>>>> I've never particularly liked the -proprietary paradigm, but I could
>>>>> think of no other way.  The reasons behind it was that we *must* legally
>>>>> exclude anything proprietary from the distribution.  By making sure that
>>>>> every proprietary jar is in a well-labeled directory, I can simply make
>>>>> sure all such directories are excluded.  But that also includes wars.
>>>>> Since wars must include all jars, that meant that the build had to create
>>>>> two different sets of wars, and put them  in different directories. 
The
>>>>> only other alternative is to guarantee that we always build our release
>>>>> candidates from a fresh checkout each time, and that seemed risky.
>>>>>
>>>>> I agree that it would be more convenient for the user if our build
>>>>> created one set of wars based on what was included by the user, and that
we
>>>>> shipped just the minimal binary that didn't include proprietary code.
 If
>>>>> we can solve the war problem, we can do that.
>>>>>
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Sep 26, 2014 at 12:51 AM, Shinichiro Abe <
>>>>> shinichiro.abe.1@gmail.com> wrote:
>>>>>
>>>>>> Hi Karl,
>>>>>>
>>>>>> Thanks for the reply.
>>>>>>
>>>>>> > (1) I don't understand what you mean by, "because build.xml
in
>>>>>> framework is
>>>>>> > currently too long to include class
>>>>>> > paths in lib".  Can you clarify?
>>>>>>
>>>>>> Rerated to (5) , manifest-cp configuration is not cool at all.
>>>>>>
>>>>>> https://svn.apache.org/viewvc/manifoldcf/trunk/framework/build.xml?view=markup#l1343
>>>>>> If new jars introduced with new feature, manifest-cp-proprietary
must
>>>>>> be tweaked, too.
>>>>>> (i18n properties files also bother us.)
>>>>>>
>>>>>> Alternatively, the ant task below is useful.
>>>>>>
>>>>>> common-build.xml:  Add artifact-version to dest.
>>>>>> + <get
>>>>>> src="${maven-base-url}/${project-path}/${artifact-name}/${artifact-version}/${artifact-name}-${artifact-version}.${artifact-type}"
>>>>>> dest="${target}/${artifact-name}-${artifact-version}.${artifact-type}"/>
>>>>>>
>>>>>> framework/build.xml: New manifest-cp property setting works fine.
>>>>>>
>>>>>> -          <property name="manifest-cp-74" value="${manifest-cp-73}
>>>>>> ../lib/xmlsec.jar"/>
>>>>>> -           <property name="manifest-cp-75" value="${manifest-cp-74}
>>>>>> ../lib/opensaml.jar"/>
>>>>>>
>>>>>> +        <property name="liblocation" location="../lib" />
>>>>>> +        <path id="lib-jars">
>>>>>> +            <fileset dir="${liblocation}" includes="*.jar"/>
>>>>>> +        </path>
>>>>>> +        <pathconvert property="manifest-cp" refid="lib-jars"
>>>>>> targetos="unix" pathsep=" ">
>>>>>> +            <map from="${liblocation}" to="../lib"/>
>>>>>> +            <map from="\" to="/"/>
>>>>>> +        </pathconvert>
>>>>>>
>>>>>> +        <!-- <property name="manifest-cp"
>>>>>> value="${manifest-cp-75}"/> -->
>>>>>>
>>>>>> > (2) Some of these suggestions seem to be making distinctions
>>>>>> between files
>>>>>> > and directories that I don't understand the reason for.
>>>>>>
>>>>>> 1. I came to mind CONNECTORS-345.
>>>>>> 2. Currently I have to create scripts in production:
>>>>>>  startup.sh --
>>>>>>     java -jar start.jar
>>>>>>     echo $! > mcf.pid
>>>>>>  shutdown.sh --
>>>>>>     kill cat(mcf.pid) <-- MCF's embedded Jetty example can't call
>>>>>> agent-shutdown AFAIK but this script isn't cool.
>>>>>>
>>>>>> 3. start.jar command line options are useful.
>>>>>>
>>>>>> http://www.eclipse.org/jetty/documentation/current/start-jar.html#d0e8418
>>>>>>
>>>>>> 4. If we can separate Jetty-lib and mcf-lib, we could deploy Tomcat.
>>>>>> Of course combined.war can do that though.
>>>>>> I thought additional jars could copy to another directory, and might
>>>>>> load these using Custom ClassLoader.
>>>>>>
>>>>>> http://www.eclipse.org/jetty/documentation/current/jetty-classloading.html
>>>>>>
>>>>>> 5. But If users don't want to config jetty.xml(e.g. thread pool,
>>>>>> requestHeaderSize and so on in jetty.xml --
>>>>>> http://wiki.eclipse.org/Jetty/Howto/Configure_Connectors),
>>>>>> I will drop these suggestions.
>>>>>>
>>>>>> > (3) I agree with "connector-specific-processes" and "plugins".
 But
>>>>>> other
>>>>>> > hierarchy additions seem less helpful, such as hiding the examples
>>>>>> under
>>>>>> > additional levels of hierarchy.  I think it should be possible
>>>>>> immediately
>>>>>> > for a user to see what examples are available.  But maybe we
could
>>>>>> change
>>>>>> > the names to be clearer.
>>>>>> Yes. Especially I want to change name *-proprietary into something
>>>>>> like *-extra because "proprietary".length() is long.
>>>>>>
>>>>>> BTW, for a few jars(oracle,mysql,alfresco,jcifs,livelink,jtds and
>>>>>> other),  to create *-proprietary directories is awkward, isn't  it?
>>>>>> I know CONNECTORS-402 intent, but more simplified, can we say that
>>>>>> "if you use these jars,
>>>>>> please get source and run ''ant make-deps build"? -- I usually say
>>>>>> so, no problems occur for now.
>>>>>>
>>>>>> > (4) Rather than group together xxx and xxx-proprietary, and
yyy and
>>>>>> > yyy-proprietary, it would be more appropriate to have a
>>>>>> "proprietary"
>>>>>> > directory and an "open" directory", with xxx and yyy under them.
>>>>>>
>>>>>> Ok.
>>>>>> But I want to know, can we deprecate example and  move
>>>>>> example-proprietary to example?
>>>>>> I'd like to merge these examples. If proprietary jars exists, MCF
>>>>>> just have to load from extra-dir.
>>>>>> I assume that we can have extra-lib which is optional, If it exists,
>>>>>> MCF will load any Jars
>>>>>> found in this directory and use them to resolve connectors specified
>>>>>> in connectors.xml.
>>>>>> (Solr has plugin class loader at $SOLR_HOME/lib)
>>>>>>
>>>>>> I just simplified directories since other Apache projects look like
>>>>>> much simpler.
>>>>>>
>>>>>> Regards,
>>>>>> Shinichiro Abe.
>>>>>>
>>>>>> On 2014/09/26, at 3:01, Karl Wright <daddywri@gmail.com> wrote:
>>>>>>
>>>>>> > Hi Abe-san,
>>>>>> >
>>>>>> > Some comments:
>>>>>> >
>>>>>> > (1) I don't understand what you mean by, "because build.xml
in
>>>>>> framework is
>>>>>> > currently too long to include class
>>>>>> > paths in lib".  Can you clarify?
>>>>>> >
>>>>>> > (2) Some of these suggestions seem to be making distinctions
>>>>>> between files
>>>>>> > and directories that I don't understand the reason for.  For
>>>>>> example: "in
>>>>>> > process-single directory we can
>>>>>> > create lib dir(for jetty jars), lib/ext dir(for logger jars),
>>>>>> resource
>>>>>> > dir(for logging.ini) and etc dir(for jetty.xml)."  Why separate
>>>>>> jars in
>>>>>> > this way?  They are all necessary at the root level to run the
>>>>>> example.  I
>>>>>> > would not understand where to add a new jar with this arrangement
>>>>>> because
>>>>>> > the meaning of the directories is unclear.
>>>>>> >
>>>>>> > (3) I agree with "connector-specific-processes" and "plugins".
 But
>>>>>> other
>>>>>> > hierarchy additions seem less helpful, such as hiding the examples
>>>>>> under
>>>>>> > additional levels of hierarchy.  I think it should be possible
>>>>>> immediately
>>>>>> > for a user to see what examples are available.  But maybe we
could
>>>>>> change
>>>>>> > the names to be clearer.
>>>>>> >
>>>>>> > (4) Rather than group together xxx and xxx-proprietary, and
yyy and
>>>>>> > yyy-proprietary, it would be more appropriate to have a
>>>>>> "proprietary"
>>>>>> > directory and an "open" directory", with xxx and yyy under them.
>>>>>> >
>>>>>> > (5) Putting version numbers on jars is difficult in some cases,
>>>>>> especially
>>>>>> > in construction of start.jar, because the ant methods for
>>>>>> constructing
>>>>>> > start.jar are poor.  The version of each jar would need to be
>>>>>> defined
>>>>>> > globally in the ant build, and included whenever the jar is
>>>>>> referenced.
>>>>>> >
>>>>>> > Karl
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message