incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JiHyoun Park <apr...@gmail.com>
Subject pluggable s4r fetching strategies
Date Tue, 16 Apr 2013 10:08:05 GMT
Hi,

I am trying to apply 'pluggable s4r fetching strategies' of S4-25 to S4
0.6.0.

Changes that I made are

1) org.apache.s4.core.BaseModule.java
    protected void configure() {
        ...
        // added this codes
        Multibinder<ArchiveFetcher> archiveFetcherMultibinder =
Multibinder.newSetBinder(binder(), ArchiveFetcher.class);

archiveFetcherMultibinder.addBinding().to(FileSystemArchiveFetcher.class);
        archiveFetcherMultibinder.addBinding().to(HttpArchiveFetcher.class);
}

2) org.apache.s4.core.util.ArchiveFetcher.java
    // added this codes
    boolean handlesProtocol(URI uri);

3) org.apache.s4.deploy.FileSystemArchiveFetcher.java
    // added this codes
    @Override
    public boolean handlesProtocol(URI uri) {
        return "file".equalsIgnoreCase(uri.getScheme());
    }

4) org.apache.s4.core.util.HttpArchiveFetcher.java
    // added this codes
    @Override
    public boolean handlesProtocol(URI uri) {
        return ("http".equalsIgnoreCase(uri.getScheme()) ||
"https".equalsIgnoreCase(uri.getScheme()));
    }

5) org.apache.s4.core.util.RemoteFileFetcher.java
    // added this codes
    private final Set<ArchiveFetcher> archiveFetchers;

    // added this codes
    @Inject
    public RemoteFileFetcher(Set<ArchiveFetcher> archiveFetchers) {
        this.archiveFetchers = archiveFetchers;
    }

    public InputStream fetch(URI uri) throws ArchiveFetchException {
        ....
        /* removed this codes
        if ("file".equalsIgnoreCase(scheme)) {
            return new FileSystemArchiveFetcher().fetch(uri);
        }
        if ("http".equalsIgnoreCase(scheme) ||
"https".equalsIgnoreCase(scheme)) {
            return new HttpArchiveFetcher().fetch(uri);
        }
        */

        // added this codes
        for (ArchiveFetcher archiveFetcher : archiveFetchers) {
            if (archiveFetcher.handlesProtocol(uri)) {
                return archiveFetcher.fetch(uri);
            }
        }
    }

6) I passed the "-modulesClasses=my.HdfsFetcherModule" argument when I
deploy my s4r file. The HdfsFetcherModule is the same HdfsFetcherModule
from S4-25.


But, when I run it, I got this error.

17:46:24.151 [S4 platform loader] ERROR org.apache.s4.core.S4Bootstrap
- Cannot start S4 node
org.apache.s4.deploy.DeploymentFailedException: Cannot deploy
application [SimpleApp] from URI
[hdfs://localhost:9000/user/root/s4data/SimpleApp/203/s4app.s4r]
	at org.apache.s4.core.S4Bootstrap.loadApp(S4Bootstrap.java:219)
[s4-core-0.6.0-incubating.jar:0.6.0-incubating]
	at org.apache.s4.core.S4Bootstrap.startS4App(S4Bootstrap.java:149)
[s4-core-0.6.0-incubating.jar:0.6.0-incubating]
	at org.apache.s4.core.S4Bootstrap.access$000(S4Bootstrap.java:80)
[s4-core-0.6.0-incubating.jar:0.6.0-incubating]
	at org.apache.s4.core.S4Bootstrap$1.run(S4Bootstrap.java:139)
[s4-core-0.6.0-incubating.jar:0.6.0-incubating]
	at java.lang.Thread.run(Thread.java:679) [na:1.6.0_22]
Caused by: org.apache.s4.core.util.ArchiveFetchException: Unsupported
protocol hdfs
	at org.apache.s4.core.util.RemoteFileFetcher.fetch(RemoteFileFetcher.java:63)
~[s4-core-0.6.0-incubating.jar:0.6.0-incubating]
	at org.apache.s4.core.S4Bootstrap.loadApp(S4Bootstrap.java:214)
[s4-core-0.6.0-incubating.jar:0.6.0-incubating]
	... 4 common frames omitted


I think there are some missing links to make S4 recognize the custom module.
Is it related with the new parameter "-modulesURIs"? How can I use it?
I found some description for this parameter at
org.apache.s4.tools.Deploy.java
        @Parameter(names = { "-modulesURIs", "-mu" }, description = "URIs
for fetching code of custom modules")
        List<String> modulesURIs = new ArrayList<String>();

But I have no idea about how to use it. My custom module will be located in
HDFS with the s4r file that I want to deploy.
Or are there any other things that I have to take into consideration in the
above implementation?

Best Regards
Jihyoun.

Mime
View raw message