incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: pluggable s4r fetching strategies
Date Wed, 17 Apr 2013 09:11:24 GMT

On Apr 17, 2013, at 09:50 , JiHyoun Park wrote:

> Hi
> 
> Supposed that I have my custom module in my local machine, 
> 
> How can I set the application configuration for custom modules? Where?
> What is the correct string format of the modulesURI parameter?
> Can you give me an example?

For specifying custom modules in the application configuration, see here: http://incubator.apache.org/s4/doc/0.6.0/configuration/
("overriding modules" section)

Regards,

Matthieu


> 
> Best Regards
> Jihyoun
> 
> 
> On Tue, Apr 16, 2013 at 7:08 PM, Matthieu Morel <mmorel@apache.org> wrote:
> 
> Hello,
> 
> a key difference with S4 0.5 is that custom modules are specified in the application
configuration, and can be loaded remotely with the modulesURI parameter.
> 
> However, if for fetching remote modules you need a special module, that does not work.

> 
> In S4ApplicationMaster for S4 0.5 we specified the hdfs fetcher for the S4 node as :
> // add module for fetching from hdfs
> extraModulesClasses.add(HdfsFetcherModule.class.getName());
> 
> However in S4 0.4 we don't have that parameter when starting the node. 
> 
> Right now there is no clean way to configure a remote file fetcher when starting a node.
That is something we need to improve, thanks for pointing this out!
> 
> 
> Ideally we should give the option to pass custom modules when starting the node. It's
actually just a few lines of code to add. (adding a parameter to get extra modules classes,
instantiate these modules by reflection and pass them to the injector in line 80 of S4Node
- we can help if you can't make it work).
> 
> Otherwise you may modify the BaseModule to include the dependency to the hdfs fetcher
(you'll need the hdfs fetcher in the classpath when compiling). 
> Or make the application available through another procotol such as http.
> 
> 
> Hope this helps,
> 
> Matthieu
> 
> 
> On Apr 16, 2013, at 12:08 , JiHyoun Park wrote:
> 
>> Hi,
>> 
>> I am trying to apply 'pluggable s4r fetching strategies' of S4-25 to S4 0.6.0.
>> 
>> Changes that I made are
>> 
>> 1) org.apache.s4.core.BaseModule.java
>>     protected void configure() {
>>         ...
>>         // added this codes
>>         Multibinder<ArchiveFetcher> archiveFetcherMultibinder = Multibinder.newSetBinder(binder(),
ArchiveFetcher.class);
>>         archiveFetcherMultibinder.addBinding().to(FileSystemArchiveFetcher.class);
>>         archiveFetcherMultibinder.addBinding().to(HttpArchiveFetcher.class);
>> }
>> 
>> 2) org.apache.s4.core.util.ArchiveFetcher.java
>>     // added this codes
>>     boolean handlesProtocol(URI uri);
>> 
>> 3) org.apache.s4.deploy.FileSystemArchiveFetcher.java
>>     // added this codes
>>     @Override
>>     public boolean handlesProtocol(URI uri) {
>>         return "file".equalsIgnoreCase(uri.getScheme());
>>     }
>> 
>> 4) org.apache.s4.core.util.HttpArchiveFetcher.java
>>     // added this codes
>>     @Override
>>     public boolean handlesProtocol(URI uri) {
>>         return ("http".equalsIgnoreCase(uri.getScheme()) || "https".equalsIgnoreCase(uri.getScheme()));
>>     }
>> 
>> 5) org.apache.s4.core.util.RemoteFileFetcher.java
>>     // added this codes
>>     private final Set<ArchiveFetcher> archiveFetchers;
>> 
>>     // added this codes
>>     @Inject
>>     public RemoteFileFetcher(Set<ArchiveFetcher> archiveFetchers) {
>>         this.archiveFetchers = archiveFetchers;
>>     }
>> 
>>     public InputStream fetch(URI uri) throws ArchiveFetchException {
>>         ....
>>         /* removed this codes
>>         if ("file".equalsIgnoreCase(scheme)) {
>>             return new FileSystemArchiveFetcher().fetch(uri);
>>         }
>>         if ("http".equalsIgnoreCase(scheme) || "https".equalsIgnoreCase(scheme))
{
>>             return new HttpArchiveFetcher().fetch(uri);
>>         }
>>         */
>> 
>>         // added this codes
>>         for (ArchiveFetcher archiveFetcher : archiveFetchers) {
>>             if (archiveFetcher.handlesProtocol(uri)) {
>>                 return archiveFetcher.fetch(uri);
>>             }
>>         }
>>     }
>> 
>> 6) I passed the "-modulesClasses=my.HdfsFetcherModule" argument when I deploy my
s4r file. The HdfsFetcherModule is the same HdfsFetcherModule from S4-25.
>> 
>> 
>> But, when I run it, I got this error.
>> 17:46:24.151 [S4 platform loader] ERROR org.apache.s4.core.S4Bootstrap - Cannot start
S4 node
>> org.apache.s4.deploy.DeploymentFailedException: Cannot deploy application [SimpleApp]
from URI [hdfs://localhost:9000/user/root/s4data/SimpleApp/203/s4app.s4r] 
>> 
>> 
>> 	at org.apache.s4.core.S4Bootstrap.loadApp(S4Bootstrap.java:219) [s4-core-0.6.0-incubating.jar:0.6.0-incubating]
>> 	at org.apache.s4.core.S4Bootstrap.startS4App(S4Bootstrap.java:149) [s4-core-0.6.0-incubating.jar:0.6.0-incubating]
>> 
>> 
>> 	at org.apache.s4.core.S4Bootstrap.access$000(S4Bootstrap.java:80) [s4-core-0.6.0-incubating.jar:0.6.0-incubating]
>> 	at org.apache.s4.core.S4Bootstrap$1.run(S4Bootstrap.java:139) [s4-core-0.6.0-incubating.jar:0.6.0-incubating]
>> 
>> 
>> 	at java.lang.Thread.run(Thread.java:679) [na:1.6.0_22]
>> Caused by: org.apache.s4.core.util.ArchiveFetchException: Unsupported protocol hdfs
>> 	at org.apache.s4.core.util.RemoteFileFetcher.fetch(RemoteFileFetcher.java:63) ~[s4-core-0.6.0-incubating.jar:0.6.0-incubating]
>> 
>> 
>> 	at org.apache.s4.core.S4Bootstrap.loadApp(S4Bootstrap.java:214) [s4-core-0.6.0-incubating.jar:0.6.0-incubating]
>> 	... 4 common frames omitted
>> 
>> I think there are some missing links to make S4 recognize the custom module.
>> Is it related with the new parameter "-modulesURIs"? How can I use it?
>> I found some description for this parameter at org.apache.s4.tools.Deploy.java
>>         @Parameter(names = { "-modulesURIs", "-mu" }, description = "URIs for fetching
code of custom modules")
>>         List<String> modulesURIs = new ArrayList<String>();
>> 
>> But I have no idea about how to use it. My custom module will be located in HDFS
with the s4r file that I want to deploy.
>> Or are there any other things that I have to take into consideration in the above
implementation?
>> 
>> Best Regards
>> Jihyoun.
>> 
> 
> 


Mime
View raw message