oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Mattmann" <mattm...@apache.org>
Subject Re: Review Request 36953: Full implementation of AvroRPC alongside XMLRPC in filemgr module
Date Tue, 25 Aug 2015 05:57:13 GMT


> On Aug. 24, 2015, 11:38 p.m., Chris Mattmann wrote:
> > This looks a lot better, Radu. A couple more things:
> > 
> > 1. Please don't use star imports. Makes it too hard to track which classes are imported.
Please use FQDN class names.
> > 2. What about the curator and other places the File Manager is used, e.g., pcs-services,
etc. It seems like these changes would break those portions of the system. Have you tried
to build all of OODT with these changes?
> > 3. As you noted, still need to update RADIX.
> 
> Lewis McGibbney wrote:
>     I am not sure that the curator was scoped for inclusion in this portion of the work.
>     RADIX needs to be updated and I am prepared to help out to get that done.

Thanks Lewis. Whether the curator is scoped or not, this code cannot be committed without
updating the curator since it depends on the FM APIs.


- Chris


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36953/#review96236
-----------------------------------------------------------


On Aug. 24, 2015, 5:19 p.m., Radu Manole wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36953/
> -----------------------------------------------------------
> 
> (Updated Aug. 24, 2015, 5:19 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> -------
> 
> Separated the FileManager logic from rpc communication createing interfaces for server/client
side (FileManagerCient, FileManagerServer).Refactor existing XMLRPC server/client so it would
use communication interfaces. Created Avro schemas and protocol for ipc communication, (classes
that have the prefixe Avro), with a Factory class AvroTypeFactory that transforms from oodt
to avro and from avro to oodt classes. Implement avro server/client. For changeability I created
a Factory (RpcCommunicationFactory) that can be configurable to use either avro or xmlrpc
for IPC. Because of this, anywhere in project where on instantiation of server/client rewrite
to use this factory.
> 
> 
> Diffs
> -----
> 
>   trunk/app/fmbrowser/src/main/java/org/apache/oodt/cas/filemgr/browser/model/CasDB.java
1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/action/FilemgrUniquenessChecker.java
1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/comparator/FilemgrUniquenessCheckComparator.java
1693501 
>   trunk/filemgr/pom.xml 1693501 
>   trunk/filemgr/src/main/avro/types/AvroComplexQuery.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroElement.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroExtractorSpec.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroFileTransferStatus.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroFilterAlgor.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroMetadata.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroMimeType.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroProduct.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroProductPage.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroProductType.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroQuery.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroQueryCriteria.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroQueryFilter.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroQueryResult.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroReference.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroTypeHandler.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/protocol.avdl PRE-CREATION 
>   trunk/filemgr/src/main/bin/filemgr 1693501 
>   trunk/filemgr/src/main/bin/filemgr-client 1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/AbstractDeleteProductCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/DeleteProductByIdCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/FileManagerCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetFirstPageCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetLastPageCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetNextPageCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetNumProductsCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetPrevPageCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetProductPercentTransferredCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/RetrieveFilesCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/AvroLocalDataTransferer.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/InPlaceDataTransferer.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/RemoteDataTransferer.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/ingest/LocalCache.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/ingest/StdIngester.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/ProductMetKeys.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/AvroFileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/AvroFileManagerServer.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManager.java PRE-CREATION

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerClientMain.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerServer.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerServerMain.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerClient.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerServer.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/AvroFileManagerClientFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/AvroFileManagerServerFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/FileManagerClientFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/FileManagerServerFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/XmlRpcFileManagerClientFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/XmlRpcFileManagerServerFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/CatalogSearch.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/DeleteProduct.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/ExpImpCatalog.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/MetadataBasedProductMover.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/MetadataDumper.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/ProductDumper.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/QueryTool.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/SolrIndexer.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/AvroTypeFactory.java PRE-CREATION

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/RpcCommunicationFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlRpcStructFactory.java
1693501 
>   trunk/filemgr/src/main/resources/filemgr.properties 1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/TestFileManagerCli.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/UseMockClientCmdLineActionStore.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/DummyFileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestAddProductTypeCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestDeleteProductByIdCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestDeleteProductByNameCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestDumpMetadataCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetCurrentTransferCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetCurrentTransfersCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetFilePercentTransferredCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetFirstPageCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetLastPageCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetNextPageCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetNumProductsCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetPrevPageCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductByIdCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductByNameCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductPercentTransferredCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductTypeByNameCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestHasProductCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestIngestProductCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestLuceneQueryCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestSqlQueryCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestCachedIngester.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestLocalCache.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestRmiCache.java 1693501

>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestStdIngester.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/TestTikaAutoDetectExtractor.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/structs/type/TestTypeHandler.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/MockFileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/TestRpcFileManager.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/TestRpcFileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/rpc/TestServerClientFactories.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/tools/TestExpImpCatalog.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/tools/TestMetadataBasedProductMover.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/util/TestAvroTypesFactory.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/util/TestXmlRpcStructFactory.java
1693501 
>   trunk/filemgr/src/test/resources/createProduct/test.txt PRE-CREATION 
>   trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/filemgr/src/main/resources/bin/filemgr
1693501 
>   trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/filemgr/src/main/resources/bin/filemgr-client
1693501 
>   trunk/pcs/core/src/main/java/org/apache/oodt/pcs/util/FileManagerUtils.java 1693501

>   trunk/pge/src/main/java/org/apache/oodt/cas/pge/staging/FileStager.java 1693501 
>   trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/XmlHelper.java 1693501 
>   trunk/pge/src/test/java/org/apache/oodt/cas/pge/staging/TestFileStager.java 1693501

>   trunk/profile/src/main/java/org/apache/oodt/profile/handlers/cas/CASProfileHandler.java
1693501 
>   trunk/webapp/components/src/main/java/org/apache/oodt/cas/webcomponents/filemgr/FileManagerConn.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/CASProductHandler.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/data/DataDeliveryServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/data/DatasetDeliveryServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/services/CasProductJaxrsService.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/servlets/CasProductJaxrsServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rdf/RDFDatasetServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rdf/RDFProductServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rss/RSSProductServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rss/RSSProductTransferServlet.java
1693501 
> 
> Diff: https://reviews.apache.org/r/36953/diff/
> 
> 
> Testing
> -------
> 
> Used existing tests, applied them for Avro RPC. 
> 
> Installed filemanager module, and started it as described in the oodt documentation.
Ingested a simple Generic file. Used querry-tool.
> 
> 
> Thanks,
> 
> Radu Manole
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message