oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Mattmann" <mattm...@apache.org>
Subject Re: Review Request 36953: Full implementation of AvroRPC alongside XMLRPC in filemgr module
Date Mon, 24 Aug 2015 23:38:23 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36953/#review96236
-----------------------------------------------------------


This looks a lot better, Radu. A couple more things:

1. Please don't use star imports. Makes it too hard to track which classes are imported. Please
use FQDN class names.
2. What about the curator and other places the File Manager is used, e.g., pcs-services, etc.
It seems like these changes would break those portions of the system. Have you tried to build
all of OODT with these changes?
3. As you noted, still need to update RADIX.

- Chris Mattmann


On Aug. 24, 2015, 5:19 p.m., Radu Manole wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36953/
> -----------------------------------------------------------
> 
> (Updated Aug. 24, 2015, 5:19 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> -------
> 
> Separated the FileManager logic from rpc communication createing interfaces for server/client
side (FileManagerCient, FileManagerServer).Refactor existing XMLRPC server/client so it would
use communication interfaces. Created Avro schemas and protocol for ipc communication, (classes
that have the prefixe Avro), with a Factory class AvroTypeFactory that transforms from oodt
to avro and from avro to oodt classes. Implement avro server/client. For changeability I created
a Factory (RpcCommunicationFactory) that can be configurable to use either avro or xmlrpc
for IPC. Because of this, anywhere in project where on instantiation of server/client rewrite
to use this factory.
> 
> 
> Diffs
> -----
> 
>   trunk/app/fmbrowser/src/main/java/org/apache/oodt/cas/filemgr/browser/model/CasDB.java
1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/action/FilemgrUniquenessChecker.java
1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/comparator/FilemgrUniquenessCheckComparator.java
1693501 
>   trunk/filemgr/pom.xml 1693501 
>   trunk/filemgr/src/main/avro/types/AvroComplexQuery.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroElement.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroExtractorSpec.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroFileTransferStatus.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroFilterAlgor.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroMetadata.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroMimeType.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroProduct.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroProductPage.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroProductType.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroQuery.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroQueryCriteria.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroQueryFilter.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroQueryResult.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroReference.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroTypeHandler.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/protocol.avdl PRE-CREATION 
>   trunk/filemgr/src/main/bin/filemgr 1693501 
>   trunk/filemgr/src/main/bin/filemgr-client 1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/AbstractDeleteProductCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/DeleteProductByIdCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/FileManagerCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetFirstPageCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetLastPageCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetNextPageCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetNumProductsCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetPrevPageCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetProductPercentTransferredCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/RetrieveFilesCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/AvroLocalDataTransferer.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/InPlaceDataTransferer.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/RemoteDataTransferer.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/ingest/LocalCache.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/ingest/StdIngester.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/ProductMetKeys.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/AvroFileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/AvroFileManagerServer.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManager.java PRE-CREATION

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerClientMain.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerServer.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerServerMain.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerClient.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerServer.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/AvroFileManagerClientFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/AvroFileManagerServerFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/FileManagerClientFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/FileManagerServerFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/XmlRpcFileManagerClientFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/XmlRpcFileManagerServerFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/CatalogSearch.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/DeleteProduct.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/ExpImpCatalog.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/MetadataBasedProductMover.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/MetadataDumper.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/ProductDumper.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/QueryTool.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/SolrIndexer.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/AvroTypeFactory.java PRE-CREATION

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/RpcCommunicationFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlRpcStructFactory.java
1693501 
>   trunk/filemgr/src/main/resources/filemgr.properties 1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/TestFileManagerCli.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/UseMockClientCmdLineActionStore.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/DummyFileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestAddProductTypeCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestDeleteProductByIdCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestDeleteProductByNameCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestDumpMetadataCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetCurrentTransferCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetCurrentTransfersCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetFilePercentTransferredCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetFirstPageCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetLastPageCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetNextPageCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetNumProductsCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetPrevPageCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductByIdCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductByNameCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductPercentTransferredCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductTypeByNameCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestHasProductCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestIngestProductCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestLuceneQueryCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestSqlQueryCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestCachedIngester.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestLocalCache.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestRmiCache.java 1693501

>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestStdIngester.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/TestTikaAutoDetectExtractor.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/structs/type/TestTypeHandler.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/MockFileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/TestRpcFileManager.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/TestRpcFileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/rpc/TestServerClientFactories.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/tools/TestExpImpCatalog.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/tools/TestMetadataBasedProductMover.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/util/TestAvroTypesFactory.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/util/TestXmlRpcStructFactory.java
1693501 
>   trunk/filemgr/src/test/resources/createProduct/test.txt PRE-CREATION 
>   trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/filemgr/src/main/resources/bin/filemgr
1693501 
>   trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/filemgr/src/main/resources/bin/filemgr-client
1693501 
>   trunk/pcs/core/src/main/java/org/apache/oodt/pcs/util/FileManagerUtils.java 1693501

>   trunk/pge/src/main/java/org/apache/oodt/cas/pge/staging/FileStager.java 1693501 
>   trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/XmlHelper.java 1693501 
>   trunk/pge/src/test/java/org/apache/oodt/cas/pge/staging/TestFileStager.java 1693501

>   trunk/profile/src/main/java/org/apache/oodt/profile/handlers/cas/CASProfileHandler.java
1693501 
>   trunk/webapp/components/src/main/java/org/apache/oodt/cas/webcomponents/filemgr/FileManagerConn.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/CASProductHandler.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/data/DataDeliveryServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/data/DatasetDeliveryServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/services/CasProductJaxrsService.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/servlets/CasProductJaxrsServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rdf/RDFDatasetServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rdf/RDFProductServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rss/RSSProductServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rss/RSSProductTransferServlet.java
1693501 
> 
> Diff: https://reviews.apache.org/r/36953/diff/
> 
> 
> Testing
> -------
> 
> Used existing tests, applied them for Avro RPC. 
> 
> Installed filemanager module, and started it as described in the oodt documentation.
Ingested a simple Generic file. Used querry-tool.
> 
> 
> Thanks,
> 
> Radu Manole
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message