oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Mattmann" <mattm...@apache.org>
Subject Re: Review Request 36953: Full implementation of AvroRPC alongside XMLRPC in filemgr module
Date Sat, 22 Aug 2015 16:38:04 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36953/#review96119
-----------------------------------------------------------


This was a LOT better Radu. You are getting close.

Some big things to remember:

1. If code is commented out, you should remove it.
2. We shouldn't disable tests e.g., Tika, by commenting it out. If the test isn't passing,
then something is wrong (we have a regression)
3. You need to update RADIX and make sure this is working with OODT RADIX.

- Chris Mattmann


On Aug. 21, 2015, 7:53 p.m., Radu Manole wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36953/
> -----------------------------------------------------------
> 
> (Updated Aug. 21, 2015, 7:53 p.m.)
> 
> 
> Review request for oodt, Lewis McGibbney and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> -------
> 
> Separated the FileManager logic from rpc communication createing interfaces for server/client
side (FileManagerCient, FileManagerServer).Refactor existing XMLRPC server/client so it would
use communication interfaces. Created Avro schemas and protocol for ipc communication, (classes
that have the prefixe Avro), with a Factory class AvroTypeFactory that transforms from oodt
to avro and from avro to oodt classes. Implement avro server/client. For changeability I created
a Factory (RpcCommunicationFactory) that can be configurable to use either avro or xmlrpc
for IPC. Because of this, anywhere in project where on instantiation of server/client rewrite
to use this factory.
> 
> 
> Diffs
> -----
> 
>   trunk/app/fmbrowser/src/main/java/org/apache/oodt/cas/filemgr/browser/model/CasDB.java
1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/action/FilemgrUniquenessChecker.java
1693501 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/comparator/FilemgrUniquenessCheckComparator.java
1693501 
>   trunk/crawler/src/site/xdoc/user/index.xml 1693501 
>   trunk/filemgr/pom.xml 1693501 
>   trunk/filemgr/src/main/avro/types/AvroComplexQuery.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroElement.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroExtractorSpec.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroFileTransferStatus.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroFilterAlgor.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroMetadata.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroMimeType.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroProduct.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroProductPage.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroProductType.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroQuery.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroQueryCriteria.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroQueryFilter.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroQueryResult.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroReference.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/AvroTypeHandler.avsc PRE-CREATION 
>   trunk/filemgr/src/main/avro/types/protocol.avdl PRE-CREATION 
>   trunk/filemgr/src/main/bin/filemgr 1693501 
>   trunk/filemgr/src/main/bin/filemgr-client 1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/AbstractDeleteProductCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/DeleteProductByIdCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/FileManagerCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetFirstPageCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetLastPageCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetNextPageCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetNumProductsCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetPrevPageCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/GetProductPercentTransferredCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/IngestProductCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/cli/action/RetrieveFilesCliAction.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/AvroLocalDataTransferer.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/InPlaceDataTransferer.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/LocalDataTransferer.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/datatransfer/RemoteDataTransferer.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/ingest/LocalCache.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/ingest/StdIngester.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/metadata/ProductMetKeys.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/AvroFileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/AvroFileManagerServer.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManager.java PRE-CREATION

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerClientMain.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerServer.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/FileManagerServerMain.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerClient.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/XmlRpcFileManagerServer.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/AvroFileManagerClientFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/AvroFileManagerServerFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/FileManagerClientFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/FileManagerServerFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/XmlRpcFileManagerClientFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/system/rpc/XmlRpcFileManagerServerFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/CatalogSearch.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/DeleteProduct.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/ExpImpCatalog.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/MetadataBasedProductMover.java
1693501 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/MetadataDumper.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/ProductDumper.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/QueryTool.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/tools/SolrIndexer.java 1693501

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/AvroTypeFactory.java PRE-CREATION

>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/RpcCommunicationFactory.java
PRE-CREATION 
>   trunk/filemgr/src/main/java/org/apache/oodt/cas/filemgr/util/XmlRpcStructFactory.java
1693501 
>   trunk/filemgr/src/main/resources/filemgr.properties 1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/TestFileManagerCli.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/UseMockClientCmdLineActionStore.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/DummyFileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestAddProductTypeCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestDeleteProductByIdCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestDeleteProductByNameCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestDumpMetadataCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetCurrentTransferCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetCurrentTransfersCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetFilePercentTransferredCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetFirstPageCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetLastPageCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetNextPageCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetNumProductsCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetPrevPageCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductByIdCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductByNameCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductPercentTransferredCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestGetProductTypeByNameCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestHasProductCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestIngestProductCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestLuceneQueryCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/cli/action/TestSqlQueryCliAction.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestCachedIngester.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestLocalCache.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestRmiCache.java 1693501

>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/ingest/TestStdIngester.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/metadata/extractors/examples/TestTikaAutoDetectExtractor.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/structs/type/TestTypeHandler.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/MockFileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/TestRpcFileManager.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/TestRpcFileManagerClient.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/system/rpc/TestServerClientFactories.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/tools/TestExpImpCatalog.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/tools/TestMetadataBasedProductMover.java
1693501 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/util/TestAvroTypesFactory.java
PRE-CREATION 
>   trunk/filemgr/src/test/java/org/apache/oodt/cas/filemgr/util/TestXmlRpcStructFactory.java
1693501 
>   trunk/filemgr/src/test/resources/createProduct/test.txt PRE-CREATION 
>   trunk/mvn/archetypes/radix/src/main/resources/archetype-resources/filemgr/src/main/resources/bin/filemgr
1693501 
>   trunk/pcs/core/src/main/java/org/apache/oodt/pcs/util/FileManagerUtils.java 1693501

>   trunk/pge/src/main/java/org/apache/oodt/cas/pge/staging/FileStager.java 1693501 
>   trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/XmlHelper.java 1693501 
>   trunk/pge/src/test/java/org/apache/oodt/cas/pge/staging/TestFileStager.java 1693501

>   trunk/profile/src/main/java/org/apache/oodt/profile/handlers/cas/CASProfileHandler.java
1693501 
>   trunk/webapp/components/src/main/java/org/apache/oodt/cas/webcomponents/filemgr/FileManagerConn.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/CASProductHandler.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/data/DataDeliveryServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/data/DatasetDeliveryServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/services/CasProductJaxrsService.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/jaxrs/servlets/CasProductJaxrsServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rdf/RDFDatasetServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rdf/RDFProductServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rss/RSSProductServlet.java
1693501 
>   trunk/webapp/fmprod/src/main/java/org/apache/oodt/cas/product/rss/RSSProductTransferServlet.java
1693501 
> 
> Diff: https://reviews.apache.org/r/36953/diff/
> 
> 
> Testing
> -------
> 
> Used existing tests, applied them for Avro RPC. 
> 
> Installed filemanager module, and started it as described in the oodt documentation.
Ingested a simple Generic file. Used querry-tool.
> 
> 
> Thanks,
> 
> Radu Manole
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message