Return-Path: X-Original-To: apmail-oodt-dev-archive@www.apache.org Delivered-To: apmail-oodt-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DACDEDF2F for ; Wed, 1 Aug 2012 17:57:35 +0000 (UTC) Received: (qmail 62659 invoked by uid 500); 1 Aug 2012 17:57:35 -0000 Delivered-To: apmail-oodt-dev-archive@oodt.apache.org Received: (qmail 62632 invoked by uid 500); 1 Aug 2012 17:57:35 -0000 Mailing-List: contact dev-help@oodt.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@oodt.apache.org Delivered-To: mailing list dev@oodt.apache.org Received: (qmail 62620 invoked by uid 99); 1 Aug 2012 17:57:35 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Aug 2012 17:57:35 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [128.149.139.109] (HELO mail.jpl.nasa.gov) (128.149.139.109) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Aug 2012 17:57:28 +0000 Received: from mail.jpl.nasa.gov (ap-ehub-sp01.jpl.nasa.gov [128.149.137.148]) by smtp.jpl.nasa.gov (Sentrion-MTA-4.2.2/Sentrion-MTA-4.2.2) with ESMTP id q71HuQFm018604 (using TLSv1/SSLv3 with cipher AES128-SHA (128 bits) verified NO); Wed, 1 Aug 2012 10:56:26 -0700 Received: from AP-EMBX-SP20.RES.AD.JPL ([169.254.8.6]) by ap-ehub-sp01.RES.AD.JPL ([169.254.3.35]) with mapi id 14.02.0298.004; Wed, 1 Aug 2012 10:56:26 -0700 From: "Mattmann, Chris A (388J)" To: "" CC: Brian Foster Subject: Re: Problem when running pushpull Thread-Topic: Problem when running pushpull Thread-Index: AQHNb9crbC3sNbVvx02VRcciLxoZsJdFtCcA Date: Wed, 1 Aug 2012 17:56:25 +0000 Message-ID: <2909B3C8-3262-4E7B-9718-BD22CABF91C0@jpl.nasa.gov> References:

In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [128.149.137.113] Content-Type: text/plain; charset="us-ascii" Content-ID: <67EE848A31B3A345B44D9C5C8201260B@ad.jpl> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Source-Sender: chris.a.mattmann@jpl.nasa.gov X-AUTH: Authorized Hey YunHee, On Aug 1, 2012, at 4:17 AM, YunHee Kang wrote: > Hi Brian, >=20 >=20 > When I was reading a file named "push_pull_framework.properties", I > understood how a metadata file is named. > #extension for temporary metadata file > org.apache.oodt.cas.pushpull.write.met.file=3Dtrue > org.apache.oodt.cas.pushpull.met.file.extension=3Dinfo.tmp >=20 > And I found a metadata file written in XML having its name ended with > ".info.tmp" in the staging area, which was generated by pushpull > script. >=20 > Hence I tried to catalog the downloaded file and its metadata file in > the staging area by filemgr-client and I am sure that the file was > ingested by filemgr: >=20 > yhkang@yunhee-kisti:~/oodt-0.5/cas-filemgr/bin$ ./filemgr-client-tes.sh > Aug 1, 2012 7:37:37 PM > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient > INFO: Loading File Manager Configuration Properties from: > [../etc/filemgr.properties] > Aug 1, 2012 7:37:38 PM > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors > INFO: Running Met Extractor: > [org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor] for > product type: [GenericFile] > Aug 1, 2012 7:37:38 PM > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors > INFO: Running Met Extractor: > [org.apache.oodt.cas.filemgr.metadata.extractors.examples.MimeTypeExtract= or] > for product type: [GenericFile] > Aug 1, 2012 7:37:38 PM > org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors > INFO: Running Met Extractor: > [org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocati= onExtractor] > for product type: [GenericFile] > Aug 1, 2012 7:37:38 PM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog = toDoc > WARNING: No Metadata specified for product [tesProduct] for required > field [DataVersion]: Attempting to continue processing metadata > Aug 1, 2012 7:37:38 PM > org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferer moveFile > INFO: LocalDataTransfer: Moving File: > file:///home/yhkang/oodt-0.5/cas-filemgr/bin/TES-Aura_L2-CO2-Nadir_r00000= 02147_F06_09.he5 > to file:/home/yhkang/files/tesProduct/TES-Aura_L2-CO2-Nadir_r0000002147_F= 06_09.he5 > Aug 1, 2012 7:37:38 PM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog = toDoc > WARNING: No Metadata specified for product [tesProduct] for required > field [DataVersion]: Attempting to continue processing metadata > ingestProduct: Result: e9f22900-dbc4-11e1-a524-6bffd22dafd0 >=20 > I also checked up the file indigested by query_tool. > ./query_tool --url http://localhost:8000 --lucene -query > CAS.ProductName:tesProduct > e9f22900-dbc4-11e1-a524-6bffd22dafd0 >=20 > I am wondering why the "catalog exception" was occurred as pushpull > was running even if my file manager was working well . > Could you recommend me any material to understand mechanism between > filemgr and pushpull. Hi YunHee, This is probably a chicken and egg issue -- if your file manager catalog ha= sn't been built yet and pushpull checks product uniquess the catalog won't exist and you'll get the CatalogException the first time it's called. All times a= fter that, the catalog will exist and you won't get it. Make sense? Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattmann@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++