ant-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Rice <rob...@windermere.com>
Subject Using Ant to Collect Data?
Date Thu, 22 May 2003 18:31:35 GMT
I had a crazy thought today, that I wanted to toss out there for some feedback.

I have stepped into a project that has a data collector component.  Data is
collected in a number of different forms, and a number of different ways. 

The process for gathering data is unique to a particular data feed.  These
feeds change over time.  New feeds are added.  Old feeds are dropped.  The
tasks involved in a data feed vary from feed to feed.  Here are a couple of
examples: 

1: Get images from data provider A
  a. download (ftp) a manifest that describes the available images
  b. look at the manifest for images that match certain criteria
  c. download (ftp) those selected images to an image server

2: Get images from data provider B
  a.  execute a query that describes a set of available images
  b. download (http) those selected images to an image server

3: TabDelim data feed from data provider C
  a. download (ftp) a tab delimeted file from the provider
  b. perform a basic validity check against the data ( correct number of
columns ) 
  c. move the file to the data process server ( where it will be parsed and
loaded to a database ). 

There are about 30 different feeds.  Scripts to perform the current feeds have
been constructed over time in a number of different languages.  There is
little inhouse knowledge about these scripts.  The scripts are not
maintainable/configurable. 

I've been tasked with adding a new data feed, but also keeping my eye on
possibly bringing some of these other feeds under the same umbrella.   

As I installed Ant to manage the build of my codebase ( Java ) to construct
this new feed, I wondered if Ant might be a good candidate to complete the
original tasks?  In other words, try to use Ant as the data collector, only
coding custom tasks where totally necessary. 

I see a number of advantages in using Ant, but I may be totally out of line:
1. highly configurable through an XML file
2. plugin architecture for extending the functionality through Java
3. task based methodology, highly responsive to the ever changing business needs
4. lower level tasks can be tested independently of higher level tasks

Is this a totally crazy idea?



Mime
View raw message