flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JOHN MILLER <jmill...@gmail.com>
Subject Fwd: DATA TRANSFORMATION PROBLEM
Date Fri, 04 Dec 2015 15:39:06 GMT
---------- Forwarded message ----------
From: JOHN MILLER <jmill383@gmail.com>
Date: Fri, Dec 4, 2015 at 10:24 AM
Subject: DATA TRANSFORMATION PROBLEM
To: info@data-artisans.com




*Greetings*


*I am writing to obtain an approach to resolve a data transformation
problem  The problem is that I want to format a new dataset which would
allow processing continue instead of bombing.  The dataset i want to
convert is a series of WARC files (currently read in as text...examples are
attached)​ CC-MAIN-TEXT-20130516092621-00003-ip-10-60-113-...
<https://drive.google.com/file/d/0B5QdPKF22EFxMDlTX3BzSW9uTDg/view?usp=drive_web>​
I am trying to parse out a field names and values and format a new dataset
which would then be converted to CSV or TSV*











*The fields in questionHeader: {WARC-Type=warcinfo,
WARC-Filename=CC-MAIN-TEXT-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz,
reader-identifier=/home/jmill383/wdcdemobucket/CC-MAIN-TEXT-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz,
WARC-Date=2013-11-22T14:51:12Z, absolute-offset=0, Content-Length=372,
WARC-Record-ID=<urn:uuid:efdf19de-e663-4747-8a98-754bd224520f>,
Content-Type=application/warc-fields}URL: null*


*Please advise if you can assist with an approach to resolve this problem
I am using Apache Flink...Scalding ....Scala   I havent been able to get
too far as of yet  Please advise if you can assist*

*John M*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message