commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "haruhiko nishi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (IO-200) CSV component
Date Sat, 21 Mar 2009 14:43:50 GMT

    [ https://issues.apache.org/jira/browse/IO-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688035#action_12688035
] 

haruhiko nishi commented on IO-200:
-----------------------------------

I know about commons-csv and I've looked at SuperCSV and others as well but they are not like
this.

> CSV component
> -------------
>
>                 Key: IO-200
>                 URL: https://issues.apache.org/jira/browse/IO-200
>             Project: Commons IO
>          Issue Type: New Feature
>          Components: Utilities
>            Reporter: haruhiko nishi
>            Priority: Trivial
>
> TableBuilder is 'Builder ' that maps the CSV to a matrix and provides interface that
allows user to manipulate after it is build by parsing a csv file to it parse() method.(There
is only one method implemented and it is for copying a column values to another position,
as I could not  think of other operation that may be useful)
> Within the TableBuilder, each column of the CSV is represented as byte[] and each becomes
a target to be validated against Rule,represented by the interface that you find in the example
code below. As TableBuilder "buildTable" ,when parse() method is invoked, a byte[]  representation
of the value of each CSV cell  is passed to isValid() method of implementations of Rules,
which you apply to the TableBuilder instance through the addRule() method. (you you can add
as many Rule as you need.)
> Rule gets executed until the validation fails or succeeds. If any of the Rule fails,
then its replace() is called and the column value being processed gets replaced by the retun
value of this method.
> Another goodie is that it is possible to refer to the values of preceding cell values
of the row within a Rule.
> It is useful if you need to see the entries of the preceding cell when validating the
value in a Rule. An example would be,
> Given a csv,
> A,B,C
> 1,2,3
> in order for the value  3 of the column C is to be validated true, the Value of A needs
to be less than the value of C.
> TableBuilder is RFC 4180 compliant and therefore distinguishes NL exists by itself and
NL found in double quotes.
> So you can add Rule that practically removes all NL chars found in value enclosed within
doublequotes. 
> (useful when you need to remove CRLF in double quotes from  CSV exported from Excel)
> Currently, TableBuilder implements a method called copyColumn with method signature of,
> copyColumn(Rule rule,int from,int to, boolean override) which allows user to manipulate
the parsed csv.
> What it does is literarly copies column from that is specified  at 'from' and to 'to'
position of the matrix.
> If override is true, the copying colum is overriden else the column is right shifted
and inserted at the specified position.
> You can specify some kind of Rule here to rewrite the value being copied from the origin.
> An example would be copy column value that all ends with .jpg or .gif and to the position
specified prefixing the column value with "http://some.server.com/imanges." after checking
the image exists, after checking that the named file exists at some location also by an implementation
of  another Rule.
> TableBuilder is just a "rough skecth" idea of CSV parsing.(The code below works fine
though) it still needs alot of refactoring and so.
> I appreciate any comment on this idea. What do you think? My code style sucks I know!

> Here is simple exampe to use TableBuilder.
> {code:title=TableBuilder|borderStyle=solid}
>     public static void main(String[] args)throws Exception{
>         TableBuilder tableBuilder=new TableBuilder("UTF-8",
>                 new MessageHandler(){
>                     public void handleMessage(String message) {
>                         System.err.println(message);
>                     }
>                 },0,true);
>         tableBuilder.addRule(3,new RemoveNLChars()); //removing NL cahracters found in
value.
>         tableBuilder.parse(new FileInputStream("test.txt"),TableBuilder.CSV);
>         List<Record> list=tableBuilder.getRowAsListOf(Record.class);
>         for(Record record:list)
>             System.out.println(record.getA());//TODO not implemented yet!
>         tableBuilder.writeTo(new FileOutputStream("test_mod.txt"),TableBuilder.CSV);
>     }
> public class RemoveNLChars extends StringValueRuleAdapter {
>     protected boolean isValid(String columnValue) {
>         return !columnValue.contains(System.getProperty("line.separator"));
>     }
>     protected String replace(String columnValue) {
>         return columnValue.replaceAll(System.getProperty("line.separator"),"");
>     }
>     public String getMessage() {
>         return "";
>     }
> }
> public interface Rule {
>     public void setRowReference(List<byte[]> rowReference);
>     public void setCharsetName(String charsetName);
>     boolean isValid(final byte[] columnValue);
>     byte[] replace(final byte[] columnValue);
>     String getMessage();
> }
> //StringValueruleAdapter is an adapter converts the byte[] representation of the cell
value.
> public  abstract class StringValueRuleAdapter implements Rule{
>     private String charsetName;
>     private List<byte[]> rowReference;
>     
>     public void setRowReference(List<byte[]> rowReference) {
>         this.rowReference=rowReference;
>     }
>     public void setCharsetName(String charsetName) {
>         this.charsetName=charsetName;
>     }
>     public final boolean isValid(final byte[] columnValue) {
>         String strValue;
>         try {
>             if(columnValue.length>0)
>                 strValue=(charsetName!=null) ? new String(columnValue,charsetName) :
new String(columnValue);
>             else
>                 strValue="";
>         } catch (UnsupportedEncodingException e) {
>             if(columnValue.length>0)
>                 strValue=new String(columnValue);
>             else
>                  strValue="";
>         }
>         return isValid(strValue);
>     }
>     public final byte[] replace(final byte[] columnValue) {
>         String strValue;
>         try {
>             if(columnValue.length>0)
>                 strValue=(charsetName!=null) ? new String(columnValue,charsetName):new
String(columnValue);
>             else
>                 strValue="";
>             return (charsetName!=null) ? replace(strValue).getBytes(charsetName):replace(strValue).getBytes();
>         } catch (UnsupportedEncodingException e) {
>             if(columnValue.length>0)
>                 strValue=new String(columnValue);
>             else
>                 strValue="";
>             return replace(strValue).getBytes();
>         }
>     }
>     protected String getRowValue(int column) {
>         try {
>             return (charsetName!=null) ? new String(rowReference.get(column),charsetName)
:
>                     new String(rowReference.get(column));
>         } catch (UnsupportedEncodingException e) {
>             return new String(rowReference.get(column));
>         } catch(IndexOutOfBoundsException noListFound){
>             throw new IllegalArgumentException("no value exists at the requested column.");
>         }
>     }
>     protected String getPrecedingRowValue(){
>         return getRowValue(rowReference.size()-1);
>     }
>     protected abstract boolean isValid(String columnValue);
>     protected abstract String replace(String columnValue);
> }
> public class TableBuilder {
>     public static int CSV=0x2c;
>     public static int TSV=0x09;
>     private Map<Integer,Set<Rule>> columnRule=new  HashMap<Integer,Set<Rule>>();
>     private Table currentTable;
>     private byte[] newLineChars;
>     private boolean endsWithNL;
>     private String charsetName;
>     private int rowOffset;
>     private boolean useFirstColumnAsRowName;
>     private MessageHandler msgHandler=new MessageHandler(){
>         public void handleMessage(String message) {
>             System.err.println(message);
>         }
>     };
>     public TableBuilder(String charsetName,MessageHandler msgHandler,int rowOffset,boolean
useFirstColumnAsRowName){
>         this.charsetName=charsetName;
>         this.rowOffset=rowOffset;
>         this.msgHandler=msgHandler;
>         this.useFirstColumnAsRowName=useFirstColumnAsRowName;
>     }
>     public TableBuilder(String charsetName){
>         this.charsetName=charsetName;
>     }
>     public TableBuilder(){
>         
>     }
>     public void addRule(int column, Rule rule){
>         Set<Rule> ruleset;
>         if((ruleset=columnRule.get(column))==null){
>             ruleset=new LinkedHashSet<Rule>();
>             columnRule.put(column,ruleset);
>         }
>         rule.setCharsetName(charsetName);
>         ruleset.add(rule);
>     }
>     public void parse(InputStream in, int delimiter)throws Exception{
>         int bytesRead;
>         byte buf[]=new byte[1024];
>         ByteArrayOutputStream outbuf=new ByteArrayOutputStream(buf.length);
>         while((bytesRead=in.read(buf,0,buf.length))!=-1)
>             outbuf.write(buf,0,bytesRead);
>         in.close();
>         ByteBuffer bytebuffer=ByteBuffer.allocateDirect(outbuf.size()).put(outbuf.toByteArray());
>         bytebuffer.flip();
>         currentTable=buildTable(bytebuffer,delimiter);
>     }
>     private class Table {
>         private List<byte[]>[] columnMatrix;
>         private List<List<byte[]>> rowMatrix;
>         
>         Table(List<byte[]>[] columnMatrix,List<List<byte[]>> rowMatrix){
>             this.columnMatrix=columnMatrix;
>             this.rowMatrix=rowMatrix;
>         }
>         public int getNumOfColumns() {
>             return columnMatrix.length;
>         }
>         public int getNumOfRows(){
>             return rowMatrix.size();
>         }
>         public byte[] getValueAt(int row, int column) {
>             return columnMatrix[column].get(row);
>         }
>         public byte[] getColumnName(int column){
>             return columnMatrix[column].get(0);
>         }
>         public List<byte[]> getColumn(int column){
>             return columnMatrix[column];
>         }
>         public List<byte[]> getRow(int row){
>             return rowMatrix.get(row);
>         }
>         
>     }
>   //TODO  extract csv row as JavaBean
>     public <E> List<E> getRowAsListOf(final Class<E> clazz){
>         List<E> list=null;
>         Iterator<byte[]> header=currentTable.getRow(0).iterator();
>         for(int i=1;i<currentTable.getNumOfRows();i++){
>             try {
>                 E instance=clazz.newInstance();
>                 for(byte[] value:currentTable.getRow(i)){
>                     String name=new String(header.next());
>                     //BeanUtils.setProperty(instance,name,value);
>                 }
>                 if(list==null)
>                     list=new ArrayList<E>();
>                 list.add(instance);
>                 header=currentTable.getRow(0).iterator();
>             } catch (IllegalAccessException e) {
>                 e.printStackTrace();
>             } catch (InvocationTargetException e) {
>                 e.printStackTrace();
>             } catch (InstantiationException e) {
>                 e.printStackTrace();
>             }
>         }
>         return list;
>     }
>     public void writeTo(OutputStream out,int delimiter) throws IOException {
>         for(int i=0,j=0;i<currentTable.getNumOfRows();i++,j=0){
>             for(byte[] value:currentTable.getRow(i)){
>                 out.write(value);
>                 if(++j<currentTable.getNumOfColumns())
>                     out.write(delimiter);
>             }
>             if(i<currentTable.getNumOfRows()-1)
>                 out.write(newLineChars);
>             else{                                             
>                 if(endsWithNL)
>                     out.write(newLineChars);
>             }
>         }
>         out.close();
>     }
>      
>      public void copyColumn(Rule rule,int from,int to, boolean override) {
>             int numOfColumns=override ? currentTable.getNumOfColumns():currentTable.getNumOfColumns()+1;
>             List<byte[]>[] columnMatrix=(List<byte[]>[])new List[numOfColumns];
>             columnMatrix[to]=new ArrayList<byte[]>();
>             for(int i=0,j=0;i<columnMatrix.length;i++){
>                 if(i==to){
>                     for(int row=0;row<currentTable.getNumOfRows();row++){
>                         byte[] value;
>                         if(row>=rowOffset)
>                             value=currentTable.getValueAt(row,from);
>                         else
>                             value=new byte[0];
>                         if(rule!=null && row>rowOffset){
>                             rule.setCharsetName(charsetName);
>                             rule.setRowReference(currentTable.getRow(row));
>                             if(!rule.isValid(value)){
>                                 String columnName;
>                                 byte[] columnNameByte=currentTable.getColumnName(from);
>                                 if(columnNameByte.length>0){
>                                     try {
>                                         if(charsetName!=null)
>                                             columnName="'"+new String(columnNameByte,charsetName).trim()+"'";
>                                         else
>                                             columnName="'"+new String(columnNameByte).trim()+"'";
>                                     } catch (UnsupportedEncodingException e) {
>                                         columnName="'"+new String(columnNameByte).trim()+"'";
>                                     }
>                                 }else
>                                     columnName="''";
>                                 value=rule.replace(value);
>                                 String msg=rule.getMessage();
>                                     if(msg.length()>0)
>                                         try {
>                                             handleMessage(msg
>                                             .replace("${column_from}",""+from)
>                                             .replace("${columnName}",columnName)
>                                             .replace("${column_to}",""+(to+1))
>                                             .replace("${row}",useFirstColumnAsRowName
? new String(currentTable.getRow(row).get(0),charsetName) : ""+(row+1)));
>                                         } catch (UnsupportedEncodingException ignored)
{
>                                             
>                                         }
>                             }
>                         }
>                     columnMatrix[i].add(value);
>                     if(override)
>                         currentTable.rowMatrix.get(row).remove(i);
>                     currentTable.rowMatrix.get(row).add(i,value);
>                     }
>                     if(override)
>                         ++j;
>                 }else
>                     columnMatrix[i]=currentTable.getColumn(j++);
>             }
>             currentTable=new Table(columnMatrix,currentTable.rowMatrix);
>     }
>     
>     private Table buildTable(ByteBuffer buf,int delimiter) throws ParseException {
>         List<byte[]>[] columnMatrix=null;
>         List<List<byte[]>> rowMatrix=new ArrayList<List<byte[]>>();
>         int i=0,j,currentRow=0,rowIndex=0,column_count=0,column=0;
>         endsWithNL=true;
>         newLineChars=null;
>         int limit=buf.limit();
>         int pos=0;
>         while(i<limit && ((j=(buf.get(i)&0xff))==0x0d||(j=(buf.get(i)&0xff))==0x0a)){
>             if(j==0x0a)
>                 ++currentRow;
>             pos=++i;
>         }
>         
>         int headRow=currentRow;
>         while(i<limit){
>             int tmp=buf.get(i) & 0xff;
>                 if(tmp==0x0a){
>                     int k=i;
>                     while(k>=0 &&((buf.get(k)&0xff)==0x0d||(buf.get(k)&0xff)==0x0a))
>                         --k;
>                     byte[] prev=new byte[++k-pos];
>                     
>                     buf.position(pos);
>                     buf.get(prev,0,prev.length);
>                     List<byte[]> row;
>                     try{
>                         row=rowMatrix.get(rowIndex);
>                     }catch(IndexOutOfBoundsException noListFound){
>                         rowMatrix.add(new ArrayList<byte[]>());
>                         row=rowMatrix.get(rowIndex);
>                     }
>                     if(currentRow==headRow){
>                         column_count=column;
>                         row.add(prev);
>                         columnMatrix=(List<byte[]>[])new ArrayList[column+1];
>                         Iterator<byte[]> itr;
>                         for(j=0,itr=row.iterator();j<columnMatrix.length;j++){
>                             columnMatrix[j]=new ArrayList<byte[]>();
>                             columnMatrix[j].add(itr.next());
>                         }
>                     }else if(column_count!=column){
>                         throw new ParseException("column count mismatch on row ",currentRow+1);
>                     }else{
>                         Set<Rule> ruleset=columnRule.get(column);
>                         if(ruleset!=null && currentRow>rowOffset+headRow){
>                             byte[] columnNameByte=rowMatrix.get(rowOffset).get(column);
>                             Rule rule=validate(ruleset,prev,row);
>                             if(rule!=null){
>                                 String columnName;
>                                 if(columnNameByte.length>0){
>                                     try {
>                                         if(charsetName!=null)
>                                             columnName="'"+new String(columnNameByte,charsetName).trim()+"'";
>                                         else
>                                             columnName="'"+new String(columnNameByte).trim()+"'";
>                                     } catch (UnsupportedEncodingException e) {
>                                         columnName="'"+new String(columnNameByte).trim()+"'";
>                                     }
>                                 }else
>                                     columnName="''";
>                                 prev=rule.replace(prev);
>                                 String msg=rule.getMessage();
>                                 if(msg.length()>0)
>                                     try {
>                                         handleMessage(msg
>                                             .replace("${column}",""+column)
>                                             .replace("${columnName}",columnName.trim())
>                                             .replace("${row}",useFirstColumnAsRowName
? new String(rowMatrix.get(rowIndex).get(0),charsetName) : ""+(currentRow+1)));
>                                     } catch (UnsupportedEncodingException ignored) {
>                                     }
>                             }
>                         }
>                         columnMatrix[column].add(prev);
>                         row.add(prev);
>                     }
>                     if(newLineChars==null){
>                         newLineChars=new byte[++i-k];
>                         buf.position(k);
>                         buf.get(newLineChars,0,newLineChars.length);
>                     }else
>                         ++i;
>                 while(i<limit && ((j=(buf.get(i)&0xff))==0x0d||(j=(buf.get(i)&0xff))==0x0a)){
>                     if(j==0x0a)
>                         ++currentRow;
>                     ++i;
>                 }
>                 column=0;
>                 ++currentRow;
>                 ++rowIndex;
>                 pos=i;
>             }else if(tmp==delimiter){
>                 List<byte[]> row;
>                 try{
>                     row=rowMatrix.get(rowIndex);
>                 }catch(IndexOutOfBoundsException noListFound){
>                     rowMatrix.add(new ArrayList<byte[]>());
>                     row=rowMatrix.get(rowIndex);
>                 }
>                 byte[] prev=new byte[i-pos];
>                 buf.position(pos);
>                 buf.get(prev,0,prev.length);
>                 if(currentRow==headRow)
>                     row.add(prev);
>                 else{
>                     Set<Rule> ruleset=columnRule.get(column);
>                     if(ruleset!=null && currentRow>rowOffset+headRow){
>                         byte[] columnNameByte=rowMatrix.get(rowOffset).get(column);
>                             Rule rule=validate(ruleset,prev,row);
>                             if(rule!=null){
>                                 String columnName;
>                                 if(columnNameByte.length>0){
>                                     try {
>                                         if(charsetName!=null)
>                                             columnName="'"+new String(columnNameByte,charsetName).trim()+"'";
>                                         else
>                                             columnName="'"+new String(columnNameByte).trim()+"'";
>                                     } catch (UnsupportedEncodingException e) {
>                                         columnName="'"+new String(columnNameByte).trim()+"'";
>                                     }
>                                 }else
>                                     columnName="''";
>                                 prev=rule.replace(prev);
>                         String msg=rule.getMessage();
>                         if(msg.length()>0)
>                             try {
>                                 handleMessage(msg
>                                         .replace("${column}",""+column)
>                                         .replace("${columnName}",columnName.trim())
>                                         .replace("${row}",useFirstColumnAsRowName ? new
String(rowMatrix.get(rowIndex).get(0),charsetName) : ""+(currentRow+1)));
>                             } catch (UnsupportedEncodingException ignored) {
>                             }
>                             }
>                     }
>                     columnMatrix[column].add(prev);
>                     row.add(prev);
>                 }
>                 ++column;
>                 pos=++i;
>             }else
>                 if((i=_ESCAPED(buf,i))==i)
>                     ++i;
>         }
>         if(pos!=limit){
>             endsWithNL=false;
>             byte[] remaining=new byte[limit-pos];
>             buf.position(pos);
>             buf.get(remaining,0,remaining.length);
>             
>             if(columnMatrix!=null){
>                 if(column_count!=column)
>                     throw new ParseException("column count mismatch on row ",+1+currentRow);
>                 List<byte[]> row=rowMatrix.get(rowIndex);
>                 row.add(remaining);
>                 Set<Rule> ruleset=columnRule.get(column);
>                 if(ruleset!=null && currentRow>rowOffset+headRow){
>                     byte[] columnNameByte=rowMatrix.get(rowOffset).get(column);
>                     Rule rule=validate(ruleset,remaining,row);
>                     if(rule!=null){
>                         String columnName;
>                         if(columnNameByte.length>0){
>                             try {
>                                 if(charsetName!=null)
>                                     columnName="'"+new String(columnNameByte,charsetName).trim()+"'";
>                                 else
>                                     columnName="'"+new String(columnNameByte).trim()+"'";
>                             } catch (UnsupportedEncodingException e) {
>                                 columnName="'"+new String(columnNameByte).trim()+"'";
>                             }
>                         }else
>                             columnName="''";
>                         remaining=rule.replace(remaining);
>                         String msg=rule.getMessage();
>                         if(msg.length()>0)
>                             try {
>                                 handleMessage(msg
>                                 .replace("${column}",""+column)
>                                 .replace("${columnName}",columnName.trim())
>                                 .replace("${row}",useFirstColumnAsRowName ? new String(rowMatrix.get(rowIndex).get(0),charsetName)
: ""+(currentRow+1)));
>                             } catch (UnsupportedEncodingException ignored) {
>                             }
>                     }
>                 }
>                 columnMatrix[column].add(remaining);
>             }else{
>                 columnMatrix=(List<byte[]>[])new List[column+1];
>                 List<byte[]> row;
>                 try{
>                     row=rowMatrix.get(rowIndex);
>                 }catch(IndexOutOfBoundsException noListFound){
>                     rowMatrix.add(new ArrayList<byte[]>());
>                     row=rowMatrix.get(rowIndex);
>                 }
>                 row.add(remaining);
>                 Iterator<byte[]> itr;
>                 for(j=0,itr=row.iterator();j<columnMatrix.length;j++){
>                     columnMatrix[j]=new ArrayList<byte[]>(1);
>                     columnMatrix[j].add(itr.next());
>                 }
>             }
>         }
>         return new Table(columnMatrix,rowMatrix);
>     }
>     private int _ESCAPED(ByteBuffer src,int i){
>         int org=i;
>         if(i==src.limit())
>             return i;
>         int j;
>         if((j=_DQUOTE(src,i))==i)
>             return i;
>         for(i=j;(j=_TEXTDATA(src,i))>i||(j=_COMMA(src,i))>i||(j=_CR(src,i))>i||(j=_LF(src,i))>i||(j=_2DQUOTE(src,i))>i;)
>             i=j;
>         if(i==_DQUOTE(src,i))
>             return org;
>         return i;
>     }
>     private int _TEXTDATA(ByteBuffer src,int i){
>         if(i==src.limit())
>             return i;
>         if(_COMMA(src,i)==i && _CR(src,i)==i && _LF(src,i)==i &&
_DQUOTE(src,i)==i)
>             return ++i;
>         return i;
>     }
>     private int _2DQUOTE(ByteBuffer src,int i) {
>         if(i==src.limit())
>             return i;
>         if(i==_DQUOTE(src,i))
>             return i;
>         if(i+1==_DQUOTE(src,i+1))
>             return i;
>         return i+2;
>     }
>     private int _DQUOTE(ByteBuffer src,int i) {
>         return _CHAR(src,i,0x22);
>     }
>     public int _LF(ByteBuffer src,int i) {
>         return _CHAR(src,i,0x0a);
>     }
>     private int _CR(ByteBuffer src,int i) {
>         return _CHAR(src,i,0x0d);
>     }
>     private int _COMMA(ByteBuffer src,int i) {
>         return _CHAR(src,i,0x2c);
>     }
>     private int _CHAR(ByteBuffer src,int i,int token){
>         if(i==src.limit())
>             return i;
>         if((src.get(i) & 0xff)==token)
>             ++i;
>         return i;
>     }
>      
>     private void handleMessage(String message) {
>         msgHandler.handleMessage(message);
>     }
>     
>     public Rule validate(Set<Rule> ruleset,byte[] value, List<byte[]> rowReference)
{
>         for(Rule rule:ruleset){
>             if(rule!=null){
>                 rule.setRowReference(rowReference);
>                 if(!rule.isValid(value))
>                     return rule;
>             }
>         }
>         return null;
>     }
> }
> {code} 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message