jmeter-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sebb <seb...@gmail.com>
Subject Re: How to read a zipped text file and iterate over all the lines
Date Wed, 10 Jul 2013 11:09:22 GMT
As already noted:

Both CSV DataSet and StringFromFile have to handle repeated reads from
the same file.
They do this without closing and re-opening (except at EOF).

It should be simple to amend either or both for your needs; just
replace the file IO with zip file IO.
Apache Commons Compress has Zip code if you need it.

On 10 July 2013 11:31, Janusz Kowalczyk <kowalczykjanusz@gmail.com> wrote:
> btw. I know that closing & reopening a BufferedReader for the same file is
> lame, but I don't have enough time to optimize it :)
>
>
> On 10 July 2013 11:28, Janusz Kowalczyk <kowalczykjanusz@gmail.com> wrote:
>
>> OK, I solved this problem by:
>> adding "Only once controller" to the thread group
>> putting JSR223 Sampler to that controller
>> adding a "ForEach Controller" after that controller and iterating over all
>> the "CONTEXT" variables.
>>
>> "ForEach Controller" configuration looks like:
>> Input variable prefix: CONTEXT
>> Start index for loop: 0
>> End index for loop: ${LINE_CNT}
>> Output variable name: CONTEXT
>> Select "Add "_" before number?"
>>
>>
>> One thing to remember is that when the txt file inside the zip archive is
>> big, then you need to increase the heap size!
>>
>> *And here's the code:*
>> import java.io.*;
>> import java.util.zip.ZipEntry;
>> import java.util.zip.ZipFile;
>> import org.apache.log.Logger;
>> import org.apache.jmeter.threads.JMeterVariables;
>> import org.apache.jmeter.util.JMeterUtils;
>> import java.io.IOException;
>> import org.apache.jmeter.services.FileServer;
>>
>> /**
>> * A Class that reads text file inside a zip archive and sets the as many
>> "CONTEXT_{x}" variables as the number of lines in the text file.
>> *
>> * Author: Janusz Kowalczyk
>> * Created: 2013-07-09
>> **/
>> public class JmeterZipReader {
>>     private String zipFile;
>>     private Logger log;
>>     private JMeterVariables vars;
>>
>>     JmeterZipReader(String zipFile, Logger log, JMeterVariables vars){
>>         this.zipFile = zipFile;
>>         this.log = log;
>>         this.vars = vars;
>>         File f = new File(zipFile);
>>         if( !f.exists()) {
>>                log.error("FILE '"+zipFile+"' DOESN'T EXISTS!!!");
>>                throw new IOException("FILE '"+zipFile+"' DOESN'T
>> EXISTS!!!");
>>         }
>>     }
>>
>>     public void read(String textFile){
>>         ZipFile zip;
>>         ZipEntry ze;
>>         InputStream input;
>>         BufferedReader br;
>>
>>         try {
>>             log.info("\n/******************************************\nOpening
>> file " + textFile + " from zip archive: " + this.zipFile +
>> "\n******************************************/");
>>             zip=new ZipFile(this.zipFile);
>>             ze=zip.getEntry(textFile);
>>             input = zip.getInputStream(ze);
>>             br = new BufferedReader(new InputStreamReader(input,
>> "US-ASCII"));
>>             log.info("Reading file: " + textFile);
>>
>>             int LINE_CNT = 0;
>>             while (br.readLine() != null) LINE_CNT++;
>>             // you can limit the number of rows to a specific number
>>             // to avoid problems with the heap space
>>             //if (LINE_CNT >= 35000) {
>>             //         LINE_CNT = 35000;
>>             //}
>>             vars.put("LINE_CNT", Integer.toString(LINE_CNT));
>>             br.close();
>>             zip.close();
>>             log.info("\n/******************************************\ntext
>> file inside the zip archive has: " + LINE_CNT + "
>> lines!!!\n******************************************/");
>>
>>
>>             log.info("\n/******************************************\nReopeing
>> the zip and csv file for setting up all the
>> variables!!\n******************************************/");
>>             zip=new ZipFile(this.zipFile);
>>             ze=zip.getEntry(textFile);
>>             input = zip.getInputStream(ze);
>>             br = new BufferedReader(new InputStreamReader(input,
>> "US-ASCII"));
>>             String line;
>>             for (int i=0; i < LINE_CNT; i++) {
>>                 line = br.readLine();
>>                 vars.put("CONTEXT_"+i, line);
>>             }
>>             br.close();
>>             zip.close();
>>             log.info("\n/******************************************\nZIP
>> & CSV files were closed!!\n******************************************/");
>>         }
>>         catch (Exception e) {
>>             log.error("Unhandled exception:");
>>             e.printStackTrace();
>>         }
>>     }
>> }
>>
>> String zipFilePath = FileServer.getFileServer().getBaseDir() +
>> "/data/your_zipped_txt_file.zip";
>> JmeterZipReader jzr = new JmeterZipReader(zipFilePath, log,
>> ctx.getVariables());
>> jzr.read("name_of_the_txt_file_inside_the_zip_file.txt");
>>
>>
>> Cheers,
>> Janusz
>>
>>
>> On 9 July 2013 19:37, Deepak Shetty <shettyd@gmail.com> wrote:
>>
>>> >that I don't wan't to unzip those files,
>>> The question is why do you want to do that? whats the benefit? - Again you
>>> have setup and tear down threadgroups so you can do it before your test
>>> runs and clean it up after its done (or as part of your build) - note you
>>> have to factor in multiple threads too ..
>>>
>>>
>>> On Tue, Jul 9, 2013 at 2:28 AM, Janusz Kowalczyk
>>> <kowalczykjanusz@gmail.com>wrote:
>>>
>>> > The thing is that I don't wan't to unzip those files, but just read
>>> > directly from them.
>>> >
>>> > OK, maybe easier way would be just returning a list or a map of
>>> variables,
>>> > so that I could iterate over it?
>>> >
>>> > ps. I put this strange loop just for the sake of the demo, so it doesn't
>>> > read more that 10 lines at a time :)
>>> >
>>> >
>>> >
>>> > On 8 July 2013 23:35, Deepak Shetty <shettyd@gmail.com> wrote:
>>> >
>>> > > this.vars.put(this.outputVariable, line); ==> Always overwrites
the
>>> > current
>>> > > value
>>> > > should be the last line however unless your loop is also incorrect
>>> > >
>>> > > But the better way is to get your data in the form you want first ,
>>> then
>>> > > start the test rather than the test writing some code to read the data
>>> > i.e.
>>> > > unzip your files whatever before you mai n test begins (as part of
>>> your
>>> > > build or as a startup threadgroup )
>>> > >
>>> > >
>>> > > On Mon, Jul 8, 2013 at 12:42 PM, Janusz Kowalczyk <
>>> > > kowalczykjanusz@gmail.com
>>> > > > wrote:
>>> > >
>>> > > > Hi All,
>>> > > >
>>> > > > I'd like to create a script that would read a zipped text file,
and
>>> for
>>> > > > example for each line (passed as variable) send a HTTP request.
>>> > > >
>>> > > > So far, I've written a script that works partially :)
>>> > > > Partially, because sending a new line to log.info() prints out
this
>>> > new
>>> > > > line (which is good :) ), but when I set the value of a variable
to
>>> > newly
>>> > > > read line, then the value of that variable remains the same and
is
>>> > always
>>> > > > equal to the first line of that text file.
>>> > > >
>>> > > > Please find the code below. To test it:
>>> > > > - create a simple text file with just few lines of random text,
>>> > > > - zip that text file
>>> > > > - Add a JSR223 preprocessor & set the script language to Java
>>> > > > - configure the script accordingly
>>> > > > - add a HTTP sampler and add ${CONTEXT} to the "RAW Post Body"
field
>>> > > > - check the msg value of the http request in "View Results Tree"
>>> > > >
>>> > > >
>>> > > > import java.io.*;
>>> > > > import java.util.zip.ZipEntry;
>>> > > > import java.util.zip.ZipFile;
>>> > > > import org.apache.log.Logger;
>>> > > > import org.apache.jmeter.threads.JMeterVariables;
>>> > > >
>>> > > > public class JmeterZipReader
>>> > > > {
>>> > > >     private String zipFile;
>>> > > >     private Logger log;
>>> > > >     private String outputVariable;
>>> > > >     private JMeterVariables vars;
>>> > > >
>>> > > >     JmeterZipReader(String zipFile, Logger log, String
>>> outputVariable,
>>> > > > JMeterVariables vars){
>>> > > >         this.zipFile = zipFile;
>>> > > >         this.log = log;
>>> > > >         this.outputVariable = outputVariable;
>>> > > >         this.vars = vars;
>>> > > >     }
>>> > > >
>>> > > >     // textFile is the name of the text file inside the zip file
>>> > > >     public void read(String textFile){
>>> > > >         ZipFile zip;
>>> > > >         ZipEntry ze;
>>> > > >         InputStream input;
>>> > > >         BufferedReader br;
>>> > > >
>>> > > >         try {
>>> > > >             zip=new ZipFile(this.zipFile);
>>> > > >             ze=zip.getEntry(textFile);
>>> > > >             input = zip.getInputStream(ze);
>>> > > >             br = new BufferedReader(new InputStreamReader(input,
>>> > > > "US-ASCII"));
>>> > > >
>>> > > >             String line;
>>> > > >             int cnt = 0;
>>> > > >             while((line = br.readLine()) != null && cnt
<= 10) {
>>> > > >                 this.vars.put(this.outputVariable, line);
>>> > > >                 log.info(line);
>>> > > >                 cnt += 1;
>>> > > >             }
>>> > > >
>>> > > >             br.close();
>>> > > >             zip.close();
>>> > > >         }
>>> > > >         catch (Exception e) {
>>> > > >             log.error("Unhandled exception:");
>>> > > >             e.printStackTrace();
>>> > > >         }
>>> > > >     }
>>> > > > }
>>> > > >
>>> > > > JmeterZipReader jzr = new JmeterZipReader("/path/to/a/test.zip",
>>> log,
>>> > > > "CONTEXT", ctx.getVariables());
>>> > > > jzr.read("test.csv");
>>> > > >
>>> > > >
>>> > > >
>>> > > > Many thanks,
>>> > > > Janusz
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>>
>>
>
>
> --
> Zapraszam na swojego foto-bloga: http://na100procentchyba.wordpress.com/
> Autopodpis: Staraj się używać pola Ukryty do Wiadomości (UDW) przy
> wysyłaniu wiadomości do wielu odbiorców, ograniczysz przez to
> rozprzestrzenianie się spamu!
> Autosignature: Try to use field BCC (blind carbon copy) when sending
> message to many recepients, it will restrain spread of spam!

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org


Mime
View raw message