jmeter-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oliver Erlewein <oli...@erlewein.net>
Subject Re: CSV file iteration in remote tests
Date Mon, 30 Dec 2013 20:27:31 GMT
That looks cool. I'll have a look at if I'll use it.

Did you do some load testing on this? i.e. how does Redis need to scale to
cope with large tests? How does it affect JMeter tests?...

Cheers Oliver


On 31 December 2013 03:11, Philippe Mouawad <philippe.mouawad@gmail.com>wrote:

> Hello,
> As dev team didn't reach consensus on commiting the piece of code to
> JMeter core, it has been contributed to third party jmeter-plugins project,
> it should be in next version of the library.
> For those interested :
> Pull request:
>  - https://github.com/undera/jmeter-plugins/pull/22<https://github.com/undera/jmeter-plugins/pull/22>
>
> - Developer Snapshots download:
> -
> http://jmeter-plugins.org/downloads/file/nightly/jmeter-plugins-extras-libs-1.1.2-2013-12-30_06-16.zip
>
>
> Regards
> Philippe M.
> @philmdot
>
> On Thursday, November 7, 2013, Oliver Lloyd wrote:
>
>> I really like the idea of JMeter having the ability to use a central repo
>> of data via something like Redis. It solves not only the problem of
>> distributing unique data over multiple remote servers but also the problem
>> of data that can only be used once. I see Sebb's point though that this
>> solution will only help where the slaves have internet access - not always
>> the case. But it is the case more and more; at least in my experience, that
>> cloud-based hardware is the way to go so it makes sense to develop
>> solutions that support this.
>>
>> I think though that such an implementation should ideally abstract a lot
>> of the hassle from the user. If it would accept a csv file and a URI
>> connection string, handle the import and then internalise the methods to
>> pop / read the data, only exposing to the user some vars like ${pop_email}
>> or ${read_email}, then that would be the perfect. That way I could enter my
>> redis URI, pass a file ref to my csv and then know that each time the var
>> ${pop_email} is resolved I would have one less email in my datastore -
>> pretty cool.
>>
>>
>> On 7 Nov 2013, at 06:47, Philippe Mouawad <philippe.mouawad@gmail.com>
>> wrote:
>>
>> > you might be interested in this discussion:
>> > -
>> >
>> http://mail-archives.apache.org/mod_mbox/jmeter-dev/201310.mbox/%3cCAH9fUpZtWGR7nMhpCAROoKhsRhCGWLhksPb9CrD=8GVsRYAEsw@mail.gmail.com%3e
>> >
>> >
>> >
>> > On Thursday, November 7, 2013, Oliver Erlewein wrote:
>> >
>> >> Oooh! I like that idea! That would also make it easier for me to update
>> >> CSVs as they don't need to be distributed to the clients anymore! Would
>> >> mean a bit of work but would be totally worth it.
>> >>
>> >>
>> >>
>> >> On 7 November 2013 11:46, Tim Koopmans <tim@flood.io <javascript:;>>
>> >> wrote:
>> >>
>> >>> We approach this problem slightly differently, albeit we use an
>> external
>> >>> process ..
>> >>>
>> >>> Users upload a CSV which we bulk import into redis. Users can then
>> make
>> >>> HTTP requests to that data set via a performant API (webdis in front
>> of
>> >>> redis). This allows calls like SPOP (remove a random member) or
>> >> SRANDMEMBER
>> >>> (retrieve random member). This takes care of the headache of trying
to
>> >>> carve up CSVs across your load generators[1]
>> >>>
>> >>> A little bit more effort, but may give you some ideas.
>> >>>
>> >>> Regards,
>> >>>
>> >>> Tim Koopmans
>> >>>
>> >>> <https://flood.io>
>> >>>
>> >>> Level 27, 101 Collins Street
>> >>> Melbourne, Vic 3000
>> >>>
>> >>> [1] https://flood.io/blog/9-sharing-test-data
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Nov 7, 2013 at 9:36 AM, Oliver Erlewein <oliver@erlewein.net
>> <javascript:;>
>> >>> wrote:
>> >>>
>> >>>> Hi Sergio,
>> >>>>
>> >>>> Sorry forgot to say that that isn't a good option for me. I thought
>> >> about
>> >>>> it but I dynamically build remote agents off a standard build image
>> and
>> >>>> that makes my start procedure immensely difficult as I'd need to
>> split
>> >> the
>> >>>> files and copy them to the clients before the test. I also don't
>> have a
>> >>>> standard number of clients so the splits are different. If I do
that
>> I
>> >>>> could also just re-randomize my files too.
>> >>>>
>> >>>> Cheers Oliver
>> >>>>
>> >>>>
>> >>>> On 7 November 2013 11:28, Sergio Boso <sergio@bosoconsulting.it
>> <javascript:;>>
>> >> wrote:
>> >>>>
>> >>>>> Il 06/11/2013 23.20, Oliver Erlewein ha scritto:
>> >>>>>
>> >>>>> Hi all,
>> >>>>>>
>> >>>>>> I'm sure that this is a common problem for those using JMeter
>> >>>> executions
>> >>>>>> across several machines. Can't really find any solution
to this. So
>> >>>> here
>> >>>>>> goes:
>> >>>>>>
>> >>>>>> I have a plan that looks something like this:
>> >>>>>>
>> >>>>>> Test Plan
>> >>>>>>   |-- Thread Group
>> >>>>>>          |--CSV Dataset
>> >>>>>>          |--HTTP Sampler (login)
>> >>>>>>          |--....
>> >>>>>>
>> >>>>>> If I remotely distribute this all remotes will 1st start
with line
>> >> one
>> >>>> of
>> >>>>>> the CSV file. In my case this will cause locking in the
>> application,
>> >>>>>> thereby destroying the test. Ideally I'd like to give the
CSV file
>> a
>> >>>>>> random
>> >>>>>> offset for each remote client, so that it would start iterating
at
>> >>>> various
>> >>>>>> points in the CSV. This is not quite safe but should give
me enough
>> >>>>>> variance so that the chance of locking would be minimal.
>> >>>>>>
>> >>>>> The only way I have found to cope with this is to manually split
the
>> >> CSV
>> >>>>> file, and copy each of these parts to each remote system, so
that
>> each
>> >>>>> system uses its own set of lines.
>> >>>>>
>> >>>>> I'm looking forward to see if there is a better system
>> >>>>> regards
>> >>>>>
>> >>>>> Sergio
>> >>>>>
>> >>>>> --
>> >>>>>
>> >>>>> Ing. Sergio Boso
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message