It is possible to write parameterized tests, that get run multiple times with different parameters. RandomizedRunner needs to be used rather than the Parameterized runner provided with junit (all the base test classes already use RandomizedRunner though). The method that provides the parameters needs to be annotated with the @ParametersFactory annotation and must be static, public, without arguments and must have a return type assignable to Iterable<Object[]>. The iterable must return arrays conforming to the suite class’s constructor with respect to the number and types of parameters. The constructor’s parameters can be annotated with the @Name annotation to provide more descriptive parameter names for test descriptions. Have a look at ElasticsearchRestTests for an example.