We did not run any experiments for this assumption because not all of the planners have parameters and because it is clear from the literature that the parameters do matter. Blackbox relies heavily on random restarts and trying alternative SAT solvers. In Kautz and Selman (1999), the authors of blackbox carefully study aspects of blackbox's design and demonstrate differential performance using different SAT solvers; they propose hypotheses for the performance differences and are working on better models of performance variation.
At the heart of HSP is heuristic search. Thus, its performance varies depending on the heuristics. Experiments with both HSP and FF (a planner that builds on some ideas from HSP) have shown the importance of heuristic selection in search space expansion, computation time and problem scale up [Haslum Geffner 2000,Hoffmann Nebel 2001].
As with HSP, heuristic search is critical to UCPOP's performance. A set of studies have explored alternative settings to the flaw selection heuristics employed by UCPOP [Joslin Pollack 1994,Srinivasan Howe 1995,Genevini Schubert 1996], producing dramatic improvements on some domains with some heuristics. As Pollack et al. (1997) confirmed, a good default strategy could be derived, but its performance was not the best under some circumstances.
Thus, because parameters can control fundamental aspects of algorithms, such as their search strategies, the role of parameters in comparisons cannot be easily dismissed.