Planners

The wonderful trend of making planners publicly available has led to a dilemma in determining which to use and how to configure them. The problem is compounded by the longevity of some of these planner projects; some projects have produced multiple versions. Consequently, comparisons tend to assume that

the latest version of the planner is the best (planner assumption 1).

These planners may also include parameters. For example, the blackbox planner allows the user to define a strategy for applying different solution methods. Researchers expect that parameters affect performance. Consequently, comparisons assume that

default parameter settings approximate good performance (planner assumption 2).

Experiments invariably use time cut-offs for concluding planning that has not yet found a solution or declared failure. Many planners would need to exhaustively search a large space to declare failure. For practical reasons, a time out threshold is set to determine when to halt a planner, with a failure declared when the time-out is reached. Thus, comparisons assume that

if one picks a sufficiently high time-out threshold, then it is highly unlikely that a solution would have been found had slightly more time been granted (planner assumption 3).