Problems

Many planning systems were developed to solve a particular type of planning problem or explore a specific type of algorithmic variation. Consequently, one would expect them to perform better on the problems on which and for which they were developed. Even were they not designed for a specific purpose, the test set used during development may have subtly biased the development. The community knows that planner performance depends on problem features, but not in general, how, when and why. Researchers tend to design planners to be general purpose. Consequently, comparisons assume that

the performance of a general-purpose planner should not be penalized/biased if executed on a sampling of problems and domains (problem assumption 1).

The community also knows that problem representation influences planner performance. For example, benchmark problem sets include many versions of Blocksworld problems, designed by different planner developers. These versions vary in their problem representation, both minor apparently syntactic changes (e.g., how clauses are ordered within operators, initial conditions and goals, and whether any information is extraneous) and changes reflecting addition of domain knowledge (e.g., what constraints are included and whether variables are typed). Consequently, comparisons assume that

syntactic representational modifications either do not matter or affect each planner equally (problem assumption 2).

PDDL includes a field, :requirements, for the capabilities required of a planner to solve the problem. PDDL1.0 defined 21 values for the :requirements field; the base/default requirement is :strips, meaning STRIPS derived add and delete sets for action effects. :adl (from Pednault's Action Description Language) requires variable typing, disjunctive preconditions, equality as a built-in predicate, quantified preconditions and conditional effects in addition the :strips capability. Yet, many planners either ignore the :requirements field or reject the problem only if it specifies :adl (ignoring many of the other requirements that could also cause trouble). Thus, comparisons assume that

problems in the benchmark set should be solvable by a STRIPS planner unless they require :adl (problem assumption 3).