Our Experimental Setup

Some of the key issues have been examined previously, directly or indirectly. For those, we simply summarize the results in the subsections that follow. However, some are open questions. For those, we ran seven well known planners on a large set of 2057 benchmark problems. The planners all accept the PDDL representation, although some have built-in translators for PDDL to their internal representation and others rely on translators that we added. When several versions of a planner were available, we included them all (for a total of 13 planners). The basic problem set comprises the UCPOP benchmarks, the AIPS98 and 2000 competition test sets and an additional problem set developed for a specific application.

With the exception of the permuted problems (see the section on Problem Assumption 2 for specifics), the problems were run on 440 MHz Ultrasparc 10s with 256 Megabytes of memory running SunOS 2.8. Whenever possible, versions compiled by the developers were used; when only source code was available, we compiled the systems according to the developers' instructions. The planners written in Common Lisp were run under Allegro Common Lisp version 5.0.1. The other planners were compiled with GCC (EGCS version 2.91.66). Each planner was given a 30 minute limit of wall clock time³ to find a solution; however, all times reported are run times returned by the operating system.