Rules

The competition will consider two tracks:

  • PDDL-target track: modeling will be done in PDDL 3.1, the same language used in IPCs 2014 and 2011. As in IPCs, PDDL features from 2.1 to 3.1 can be used. Considered features include core-features for Agile and Temporal tracks of IPC 2014.
  • Open track: modeling will be done in a language of the competitors’ choice. Competitors must submit the target planner to be used in the evaluation to the judges, and a description of the modeling language used.

Stages of the competition

  • Pre-competition. Organisers will give out a sample scenario, so that the teams will get an idea of what to expect in the competition.
  • On-site Modeling. Competitors will have a limited amount of time to encode the given scenarios in formal models, possibly exploiting specifically developed (or already existing) tools.
  • A demonstration given in a workshop-like format, in one of the workshop days at ICAPS 2016. The demonstration session will be open and interactive in which the audience will be able to raise discussions and make questions. The presentation will focus on used tools and issues encountered during the Knowledge Engineering process of the scenarios.

Evaluation criteria

Teams will be evaluated after the on-site encoding process. Evaluation will be based on both quantitative and qualitative evidences, and will focus on three aspects:

  • KE tools exploited.
  • Model characteristics.
  • Observed planners’ performance.

The KE tools exploited aspect will be evaluated using the following criteria:

  • Number and typology of Tools exploited.
  • KE step in which a tool has been exploited.
  • Originality of the tool (if developed by the team).

For each given scenario, the models will be evaluated using criteria such as:

  • Presence of bugs
  • Validity (does it actually encode the provided scenario?)
  • # types
  • # predicates
  • # operators
  • Mean (max) parameters
  • Mean (max) preconditions
  • Mean (max) eff
  • Readability (e.g., name of objects, predicates, presence of comments…)

For each scenario, the domain model and PDDL problems will be empirically evaluated using a set of planners from IPCs (PDDL-target track) or solvers provided by the competitors (open track):

  • Runtime
  • Size and shape of state space (e.g., number of instantiated actions, mutexes, …)
  • Quality of plans

As for the PDDL-target track, a preliminary list of planners is: LAMA, Madagascar, LPG, probe for classical planning, and YAHSP3, Temporal-Fast Downward, OPTIC, popf2 and LPG for temporal problems.

The winners of the competition will be decided by a board of judges after a careful evaluation of the above criteria and the workshop presentation. During the workshop presentation, teams will be given the opportunity to present and discuss the tools exploited (and the way tools have been used) as well as the formal models design decisions.

The final overall score of a team will be given by summing the evaluation obtained for each of the above criteria. Scores will be assigned by the judges board.

All the generated PDDL models will be made publicly available.


Awards

A prize will be given to the best team, based on the evidence given in the submitted model and at the workshop demonstration, while a prize will be given to the most innovative (set of) modeling tool(s). The winners will be announced at the ICAPS 2016 best paper award ceremony.

Advertisements