A proven process for creating a prioritisation system

Point*Wizard is a decision-making and prioritisation tool with a myriad of uses, including ranking roading project proposals, appraising investment options, assessing environmental risks, etc. To simplify the following discussion, it is assumed for this page that you want to create a points system for the purpose of prioritising people; e.g., patients on a waiting list, immigrants applying for entry to a country, or students competing for scholarships.

To get the most out of Point*Wizard, your use of the software is best set within a wider managed process. This page is a guide, based on our experience, to help you plan such a process to create a new prioritisation system.

Suggested Steps

1.   Gather preliminary information

Determine what is already known about the subject under consideration. What is the problem you are trying to solve? Who (or what) is to be prioritised and for what purpose? What are other organisations doing in this area? How do they make such decisions?

If you are reviewing an existing prioritisation system, how was it created? What needs to be improved about it? Is the system currently being used or is it being avoided or 'fiddled' (or 'gamed') in some way?

Who are the key stakeholders, the budget managers, the dissenters and the champions? How will you draw these people together? Will you need to convince them that it is their job or that it is worthwhile? Who will lead and with what authority?

2.   Demonstrate the need for a new prioritisation system

Consider the possibility of demonstrating the need for a new system. Here are some ways this may be done:

• Ask multiple decision makers to rank person profiles ('vignettes') and compare the decision makers' rankings to assess their consistency.
• Show that with the current system, individuals are not prioritised in the way that one would expect. In other words, there is little or no 'face validity'.
• Review the literature for any methodological weaknesses of the current system', or the way in which it was created.

Consider how you might later use the above tests to demonstrate how the situation has been improved by the new prioritisation system. Will there be increased consistency? Could you assess the individuals being ranked with the old and the new systems to demonstrate an improvement? Will other statistics demonstrate such an improvement; e.g., a reduction in the number of deaths while waiting for surgery?

3.   Draft the decision criteria and categories

What criteria will be used for prioritisation? How will you establish these? Some means of doing this are:

• Review any evidence that is available from research.
• Ask experts in the area and others who have witnessed the system in action.
• Brainstorm with groups � perhaps a cross-section of society (or other relevant stakeholders).
• Compare a set of person profiles (vignettes) and consider how they differ. Which of these differences matter for the prioritisation exercise?

Once the criteria are identified, you need to determine the appropriate categories for each criterion. These are likely to come from the same sources as above. However, you may wish to perform more in-depth analysis, such as assigning a sub-group of participants to research a particular criterion.

Try to differentiate between criteria that are used to pre-emptively exclude individuals from those used to rank (non-excluded) individuals. For example, patients might be excluded from being considered for treatment if they are not expected to benefit from the treatment; or children may be excluded for legal reasons. Such exclusion criteria may be noted and later applied prior to the prioritisation system's use. With reference to the above examples, this is not to say that the criteria 'degree of benefit' and 'age' per se should not be included in the prioritisation system; rather it is the categories that would exclude the individual from further consideration that should not appear in the system. In essence, the objective is to simplify the overall process by considering eligibility separately from ranking or priority.

If you feel you're getting bogged down at this step of the process, consider moving on to the next step and returning to step 3, as may be required.

4.   Pre-test the criteria and categories using Point*Wizard

This step is primarily about pre-testing the criteria and categories, in contrast to the next step (step 5), which is about determining the relative weights of the criteria/categories. Nonetheless, these steps can be combined if desired.

Begin by loading the draft criteria and categories into Point*Wizard using the 'Criteria' page.

Next get participants to "give it a go". Point*Wizard is based on decisions involving tradeoffs between alternatives defined on the criteria/categories.

At this step of the process, however, Point*Wizard is used to enable participants to consider such tradeoffs in order to help them refine the criteria and categories. Specifically, this will help participants to identify criteria and categories that can be dropped or amalgamated, those that should be split, and those requiring more research or greater clarity. The wordings of the criteria and the categories, as well as the preference question (e.g., "Who should be treated first?") are also refined.

Ideally this should be done in groups as there will be frequent discussion: Is it the individual on the left or the right that is preferred? Or are they equal? Such discussion may lead back to further refinement of the criteria and categories, or their definitions. Or it may rest finally on expert judgements or values; � if so, the dilemma may be skipped, as this is dealt with by the next step.

Where appropriate, keep a record of how the criteria and categories are to be interpreted, perhaps including real examples of individuals defined on the criteria/categories with which participants may be familiar.

Note that it is with this step that most participants will have experienced Point*Wizard for the first time. Therefore it is worthwhile having someone who is very familiar with Point*Wizard on hand to answer questions about its workings and to allay any unwarranted fears. Although this step is primarily about reflecting on the criteria and categories (in order to refine them), it is important that participants take care over their decisions, as it is from them that insights into the criteria/categories emerge.

This and the previous step may be repeated until participants have confidence in the criteria/categories (and, ideally, consensus is reached).

5.   Elicit preferences using Point*Wizard

This step is a natural progression from the previous one, but it should be marked by an understanding having been reached amongst participants that the criteria and categories are right, or right enough for now to proceed. If appropriate, they might be told that there will be future opportunities for revisions to be made.

Naturally, the participants in the process could change at this step. Someone who is good at specifying the appropriate criteria (because of their expertise), or who is a knowledgeable about a particular criterion, is not necessarily the right person to be making prioritisation decisions. Depending on the decision situation, it may be appropriate for members of the general public ('stakeholders') or an organisation's senior managers to take a more active role in the present step.

Using Point*Wizard, decision makers may individually calibrate their system, or they may do so together in a single group, or in several smaller groups. The appropriate level of aggregation is influenced by decision maker availability, the extent of unanimity and the group dynamics observed at the previous step. Needless to say, logistically, this step is much simplified when it is performed via a single group.

Group decision-making

Typically (single) group decision-making is implemented from a single location with Point*Wizard projected onto a screen (or a wall) with a computer projector. If the group is small enough, each dilemma may be discussed and a decision reached by consensus. For larger groups, decision makers may vote. This may be done by: a show of hands ("Hands up for the alternative on the left?... For the right?... Equal"?..."); by raising one of three cards ('Left', 'Right', 'Equal'); by secret ballot; or by electronic voting tools like those used for audience polling (e.g., see http://www.votestation.co.uk).

Alternatively, Point*Wizard may be applied via a telephone conference with decision makers viewing the same dilemmas on the Internet and using Point*Wizard's built-in online voting system. This can potentially save considerable time and avoid travel costs.

Clearly, a key issue is what to do if not everyone agrees with a particular decision. one simple solution is to specify a majority threshold; e.g., for an alternative to 'win' the vote a 75% majority is required''. If such a threshold is not met, further discussion might be undertaken and another vote held. If agreement still can't be found, the group may choose to simply "Skip this dilemma for now".

Note that Point*Wizard is very flexible with respect to the criteria and/or categories being changed as additional insights are gained via the above-mentioned group discussions. If a decision to change the criteria and/or categories is reached, this can be done quickly, and the process resumed. In addition, the "Comment (optional)" field can be used to record vote numbers or discussion points if desired.

Comparing multiple preference elicitation sessions

Then a system has been calibrated individually by multiple individuals or multiple groups, their respective systems can be compared using Point*Wizard's Multi-Compare function. The effects of different decision makers' preferences can be illustrated in a number of ways: by comparing point values; by comparing the resulting rankings of a set of vignettes; by spearman rank correlation coefficients; and by the decisions that are agreed or disagreed among the multiple systems.

Areas of concordance (i.e., agreed decisions) can be retained to minimise future reworkings of the system. Attention is then drawn to areas of discordance (contradictory decisions, where one decision makers prefers the 'left' alternative, another prefers the 'right'). If possible, individuals should be brought together to resolve these contradictions by consensus (if such resolutions are possible � naturally, they might not be). Again, such discussion may lead back to further refinement of the criteria and categories, or their definitions, or on expert judgements or values. As decisions are reached, a new system can be calibrated.

Towards the end of the calibration process, any remaining contentious dilemmas may be regarded as being very fine distinctions. In such cases, consensus to accept equality (indifference) as a compromise may be appropriate so that a new or better (though perhaps imperfect) prioritisation system can be put into effect. The system can be refined later, if desired.

6.   Test the new system

The newly created system can be tested using the same methods referred to above (step 2) for testing current (old) systems. In particular, applying the new points system to a set of person profiles (vignettes) and comparing the resulting ranking with the ranking produced by an old system is usually very revealing. The particular decisions reached by decision makers at the previous step (step 5) may also be shared with others to test their acceptability ('face validity').

7.   Implementation

The benefits of the system need to be communicated to the people it will affect, and then procedures introduced. Paper or electronic forms are easily generated from Point*Wizard, and it is very simple to apply the new points system to system or database logic to produce new rankings of individuals.

Consideration might be given to include a space on such forms for a comment to be recorded so that those who apply the system can note special circumstances (criteria) that should be considered in relation to the individual. This can be used immediately to affect the ranking, or later in periodic reviews of the system.

Ways to demonstrate the validity and reliability of the new system should also be considered, such as, running the two systems (the old and the new) in parallel, and continuing to gather related statistics in the same way.

8.   Reviews and fine-tuning the system

The system can be continually or periodically reviewed as new research emerges providing more information about the appropriate criteria and categories, or from users' comments on forms (mentioned above) or changes in stakeholder priorities. Sub-groups of decision makers can be reconvened to just review those decisions that are affected by the change.

In addition, the decisions themselves can be reassessed periodically; e.g., if a decision maker feels that a particular decision should be changed, this can be shared with the rest of the group at any time (perhaps simply by email), and the single decision changed, resulting in an updated system.

Point*Wizard is flexible enough to enable such changes to be made easily. The software's incremental processes mean significantly lower costs of maintenance, and with a better maintained system, improved outcomes and greater stakeholder confidence.