Is client involvement in simulation studies beneficial for future problem solving?

It has been argued that involving clients in operational research studies, particularly model building, is beneficial for the successful implementation of study findings. However, there is very little evidence, besides the anecdotal, that this relationship actually exists. The operational research literature also often comments on the assumption that modelling in a social context leads to a deeper learning about concepts that can be transferred and adapted for future problem solving.

A recent paper¹ tests the impact of client involvement in a discrete event simulation study on the ability of the clients to transfer their learning to another context.

A behavioural experiment was conducted in which novice simulation users were involved in simulated training addressing a queuing problem is a fictitious hospital emergency department. The researchers created a condensed simulation study process with a modeller and a client so that participants could be involved in model building. Participant involvement in the model building was manipulated through three experimental conditions: low involvement in model building (participants reuse a pre-existing model), high involvement in model building with limited time for experimentation, and high involvement in model building with 45 minutes of additional time enabling more extensive simulation.

Eight transfer problem scenarios were created with varying distances of surface similarity to the emergency department problem. Surface similarity refers to whether an individual perceives the training task and transfer task to be similar. The transfer problems classified as “close” have high surface and structural similarity to the training problem, while the transfer problems classified as “far” have low surface similarity but are still structurally similar (i.e. a queuing problem).

Close transfer scenarios:

1. GP surgery waiting times: Participants must consider two options: running the clinic as a drop-in system or making use of appointment slots.

2. Emergency department waiting times: Participants must consider how to manage the resource levels on a shift in the emergency department in face of an increase in demand. Summary statistics for the waiting time and utilisation of resources are provided.

3. Operating theatre waiting times: Participants are asked to consider three options for reducing theatre waiting times. Attempt to maximise current utilisation, increasing capacity and extending the duration of slots.

4. Receptionist waiting times: A walk in centre and a GPs surgery are being co-located. Participants are asked to consider if the queues for reception should be pooled or un-pooled.

Far transfer scenarios:

5. Food manufacturing cycle time: Participants are asked to how they would reduce the lead time of a three part assembly process with deterministic arrivals and no breakdowns. Summary statistics of the utilisation of each work station are provided.

6. Police call centre waiting times: Participants must consider if performance will be improved at a call centre, by increasing resource, pooling resource or creating dedicated resource.

7. Food manufacturing cycle time: Participants are asked how they would maintain cycle time of a four part assembly process with a bound queue. Summary statistics are provided for process and machine breakdown times.

8. Service call centre waiting times: Participants must consider how manage the resource across four shifts at a service call centre. There are options for hiring additional call takers, reallocation of resource and splitting shifts. Summary statistics are provided for the utilisation and performance of each shift.

Transfer of learning is difficult, but possible

The researchers conclude that “The main difficulty for participants was recognising the structural similarity of the transfer and training problems; however, access problems were reduced when participants were provided sufficient time for involvement in both model building and experimentation.” They advise that practical simulation studies aiming to help clients and businesses better manage their processes should take note of these results.

Overall, success in transfer of learning was as expected given the levels of surface similarity, occurring most frequently in the healthcare scenarios (44 percent), followed by call centre scenarios (35 percent), and least frequently in the manufacturing scenarios.

In regard to the perceived similarity of transfer scenarios to the emergency department training, participants generally performed better in the scenarios initially classified as close transfer (1 to 4) than those classified as far transfer (5 to 8). An exception is scenario 8, which was ranked as the second most successful scenario and a substantial improvement over the other far transfer scenarios. An explanation for this is that scenario 8 is more similar to the training task than expected as there are four shifts across the day.

In regard to the three experimental conditions, participants with high involvement in model building and more time for extensive simulation typically transferred learning to one problem more than participants with high involvement in model building but limited time for experimentation and participants with low involvement in model building. Overall, participants with high involvement in model building but limited time for experimentation and participants with low involvement in model building performed similarly. All three groups performed poorly in the far transfer scenarios.

Level of involvement affects overconfidence

Participants with high involvement in model building were found to have made a higher proportion of high confidence errors than participants who reused a pre-existing model. The researchers advise that “Further fieldwork research is needed to replicate and understand these confidence effects. For instance are differences due to model ownership and pride, achieved through involvement in model building, or due to not invented here syndrome, an issue with model reuse.”

Reference:

Also published on Medium.