Method for evaluating and improving of the expert estimates consistency in group decision-making
DOI:
https://doi.org/10.20535/SRIT.2308-8893.2018.3.10Keywords:
group decision making, expert estimation, index of consistency of expert estimates, spectral approach, consistency threshold, expert feedbackAbstract
The issue of the assessment of an expert estimates consistency level for group decision making is considered. An analysis of bibliographic sources testifies that despite the availability of multiple practical tools and solutions for assessing the consistency index, all such methods have considerable shortcomings affecting significantly the accurateness of research results. The goal of this research paper is the development of a method for assessing the consistency between expert estimates, with the elimination of a number of key shortcomings of existing methods. It is suggested to calculate the consistency index using a spectral approach, under which expert estimates are presented as a spectrum on a bounded, continuous or discrete scale. The index is calculated as the normalized value of the sum of distances between expert estimates for all possible estimate pairs. This formula for calculating the consistency index has been tested for the function of squared pairwise differences in estimate pairs. The conducted analysis shows that the function of distance fits better for a detailed practical analysis of consistency between expert estimates. Imitation modeling has been carried out, and an evaluation has been offered for a threshold consistency value above which aggregation of expert estimates becomes possible. In order to make it possible to raise the level of consistency, a procedure for gathering feedback from an expert is proposed, provided that no pressure is put on him or her.References
Totsenko V.G. The Agreement Degree of Estimations Set with Regard of Experts Competency / V.G. Totsenko // Proceedings of the Fourth International Symposium on the Analytic Hierarchy Process. Simon Fraser University. — Vancouver, Canada. — 1996. — P. 229–241.
Tsyhanok V.V. Elementy kombinatornoho pidkhodu pry vyznachenni spektral'noho koefitsiyenta uzhodzhenosti ekspertnykh parnykh porivnjan' / V.V. Tsyhanok // Reyestratsija, zberihannja i obrobka danykh. — 2012. —T. 14, № 2. — S. 98–105.
Olenko A. Double Entropy Inter-Rater Agreement Indices, Applied Psychological Measurement / A. Olenko, V. Tsyganok. — 2016. — Vol. 40(1). — P. 37–55.
Schmidt F.L. Interrater Reliability Coefficients Cannot be Computed When Only One Stimulus is Rated / F.L. Schmidt, J.E. Hunter // Journal of Applied Psychology. — 1989. — 74. — P. 368–370.
Burke M.J. On Average Deviation Indices for Estimating Interrater Agreement / M.J. Burke, L.M. Finkelstein, M.S. Dusig // Organizational Research Methods. — 1999. — 2. —P. 49–68.
James L.R. Estimating Within-Group Interrater Reliability With and Without Response Bias / L.R. James, R.G. Demaree and G. Wolf // Journal of Applied Psychology. — 1984. — 69. — P. 85–98.
Lindell M.K. A Revised Index of Agreement for Multi-Item Ratings of a Single Target / M.K. Lindell, C.J. Brandt, D.J. Whitney // Applied Psychological Measurement. — 1999. — 23. — P. 127–135
Brown R.D. Interrater Agreement Reconsidered: An Alternative to the rwg Indices / R.D. Brown, N.M.A. Hauenstein // Organizational Research Methods. — 2005. — 8. — P. 165–184.
Zgurovsky M.Z. Group Incomplete Paired Comparisons with Account of Expert Competence / M.Z. Zgurovsky, V.G. Totsenko, V.V. Tsyganok // Mathematical and Computer Modelling. — 2004. — 39(4–5). — P. 349–361.
Cicchetti D.V. (1997). A New Method for Assessing Interexaminer Agreement when Multiple Ratings are Made on a Single Subject: Applications to the Assessment of Neuropsychiatric Symtomatology / D.V. Cicchetti, D. Showalter, R. Rosenheck // Psychiatry research. — 1997. — 72(1). — P. 51–63.
Constrained Optimization. — Available at: http://mat.gsia.cmu.edu/classes/QUANT/NOTES/chap4/node6.html.
