Potential Problem: Construct Validity

ACs are sometimes criticized for their failure to demonstrate the pattern of correlations among dimension ratings that they are designed to produce. In an AC, each behavioral dimension is assessed in more than one exercise. The rating of each behavioral dimension should be similar across different exercises. For example, communication is measured in the LGD, interview, and role-play exercises and it should be scored more or less the same in all three different exercises. On the other hand, each exercise is designed to assess more than one behavioral dimension, thus the scores for each dimension in one exercise should be quite different. For example, the in-basket exercise is designed to measure decision-making, planning and organizing, ability to delegate, decisiveness, independence, and initiative. The rating for each of these dimensions for one person in the in-basket exercise should not be highly correlated. They are different dimensions and people are expected to scrore differently on them. However, the correlations among dimensions measured in the same exercise typically have been found to be very high (Bycio, Alvares, & Hahn, 1987). Moreover, it turns out that the correlation among the scores on the same dimension across different exercises is very low (Sackett & Dreher, 1982).

These empirical results suggest that it is possible that a single exercise in an AC is actually measuring one single behavior rather than multiple dimensions, and different exercises are measuring different things. Regardless of the above disturbing findings, ACs are found to be highly predictive of job performance across different occupations (e.g. Arthur, Day, McNelly, & Edens, 2003; Dayan, Kasten, & Fox, 2002). Another fact is that candidates tend to view ACs as more face valid (i.e., they seem to measure important things) than cognitive ability tests. As a result, candidates typically are more satisfied with the selection process, the job, and the organization (Macan, Avendon, Paese, & Smith, 1994) after being selected through an AC.