William S. Verplanck
University of Tennessee, Knoxville
An examination format assessing the intraverbal repertoire of individuals in psychology is described and results using it reported. The Associate Test is easy to prepare, to take, and to grade. Its reliability measures are satisfactory; its ability to predict later behavior is reported upon. The Associate Test is computer friendly, and its methods can be applied for examination in any field, and at any level.
The writer would like to acknowledge gratefully the work of J.R. Caplan, R. Fleischer, A. Gordon, W.N. Jennings, R.N. Krueger, R. A. Ramos, W. Saeger, and J. Shelnutt. P.J. Lazzara and R.D. Kinser assisted in summarizing this research and in assembling the appendices.
The Word Associate Test (WAT) (or Short-Short Answer Test) is a format for preparing brief tests designed to demonstrate the student’s command of the critical terms of specific areas of study. It provides a measure of the student’s vocabulary in specialized and technical areas. That is, it samples the minimal intraverbal repertoires of the individual
The Word Associate Test has been developed over some 30 years of research. [The progressive development of the Associate Test and its methodology into its present form have been reported at meetings of The Psychonomic Society (6), of the Southeast Psychol. Ass’n. (6), and of the American Education Research Ass’n. More detailed reports have appeared in 1 Honors Paper, 2 MA Theses, and 2 Doctoral Dissertations. A listing of these papers may be requested of the author.] This work was instigated by dissatisfaction with the multiple-choice test, whose scores are patently associated with specific content-independent verbal skills and with well shaped monents for guessing.
The primary objective of the WAT is the rapid examination and grading of large numbers of students in one or another of a broad variety of areas (Jennings, W.N., 1968; Saeger, W., 1975), and at a number of levels of achievement. It was not developed to find how people think, how original [Jennings (1966) has nevertheless shown that there is a relationship between word-associate measures and evaluation of the “originality” shown in compositions written for the instructors in The English Department at UTK. These instructors had no information whatever on the WAT, much less of each student’s performance.] they are, how well they can explain something, or write, much less to predict academic success, life achievement, or aptitude for something or other. The command of specified vocabularies is, however, necessary for all these performances.
The WAT asks only one question, which has many paraphrases in ordinary language: “How successfully has he/she studied?”, “What words can the teacher or writer use and expect the students to be able to deal with at an acceptable level?”, and loosely, “How much does the student know?”. It does so without the use of the prompts that appear in multiple-choice, true-false, matching, and fill-in-the-blanks examination items. Incorrect prompts may indeed be added to the examinee’s intraverbal repertoire.
The WAT may be termed “robust”: Instructions to guess do not increase scores; instructions “Do NOT guess,” however, decrease them.
The WAT demonstrates the examinees minimum level of command of a selected vocabulary. The WAT does not enable students to demonstrate their full knowledge. For this, a further WAT, presenting a more advanced technical vocabulary should be taken. In essence, a WAT can be prepared for any level of study, from kindergarten through the doctorate, in any field (with the possible exception of some branches of mathematics).
Although the test was developed before computer technology had reached its present level of sophistication and accessibility, it was preadapted for computer administration, grading, and presentation of results. Computer technology will enable a student to take an Associate Test on any subject at any level of achievement at any time at any terminal with immediate scoring against national norms.
It may now be applied to evaluate the achievement of individual students or specific classes and the effectiveness of various teaching methods, or of instructors. (Jennings, 1968; Saeger, 1975; Shelnutt, J., 1969) The description which follows gives the present status of the Word Associate Test. The appendices present sets of data relevant to the use of the test and to how it works in practice. An extended monograph on the WAT is now in preparation, and is to be published by the Cambridge Center for Behavioral Studies. This will report on the research on the WAT carried out to date, and will indicate its potential as a research instrument as well as a method for examination of substantive command of subject matter. Users will evaluate its face validity on the basis of their experience with its use and with the students who have taken it.
Given the current state of understanding of relevant criteria and of “psychometric validity,” the psychometric validity of the WAT is no better or worse than that of any other form of examination, whether essay, take-home, true-false, or multiple choice. It can and will be compared with others at such time as acceptable sets of well-defined criteria can be agreed upon.
Although the WAT has been independently developed, the readers of Rosanoff (1918), Deese and Loftus will find it of both practical and theoretical importance. The reader will also recognize that “Word Associate Test” is perhaps too restricted a name. Besides words, names, and dates, one may also use equations, curves, chemical formulas, diagrams and other visual material as stimulus items in preparing examinations using this design, and examinees may respond by drawing pictures and sketching graphs or diagrams, as some have done.
When the response given to the stimulus word or term is verbal, rather than a diagram, sketch, or graph, the associate may be considered a primary or first-order intraverbal, one that is given directly to the stimulus. These are the responses most likely to appear in sentences written when the examinee is asked to write brief essays on the term. In these answers, the first term to appear is most often the stimulus term itself–in this context, a notate, or tact.
The possibility of the use of stimuli other than written or spoken words, such as pictures, diagrams, sounds, and so on, is one reason why the term “associate,” rather than “intraverbal behavior” (Skinner, B.F., 1957), is used. Certainly the specific methodology presented deals with intraverbal behavior. But the methodology lends itself equally well to the investigation of tacts, as distinguished from “intraverbal responses.” Word associates are intraverbal behaviors only when the stimuli themselves are words or phrases; otherwise, they are tacts. In Skinner’s Verbal Behavior, the terms “tact” and “intraverbal response” have extensive connotations, which may or may not be relevant to the verbal behaviors reported on here. A second reason for omitting these terms is that psychologists unfamiliar with or ignorant of the vocabulary of behavior analysis may not be able to take advantage of this methodology for examining what they would not refer to as either “verbal repertory” or “intraverbal behavior,” even though this association procedure assesses the intraverbal repertoire. A vocabulary serves better as a bridge between divergent systematic viewpoints than as a barrier between them.
Finally, and by no means least important, students seem to like it (Attkisson & Snyder, 1975).
THE WORD ASSOCIATE TEST: PREPARATION, ADMINISTRATION, GRADING
The Word Associate Test has been developed to fill the need for a test that can be easily and “objectively” graded, that can sample broadly the content of a course, that can be readily prepared and that avoids the multiple choice format. The format of the test is straightforward: Stimulus terms–key words–are chosen from the material on which a class is to be tested (e.g., terms in the lecturer’s notes, in the body, index or glossary of a text). Examinees are given the opportunity to demonstrate their familiarity and grasp of each term by responding to it appropriately. The stimulus terms selected are arrayed vertically; the number of responses required is chosen by the examiner. Instructions are brief; grading is similarly uncomplicated. Each response is graded on a scale chosen by the instructor. The distribution of summed scores is the basis on which letter grades can be assigned. This presentation will provide the readers with the information they will need if they wish to use this examination format. A monograph summarizing the findings of research on this method of examination is in preparation. Appendices present several WAT’s, completed and graded, as well as other findings.
Introducing the WAT to Students
Before giving a WAT (“The Short-short Answer Test”) as a grade determining tool, the instructor should give the students one or more brief practice sessions. These serve as a quick review for the students, showing them what they need to study more, as well as giving them an opportunity for them to know how the method works on a test.
Construction
Instructions
The instructions below have proven easy to follow. A briefer form may be used at the top of each page of an examination. (See Appendices)
In each space provided for each item write a word, date, phrase, term, name, or concept relevantly associated to it in the subject matter of this course. Do not repeat. You may review your answers before passing your paper in–you’ll probably be able to fill in some of the spaces you left blank.
References to grading need not occur in the instructions. Dry-runs, advisable before the first administration, serve to desensitize students to the novelty of the exam (new kinds of tests are always scary), as well as to eliminate the possibility that a student has misconstrued the instructions.
Choice of Stimulus Terms
The number of stimulus terms in a given test will be determined by how broad a sample the instructor wishes to examine on, how many responses he or she wants, and on the time allotted for the test. When two responses are required, students take approximately one minute per item to take the test.
The instructor should choose the terms that he or she thinks the students should know something about from lectures, discussions, or other assigned readings. Stimulus terms can and should sample a wide range of the materials covered in the class. Tables of contents, indexes of textbooks, and glossaries are especially convenient sources. Boneau’s paper (1990) is a useful source.
Overly general or vague terms should be avoided; anybody can give acceptable associates to, say, “motivation,” or “motherhood.” Also avoid names easily identified generically, e.g., “Schultz,” which will elicit “German psychologist.” Related terms, i.e., words that would be acceptable as responses to one another, may both be included, but should be presented distantly from one another on the examination sheet.
The wide range of sources for stimulus items allows the test to be finely tuned to the content and level of the course. The test itself can be a very specific probe of student knowledge in given subareas. Eventually each instructor will be able to assemble a ‘master list’ of terms that can be given to students as study guides–the concepts of which they should be in command.
Appendix A lists, in alphabetic order, approximately 200 items from a set of over 2,000 psychological terms that have been used in first courses in General Psychology and Introduction to Biological Psychology at U.T. Knoxville.
Format
Terms are arranged vertically, isolated from one another by lines. This format discourages sentence writing, and increases the likelihood that the second, third, and fourth associates are given to the stimulus term, rather than to the immediately previous written response.
The student is usually required to produce four associates for each stimulus term, although this number can vary as circumstances require. Research indicates that four associates are ample to provide discriminatory power for evaluating student knowledge. Fact is, two associates prove to be enough to demonstrate basic knowledge. A test including 64 two-response items can usually be completed by students within an hour.
Appendix B presents a number of test formats, some completed and graded.
Grading
Ideally, grading should begin with a clerical procedure tabulating together the responses from all students with frequency counts. Responses can be listed in order of frequency of occurrence and then graded. Appendices D and E present such distributions. Note that unacceptable responses are infrequent. Scores may then be transposed to the responses on each of the original test papers. This procedure guarantees a high reliability in grading.
Lacking the computer technology, hand grading, as seen in Appendix D, must be relied upon. It goes remarkably easily, and does not take as much time overall as putting together an equivalent multiple choice exam.
Three methods have been used in grading, and inter-grader reliabilities measured for each.
A. Four Point Scale for Grading
+2 – A good associate, relevant, clearly demonstrating familiarity with the material.
+1 – Acceptable, but strained, ambiguous, overly general, or a reasonable guess.
0 – No response, irrelevant, indeterminate or out of context.
-1 – Positively incorrect.
As an example of the four point system, scores would be assigned to the responses below as follows:
Binet
Intelligence test +2 French +1 Stephen Vincent 0 Freud's student -1
B. Three Step Scale
The preceding system was simplified to a +2, +1, or 0 scale. The -1 score was eliminated since it seemed to measure the grader’s response to the inadequate response. Further, relative to the number of “correct” or “acceptable” responses, few clearly incorrect rather than irrelevant responses are given, so few associates appear that get graded -1. Dropping the penalty for “wrong” answers proves not to affect the grade curve.
On the 2,1,0 scale, do not limit the use of 2 to indicate the “exceptionally good” response (corresponding to a -1, as somehow especially bad). A score of “2” should mean that “no one could take exception” to the associate, while a “1” indicates a response that is in the “right direction,” but not adequate to “pinpoint” the stimulus term. With experience in grading, the grader will develop both speed and confidence in his or her objectivity and consistency in grading, especially if the paper graded is identified by student number, rather than by name.
C. Simplest Scale: Two Steps
The third scale is the simplest; responses are graded 1 and 0, acceptable or not.
Saeger (1975) states, “Review of the problem of item weighting in general and the magnitude of the correlations between weighted versus unweighted scoring of responses on the WAT in particular indicates that a simplified scoring system of +1 for an acceptable associate (encompassing +2 and +1) and 0 for an unacceptable associate (encompassing 0 and -1) will provide sufficient estimates of test characteristics.”
D. Count Score
Those who have had considerable experience reviewing WAT scores are not surprised to find that simple counts of responses, without grading, correlate highly with scores.
PSYCHOMETRIC PROPERTIES
Reliability
Variability distributions of scores made in a class will show a broad range of performances–from those who fill in almost every blank correctly, to those whose answer sheets are less than half full.
Research on the Tennessee WAT indicates that the procedure is acceptably reliable. Various coefficients of reliability range from +.592 to +.899. See appendix F.
Validity
The measurement of the validity of any examination of a student’s achievement in a course is a matter of faith. The question of the criterion or criteria against which the validity of any one test is to be evaluated is very much open to debate, the question arising just as critically with essay, short answer, take-home, multiple choice exams as well as with the WAT. Appendix F presents validity coefficients of WAT scores with the essay, short answer, and GPA. “Validity” estimates range from .447 (WAT and overall GPA) to +.773 (WAT and Short Answer Test) depending on the criterion. Most coefficients are between +.45 and +.77 and are thus well within the acceptable range.
Those who developed the test have done so in many smallish (10-20) classes in both beginning and advanced Honors Psychology groups, in which it is possible to get to know the students and their performances reasonably well. One is very seldom surprised by scores on WAT, and then it is by the score of the shy, quiet student who performs very well indeed on the Associate Test.
References
Attkisson, C. C., & Snyder, C. R. (1975) Student evaluation of mulitple choice and word association exams. Journal of Instructional Psychology, 2, 9-15.
Boneau, C. A. (1990). Psychological literacy: A first approximation. American Psychologist, 45, 891-900.
Jennings, W. N. (1966). Originality as a function of uncommon associations to common stimulus words. Unpublished master’s thesis. University of Tennessee.
Jennings, W. N. (1968). The assessment of instructor differences using the Word-association examination. Unpublished doctoral dissertation. University of Tennessee.
Rosanoff, A. J., Martin, H. E. and Rosanoff, I. R. (1918). A higher scale of mental measurement and its application to cases of insanity. Psychological Monographs. Princeton, N.J.
Shelnutt, J. B. (1969). Some characteristics of the Word-Association Test used as a measure of academic knowledge with respect to individual differences in student’s patterns of responding, difficulty of stimulus items, and heterogenity of the response distributions. Unpublished honors thesis. University of Tennessee.
Saeger, W. (1975) An associative evaluation of selected interdisciplinary courses. Unpublished doctoral dissertation. University of Tennessee.
Skinner, B. F. (1957) Verbal behavior. New York: Appleton-Century-Crofts.
Appendix A
ablation aversive stimulus cell differentiation consciousness abscissa axon hillock central limit theorem consistency accountants Babinski central tendency consolidation theory Ach balance theory centrality index consonance dissonance acquired drive balloon cephalocaudal constancy acquisition Bartlett cerebrotonic constant method activation syndrome basic anxiety cerebrum constitutional types active vocabulary Beach chained schedule construct validity actuarial behavior therapy chance errors consummatory act adaptation behaviorism character consummatory behavior adaptation level Bekesy Charcot content analysis adipsia Bekhterev checking force content psychology adrenal Bell Chomsky context theory adrenalin Berger choroid contiguity affectional drive Berkeley chromosome contraction affective congruency beta weights chronograph control all or none Binet classical convergent hierarchy alpha biosocial theory classical conditioning core and context alpha rhythm bipolar cell clinical interaction corporal data Ames bit closure correlation amnesia black headed gull clustering correlation asch ampulla blast injection cognitive change correspondence amygdala blockings cognitive dissonance cortex analyzer Boring cognitive maps cortical steroid animal consciousness Bradley cognitive theory corticospinal anticathexis brain cohesiveness counter conditioning anticipation error brain washing collective unconscious counterbalancing anxiety Brentano color blindness counterconformity anxiety reduction Breuer color solid cranial nerves aphagia brightness color vision Crespi-effect aphalic index brightness constancy common elements cretinism approach gradient Broadbent's model common trait criterion apraxia Broca comparative judgement criterion analysis Aquinas Broca's area compromise response critical period area sampling Brunswik computers cross validation arousal Bunsen-Roscoe law concept formation csr association test C. Lloyd Morgan conditional achievement cultural conflict asymptote canalization conditioned elements cumulative record atomism Cannon conditioned inhibition cumulative recorder atropine case history conditioned response curare attensity castes cones curiosity attention castration complex conflict dark adaptation attenuation catatonic conformity dat authoritarian catharsis confounding ddd syndrome autism cathexis congruity incongruity De La Mettrie autokinetic Cattell conjunction decerebrate rigidity automated teaching cell assembly connector neuron decibel autonomic cell body connotative meaning decision theory
Appendix B-1
Completed Test
Psychology 251-B
Honors General Psychology
COURSE EVALUATION EXAM
Short-Answer Score 115
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
||||||||||||
|
|
|
|
|
Appendix D-1
Response distribution to the Stimulus Term ‘Cone’.
N students = 166; N possible responses = 664
Associates with F less than 5 ommitted
Associate: F RF Associate: F RF Primary Intraverbal Primary Intraverbal Response Response rods 279 .42 3 types 11 .02 eye 254 .38 vis. acuity 9 .01 fovea 247 .37 green 9 .01 color 175 .26 night 9 .01 retina 156 .24 night vision 9 .01 vision 87 .13 in fovea 9 .01 color vision 75 .11 Young-Helmholtz 9 .01 light 42 .06 part of eye 8 .01 daylight 36 .05 yellow 8 .01 day vision 27 .04 daytime 8 .01 sight 26 .04 blind spot 8 .01 iodopsin 26 .04 optic nerve 8 .01 lt. sensitive 18 .03 cells 7 .01 3 colors 17 .03 in eyes 7 .01 receptors 16 .02 dark 7 .01 conc. in fovea 15 .02 rhodopsin 7 .01 three 14 .02 ice-cream 7 .01 day lt. vision 13 .02 Hering theory 7 .01 color blindness 13 .02 red 6 .01 yellow, green, blue 12 .02 sensitive 5 .01 blue 11 .02 blanks & lower 257 .39 perception 11 .02 frequency responses
Appendix D-2
How concepts change
Group A
Response distributions (four associates) to items appearing on final examination of one class in General Psychology, and on “orientation” pretest of another. Number of blanks and low-frequency responses omitted. N = number of students.
1. Stimulus word: CONTROL
Pretest (N=87) Final (N=115)
Response F RF Response F RF Power (over) 31 .36 Group 58 .50 Experiment 21 .24 Experimental 54 .47 Sociological Factors 15 .17 Experi(ments, tation) 35 .30 Variables 13 .15 Independent Variable 20 .17 Self 13 .15 Variables 16 .14 Emotions 12 .14 No Special Treatment 15 .13 Stimulus (&/or) Response 12 .14 Generic 11 .09 Mind 11 .12 Baseline (basis) 9 .08 Behavior 8 .09 Validity 7 .06 Groups 7 .08 Necessary 6 .05 Reward 4 .05 Test 4 .05 Biological 4 .05
2. Stimulus word: CENTRAL TENDENCY
Pretest (N=87) Final (N=115)
Response F RF Response F RF Habits, Drives, etc. 15 .17 Mode 96 .83 X, Median, Mode 12 .13 Median 96 .83 Stimulus (&/or) Response 9 .10 Mean 84 .73 Cause 7 .08 Average 14 .12 Most Likely 7 .08 Standard Deviation (S.D.) 11 .10 Physiology Frequency Distribution 9 .08 (brain, nervous system) 6 .07 Graph 6 .05 Inward Feelings 6 .07 Emotional (love, hate, anger) 5 .06 Standard Deviation 4 .05 Average 4 .05 Behavior 4 .05
Group B
Response distribution (four associates) to items appearing on the pretest and final examination of the same class.
1. Stimulus Word: HABITUATION
Pretest (N=68) Final (N=60)
Response F RF Response F RF
Environment 10 .15 Conditioning or Learning 33 .55
Habit 5 .07 Clothes 13 .21
Continue 3 .04 Adaptation 11 .18
Stimulus 2 .03 Perception 9 .15
Desires 2 .03 Getting used to 7 .12
Need 2 .03 Accustomed 7 .12
Home 2 .03 Unaware(ness) 6 .10
Life 2 .03 Behavior 5 .08
Condition 2 .03 Adjustment 5 .08
Response 2 .03 Response 5 .08
Exhaustion 2 .03 Environment 4 .07
Reflex 2 .03 Motivation 4 .07
Habit Formation 4 .07
Repeated Stimuli 4 .07
Experience 3 .05
Babinski 3 .05
2. Stimulus Word: HOMEOSTASIS
Pretest (N=68) Final (N=60)
Response F RF Response F RF
Stable 3 .04 Balance 34 .57
Internal Environment 3 .04 Autonomic 17 .28
Map 2 .03 Optimal 12 .20
Dynamic 2 .03 Equilibrium 11 .18
Tension 9 .15
Behavior 6 .10
Temperature 6 .10
Internal 4 .07
3. Stimulus Term: FUNCTIONAL AUTONOMY
Pretest (N=68) Final (N=60)
Response F RF Response F RF Movement 4 .06 Nervous System 17 .28 Muscles 2 .03 Motivation 8 .13 Self 2 .03 Miser 7 .11 Involuntary 2 .03 Homeostasis 3 .05 Nerves 2 .03
4. Stimulus Term: ROD-CONE BREAK
Pretest (N=68) Final (N=60)
Response F RF Response F RF
Eye 15 .22 Dark Adaptation 23 .38
Color 5 .07 Rods-cones 15 .25
Sight 2 .03 Rhodopsin 14 .23
Nerve 2 .03 Day-night vision 13 .22
Back of Eyeball 2 .03 Twilight vision 12 .20
Purkinje (shift, effect
or phenonomenon) 11 .18
Eyes 7 .12
Red (lenses, vision,
goggles) 7 .12
Vision(ual) 6 .10
Threshold 6 .10
Appendix D-3
Psychology-Zoology 472. Winter Quarter 1963-1964
Item Difficulty List – Final Exam
Maximum points per item (8 X 20) = 160
Four responses; graded 2, 1, 0, -1
Total Total
Score Score
on on
item RF Stimulus word item RF Stimulus word
157 .98 peck-order 101 .63 consummatory act
149 .93 threat 100 .63 habituation
148 .93 stickleback 99 .62 mobbing
146 .91 queen substance 97 .61 population homeostasis
145 .91 jackdaw 96 .60 spacing
144 .90 paleontology 95 .59 cortical steroid
144 .90 gonadotropins 93 .58 trophallaxis
143 .89 innate 92 .58 sensitization
141 .88 pheromones 88 .55 "local dialects"
140 .88 castes 87 .54 Tribolium confusum
139 .87 estrogen 87 .54 intra-specific fighting
139 .87 waggle dance after mutual recognition
138 .86 population pressure 84 .53 discriminative stimulus
138 .86 freezing 81 .51 interspecific signals
138 .86 instinct 77 .48 general adaptation syndrome
138 .86 dominance-subordination 76 .48 vasopression
137 .86 immobilization 76 .48 ecological niche
135 .84 comparative method 73 .46 MacDougall
134 .84 sage grouse 67 .42 ecosystem
134 .84 imprinting 67 .42 king
134 .84 hormones of anterior pituitary 66 .41 vacuum activity
132 .83 classical conditioning 59 .37 cockatoo
131 .82 hypothalamus 58 .36 personal distance
129 .81 linear organization 54 .34 individual distance
127 .79 reciprocal preening 53 .33 social reflex no. 1
126 .79 baboon harem 52 .33 internal releasing mechanisms
125 .78 permanent mateship 51 .32 paradigm
120 .75 Mesozoic 50 .31 genetic control
119 .74 supplementary reproductives 48 .30 activation syndrome
118 .74 rate of response 37 .23 epideictic
118 .74 hypertrophy of sex glands 35 .22 ecdyson
117 .73 master gland 32 .20 genetic drift
115 .72 scouts 28 .18 setting operations
114 .71 interspecific recognition 28 .18 corpora alata
of calls 28 .18 subterritory
112 .70 phylogeny 25 .16 French corvids
111 .69 LH 24 .15 repletes
105 .66 operant 18 .16 cycling in population
104 .65 home range 10 .06 statary phase
104 .65 mood 6 .04 lek
104 .65 response repetoire 2 .01 local enhancement
101 .63 S-R; S 0 .00 environmental continum
Appendix D-4
The recoverability of
Stimulus terms
The left-hand column gives the response-distributions from which the “key words” were chosen. The right-hand column gives the response-distributions obtained when the subjects were given one blank to fill in, with an instruction; find the “key word” that links together the four words presented.
For Group I, the key-words were selected from the response-distributions given by the same class on a previous exam.
For Group II, the key-words were selected from the response-distributions given by a class other than the one seeking the key-word.
I (a) 212-2a N=74 212-2a N=74
4 Associates to Stimulus Word: "Key Word" to "G.&S., bumps,
Phrenology localization, pseudo science."
Response N RF Response N RF
Gall and/or Spurz. 59 .80 phrenology 62 .84
localization 48 .65 repetition of S 2 .03
pseudo science 48 .65 others 10 .13
bumps 37 .50
size 20 .27
brain 16 .22
trait char. 13 .18
I (b) 212-2a N=74 212-2a N=74
4 Associates to Stimulus Word: "Key Word" to "ablation, removal,
Extirpation localization, lesions."
Response N RF Response N RF
ablation 44 .60 extirpation 19 .36
removal 38 .51 brain damage 11 .31
study-investigate 26 .35 brain 10 .13
lesions 23 .31 lobotomy 7 .09
localization 21 .27 brain functions 6 .08
brain 13 .18 repetition 2 .03
drugs 6 .08 others 19 .36
Names-(Lashley, Magoun, etc.) 6 .08
II (a) 212-1 N=33 212-2b N=81
4 Associates to Stimulus Word: "Key Word" to "spatial, temporal,
Summation threshold, firing of neuron."
Response N RF Response N RF
spatial 18 .55 summation 35 .43
temporal 18 .55 stimulus 10 .20
threshold 15 .45 others 40 .37
compound stimuli 8 .24
firing of neurons 5 .15
synapse 5 .15
addition of output 3 .09
intensity 3 .09
neuron 3 .09
stimulation 2 .06
II (b) 212-1 N=33 212-2b N=81
4 Associates to Stimulus Term: "Key Word" to "neuron, nodes of
Myelinated Fibers Ranvier, insulation, faster trans."
Response N RF Response N RF
neuron 26 .70 myelin(ated) 57 .70
faster transmission 20 .61 (covering, sheath, tissue,
axon covering, shield 20 .61 fibers, insulation) others 21 .30
sheath 19 .58
insulation 6 .18
nodes of Ranvier 4 .12
around axon 2 .06
not always present 2 .06
Appendix E-1
Frequency Relative to Total Possible
Number of Associates to Each of Four Related Terms
Stimulus Term Harlow Cloth Wire Affectional
mother mother drive
(n = 153) (n = 149) (n = 181) (n = 179)
Harlow 1.00 .28 .19 .03
cloth mother .28 - .33 .08
wire mother .22 .30 - .07
experiment .06 .06 .07 .00
Sci. American .06 .01 .00 .00
behavior .05 .00 .00 .01
imprinting .04 .08 .08 .07
instinct .04 .03 .05 .09
rats .04 .00 .01 .02
motivation .03 .00 .03 .20
monkey baby .03 .03 .04 .00
wire .03 .06 .00 .00
cloth .03 .00 .04 .00
maternal .02 .03 .00 .14
surrogate .02 .03 .06 .00
maternal instinct .02 .02 .03 .03
feeding .01 .03 .06 .00
Hebb .01 .03 .02 .01
love .00 .03 .02 .28
sex .00 .00 .00 .12
security .00 .09 .02 .02
emotion .00 .00 .03 .08
innate .00 .00 .00 .10
fear .00 .01 .06 .02
affection .00 .04 .04 .00
hypothalamus .00 .00 .00 .07
protection .00 .03 .03 .00
Each stimulus term was given to approximately 1/4 of the same
class on a final examination. Four responses were possible.
Appendix E-2
The probabilistic relationship of
primary intraverbals in a group
133 students took this WAT; all stimulus terms were presented four (4) times. The terms in squares were presented as stimuli in the WAT. The encircled terms appeared as responses. Arrows connecting S and R give relative frequency of occurrences of each response relative to the maximum possible number (532). The careful reader will note that one response that occurred relatively frequently is not appropriate (“wrong”). Investigation showed that this word appeared in a paragraph of the text book that included the stimulus term and two or three “correct” associates. (Draw your own conclusions.)
Appendix F-1
Some reliability and validity measures of the WAT and other tests
Introductory Psychology, Honors Section (not an exhaustive set)
- Reliability
- Split half
- First half-second half: r = +.724
- Odd-Even: r = +.686
- Ranked difficulty of terms: r = +.683
- Kuder-Richardson forumula 20
- Original test (36 items): r = +.724
- Original test (1st 20 items only): r = +.592
- Retest (20 items): r = +.736
- Test-retest (20 items)
- Students’ scores: r = +.624
- item difficulty: r = +.899
- Split half
- Validity
- Rank-order correlation between test part scores
- Essay and short answer: +.804
- Association and short answer: +.733
- Association and essay: +.602
- Correlation between GPA and scores of:
- Association part: r = +.447
- Entire test: r = +.382
- Short answer part: r = +.265
- Essay part: r = +.254
- Correlation between position on Dean’s List and rank of score on:
- Association part: r = +.678
- Entire test: r = +.677
- Short answer part: r = +.609
- Essay part: r = +.409
- Rank-order correlation between test part scores
Appendix F-2
Some correlations on
five examinations
N = No. students n = No. st terms 4 resp; 2, 1, 0, -1 grading
1. General Psych., weekly quiz 2. General Psych., Honors Final Exam
N = 133 n = 25 N = 25 n = 36
WAT: Mean score 137.0 SD = 23.3 M = 180.2 SD = 28.2
% of max score possible = 68.5 % of max score = 62.6
GPA: Mean 2.30 SD = 0.54 M = 3.44 SD = 0.28
int. consist (KR) r = 0.816 (KR) r = 0.724
WAT with GPA r = 0.356 WAT with GPA rho = 0.447
WAT with M.C. r = 0.459 WAT with short ans. rho = 0.773
M.C. with GPA r = 0.359 WAT with essay rho = 0.602
Short ans. with GPA rho = 0.265
Essay with GPA rho = 0.254
3. Comparative Psych., Upper Div. 4. History and Systems, Upper Div.
N = 20 n = 81 N = 28 n = 49
WAT: Mean score 388.1 SD = 69.3 M = 218.39 SD = 78.1
% of max score possible = 59.9 % of max score = 65.8
GPA: Mean score 2.52 SD = 0.44 M = 2.73 SD = 0.87
WAT with GPA rho = 0.647 WAT with GPA rho = 0.455
Appendix G-1 Research on Honors Comprehensive Examination.
Research on Undergraduate Honors Comprehensive examinations given at the end of the first year of the Honors program. If a student does not pass at that time he or she is examined the following year. The Comprehensive Examination has ten subsections, of which History and Systems is one. Different samples of terms are prepared for each administration. The Comprehensive Exam was taken by Advanced Honors Students at the end of the Junior Year, which had been devoted to intensive readings, with weekly seminars.
The same examination was given experimentally to graduate students in Psychology at the Master’s level to replace a departmentally prepared ten-section multiple-choice examination. It was replaced by the appropriate Graduate Record Examination, which placed no demands on faculty time.
Listen on Spotify
