|
TEST SCORING
One of the functions of the Office of Measurement and Evaluation of Teaching (OMET) is the machine scoring of objective tests and scanning of questionnaires for faculty and staff throughout the University. The information below has been prepared to familiarize you with this service and outline the procedures required to enable efficient use of the scoring machine. Machine scoring can relieve faculty and their assistants of a routine, time-consuming task which can be accurately carried out by machine. In addition to this scoring service, analysis and consultation can be provided to assist in the interpretation of test results and future revision of items.
The office uses an Opscan 10 National Computer Systems optical scanner for the machine scoring of objective examinations. Any objective test may be machine scored if it is designed so that the choice of the alternatives can be indicated by marking the corresponding spaces on standard NCS answer sheets. The BLUE answer sheet can be used for a test or questionnaire having five or fewer options. This is the sheet which will be used for most of your classroom testing. The GREEN answer sheet with ten options will primarily be used for questionnaires but may also be used for tests having more than five options. You will notice that both sheets are printed with alphabetic and numeric options, so that you may design your test or questionnaire with either.
GENERAL PROCEDURES
NCS sheets may be obtained from the Office of Measurement and Evaluation of Teaching (G-33 Cathedral of Learning, 412 624-6440) weekdays from 8:30 a.m. to 12 noon and 1 p.m. to 5 p.m. Sheets and the scanning service are available at no charge for classroom testing. Answer sheets should be returned to OMET for scanning, accompanied by a completed REQUEST FOR SCANNING SERVICE form. The answer sheets must be checked before submission to make sure they are all facing the same direction and have been completed in pencil. Sheets not meeting these requirements will not be processed and will be returned to you for hand scoring. Do not staple or fold the answer sheets, as they must be flat in order to be scanned.
Test answer sheets submitted by 5:00 p.m. for scoring will usually be available for pick-up at 10:00 a.m. the next morning. The answer sheets will be returned with the score(s) printed in the margin next to the black timing marks. An item analysis summarizing the individual and class performance on the test will be available.
Processing of questionnaires will generally follow the same turnaround schedule as that for tests. During busy testing periods, you will be given an estimate of turnaround time when you submit your sheets. No computer printout is provided with the scanning of questionnaires. Your data will be stored on a file and will be available for you to transfer to your computer account on VMS. A description of the format of this file can be obtained from our office. If you require assistance with the analysis of questionnaire data, please request it when you submit your sheets.
SPECIFIC INSTRUCTIONS
A. Key
A key for your test must accompany the completed answer sheets and the REQUEST FOR SCANNING SERVICE form. A standard NCS answer sheet is used for your key. Correct answers should be recorded with a number 2 pencil; no other stray marks should be made on the sheet since these may result in inaccurate scoring. Only one correct answer for each item may be marked; credit for additional correct responses must be given by examining the sheets by hand afterwards. If you wish to omit an item from scoring, leave it blank on the key. The key should be clearly labeled KEY and the course and number should be indicated in the margin above the NAME grid at the top of the sheet. Please leave the NAME grid blank for our use.
If you would like your test to be scored in parts a separate key for each part should be prepared and labeled as Key 1, Key 2, etc. You may have up to eight parts on your test. If a part includes items on both the front and back of the sheet, you still need to complete only one key for that part. The front and back of these sheets are scanned simultaneously. Make sure that the marks on your key(s) are dark and completely fill the circles. Erase completely any answers you wish to change. PLEASE CHECK YOUR KEY(S) CAREFULLY FOR ACCURACY BEFORE SUBMISSION! Rescanning of tests is time-consuming and may require longer turnaround time.
B. Student Identification and Completion of Answer Sheets
(A summary of this section appears on a separate sheet which can be read to students.) Items on both answer sheets go down the page rather than across. The BLUE answer sheet has 200 items; the GREEN answer sheet has 120 items. The left sections of both answer sheets are identical and include identifying information, some of which you may want to ignore. The critical parts are the NAME grid and the IDENTIFICATION NUMBER grid.
For tests, each student sheet must have either the NAME grid completed or the IDENTIFICATION NUMBER completed CORRECTLY so that you will be able to identify scores on the item analysis. You may request that students complete both grids so that your listing of scores on item analysis will include both name and identification number. Remind students to check the accuracy of their marks. The name and/or identification number should be printed in pencil in the blocks provided. A space should be left in the NAME grid between the last and first names. The letters or numbers in the corresponding circles in each column are then blackened, including the space circle between the last and the first names. If you do not want the NAME grid completed, the students should still print their names in the spaces above the grid if you plan to return tests to students. ID numbers must begin in column A of the grid and should not have any embedded blanks.
There is one column on the left section of the answer sheet which may be important to you. Some instructors like to post student's identification numbers and tests scores, but should do so only for those students who have given their permission. In addition to the overall class list provided on the item analysis for your records, a list will also be printed of identification numbers and scores for students who have darkened the 1 in the P column of the special codes block. IF you would like to have such a list to post, instruct your students to blacken the circle if they are willing to have their scores posted. Students leaving this column blank will be omitted from the list.
If you are using either answer sheet for survey purposes where no key will be provided, completion of the information at the left half of side 1 is optional; you may allow complete anonymity of responses if you wish by leaving all identifying information blank.
Students must complete the answer sheets in pencil and should be reminded to answer only one option for each item. Items with more than one option blackened will be scored as incorrect. The number of students who choose more than one response to an item is reported on the item analysis printout.
It is important that the marks on the answer sheet be dark and completely fill the circles. IF the marks are too light, the student's score may not be accurate.
C. Multiple Forms of a Test with Scrambled Items
Some instructors develop multiple forms of a test by using identical items but scrambling the order in which they appear on the test. Separate item analyses for each form and/or one combined analysis across the various forms can be requested. You must provide a key for each form of the test, and submit the sheets and key for each form as separate scoring requests. If you would like the combined item analysis you should also complete an equating form we have developed for this purpose, indicating how the item numbers on each additional form are equivalent to the items on the test you have specified as the original. Copies of this form can be requested when you pick up answer sheets for your test.
D. Scanning Request Form
A REQUEST FOR SCANNING SERVICE form is available in our office. This form should be completed and submitted with your test or questionnaire sheets. At that time a receipt containing a request number will be given to you. The receipt must be returned to us when you pick up your corrected tests. If you designate someone else to pick up your tests, please make sure s(he) brings the receipt. This procedure is followed to insure the security of your test.
In addition to department and course identification information, for a test you will be asked to specify:
1) the total number of items on your test (do NOT include items skipped in this total) and;
2) If you have more than one part on your test, the number of items keyed on each part.
You will also be asked to indicate whether on the item analysis you would like the complete list of students' scores to be alphabetically ordered by name or numerically ordered by identification number. If you have asked students to complete both grids, all information will appear but will be listed either alphabetically or numerically, according to your preference.
One other option is available on the item analysis and you will be asked to indicate a preference on the request form. An additional listing of the item numbers which each student answered incorrectly may be requested and will be ordered according to your selection above for total score. This list is helpful when reviewing performance on the test with a student since the answer sheet is not marked to indicate which items were answered incorrectly.
E. Student Data Record
When your test is scanned, a record of each student's responses is saved in our system for one year. It is sometimes necessary for you to know the actual responses that were recorded by the scoring machine. This information can be requested at any time after your test has been scored by referring to your Request Number. If you hand back answer sheets to students, be aware that items originally left blank can be filled in by students at that time, and other answers might be erased and changed. This might account for discrepancies in the score computed by a student and by the NCS machine. Our records would indicate the original response made by the student.
F. Directions For Copying Student Files To Your VMS Computer Account
If you would like to copy the file that contains the student's identifying information, their responses to the items and their total scores from our VMS account to yours, you can use the following directions.
- When you submit your test to be scanning, check the line that requests OMET to make the file available to you on VMS.
-
Any time after 10 am the morning after you have submitted your test for scanning (Monday if you submit on Friday) you can copy your file. Attach to VMS and enter your username and password.
-
If you would like to copy the complete data file (name, social security number, individual item responses and total score for each student), at the $ prompt type the following line and press the ENTER key:
PCOPY USR1:[10143.OME]XXX.DAT/TRIM
- Use your request number (on the blue receipt given to you by OMET) where you see the XXX.
- You will see TO: on the screen. At that point, type in the name that you would like your file to have on your mainframe VMS directory.
- NOTE: There is only one space in the command you are entering, between PCOPY and USR1.
-
If you would like to copy only the social security number, student name and total score for each student instead of all of the individual item data, at the $ prompt type the following line and press the ENTER key:
PCOPY USR1:[10143.OME]XXX.DAT/COL=(13-42, 278-280)
- Use your request number (on the blue receipt given to you by OMET) where you see the XXX.
- You will see TO: on the screen. At that point, type in the name that you would like your file to have on your mainframe VMS directory.
- NOTE: There is only one space in the command you are entering, between PCOPY and USR1. If you have more than one key on your test, call and we will give you additional columns you will need to copy for the subtest scores.
- Regardless of whether you select the command in 2a or in 2b, the first line of the file will be your test key.
G. Copying The Files To Disk
If you would like OMET to copy your test files to a disk, zip disk or CD, please bring it with you when you submit your test to be scanned and check the appropriate line on the gold request form.
H. Item Analysis
A computer printout of item analysis data will be returned to you with the scored tests. This data summarizes the group's performance on the test and provides individual item statistics which can aid in receiving and revising your test. If your test has been scored in parts, individual's scores and a summary of the group's performance on each part as well as the total will be provided by the item analysis. The following statistics will be included:
Total Test Statistics
- Number of Persons
- Number of Items
- Mean Score
- Median Score
- Standard Deviation
- Reliability Estimate (KR-20)
- Possible Low Score
- Possible High Score
- Obtained Low Score
- Obtained High Score
- Number of Items, Mean, Standard Deviation, KR-20 for Each Part
- List of student Names and/or Identification Numbers and Score(s)
- Frequency Distribution of Total Scores
- Percentile Ranks
- Histogram
Individual Item Data
- Frequency of Responses to Each Option
- Number of Omits
- Number of Multiple Responses
- Difficulty Index
- Discrimination Index Distribution
- Frequency of Keyed Responses
- List of Items Answered Incorrectly by Student Name or Identification Number (optional)
A detailed description of this item analysis is available from the OMET. Staff members are available to discuss the item analysis. Suggestions for the development of test items can be found in Chapter 8 of the TA Handbook developed by Center for Instructional Development & Distance Education.
|
