The Chemistry Concept Inventory (ChCI) is a multiple-choice test designed to assess the effect of curriculum changes. The goal is to make a reliable, easy-to-use instrument that is administered in a short period of time and that can accurately assess student understanding of general chemistry topics. If the ChCI can easily and accurately assess student understanding, then teaching techniques can be evaluated for effectiveness by comparing the group that used the technique to a control group.

Topics covered by the ChCI were selected by a group of chemistry educators in consultation with engineering faculty members. These topics are introduced in general chemistry and reappear in later engineering courses. Topics selected from both semesters of a two-semester chemistry sequence were

Table 1. Topics covered.

Test	Topic	Subtopics
Chemistry I	Thermochemistry	Heat Concept Thermal Conductivity Thermal Equilibrium
	Bonding	Bond Polarity Octet Rule
	Intermolecular Forces	Intermolecular Forces
Chemistry II	Equilibrium	Equilibrium Rate Dynamic vs. Static Le Chatelier’s Principle Equilibrium Constant
	Acids and Bases	Acid/Base Neutralization Acid Strength PH
	Electrochemistry	Oxidation/Reduction Voltaic Cells Electrolytic Cells

Once the topic areas were determined, an extensive literature search was carried out to identify misconceptions associated with the ChCI topics that have already been studied. There is a large body of literature available on chemistry misconceptions. Because of this literature, distracters could be designed to test for known misconceptions. This search also led to natural subtopics within each of the main topics. For each subtopic, at least three questions were written, giving a total of 30 questions for the Chemistry I inventory and 31 questions for the Chemistry II inventory. The questions were intended to be conceptual, not mathematical or algorithmic. These questions were initially given to the students in Chemistry I and II as part of their weekly quizzes. The questions were then compiled into Version A of the ChCI. Because the questions were given after the students had covered the information in lecture, only post-test data were available. Table 2 summarizes these results.

Table 2. Results from ChCI version A.

Chemistry I	Chemistry II
N = 326 Students	N = 158 Students
Alpha = 0.7883	Alpha = 0.7855
Post-test Mean = 14.73/30 = 49.1%	Post-test Mean = 18.53/31 = 59.8%

The tests were analyzed and descriptive data were gathered. The coefficient alpha, discrimination index, and difficulty index were used to evaluate this first version of the ChCI. The coefficient alpha is a measure of the internal reliability of the test, the ability of the test to evaluate an individual consistently. Alpha ranges from 0 to 1, with 0.7 or higher indicating the test is reasonably reliable. The Chemistry I and Chemistry II inventories both scored above 0.7 (a pleasant surprise). An alpha is calculated for the whole test.

In contrast, discrimination and difficulty indices are calculated for each question. The discrimination index is a measure of how well the question discriminates between the students. To calculate a discrimination index, first the students are ranked by performance on the exam. A top portion of the class is compared to the bottom portion. If every student in the top portion answered the question correctly and every student in the bottom portion answered the question incorrectly, then the question perfectly discriminates between good students and poor students. The discrimination index would be 1. The discrimination index ranges from –1 to 1.

Along with the discrimination index, consider the difficulty index. The difficulty index is the percentage of students who answered the question correctly. Since a question can receive a low discrimination index because the majority of the class answers the question correctly or the majority of the class answers it incorrectly, combining these two indices when evaluating a question is important.

Another way of evaluating questions is to see what effect eliminating a question will have on the coefficient alpha. One of the goals of the ChCI is for it to be administered it in a short period of time. In order to do so, the test had to be shortened. Questions were eliminated so that the Chemistry I and Chemistry II inventories were 20 questions long. Three questions were written on each subtopic so that one question could be eliminated, leaving two questions on that subtopic. But eliminating questions also has a converse affect on the alpha, thus lowering the reliability of the ChCI. Therefore, after a question was eliminated, the alpha was calculated to see what effect that elimination had. Eliminated questions were those that would have the least negative affect on the alpha.

These data, as well as expert judgment, were combined to eliminate weak questions to leave two 20-question tests. Also, a number of questions were modified to make them clearer. The Chemistry I and II Version B were then piloted during the summer of 2003.

The Chemistry I Version B of ChCI was given to university students at the beginning of the semester for a pre-test and again at the end for a post-test. The Chemistry II Version B was given at a community college at the beginning of the semester as a pre-test. Then the questions were spread out over several weekly quizzes. Results from the weekly quizzes were combined to use as the post-test. Results from Version B of the ChCI are summarized in Table 3.

Table 3. Results from ChCI version B.

Chemistry I	Chemistry II
N = 42 University Students	N = 42 Community College Students
Alpha = 0.7135	Alpha = 0.4188
Pre-test Mean = 5.48/20 = 27.4%	Pre-test Mean = 7.17/20 = 35.9%
Post-test Mean = 10.60/20 = 53.0%	Post-test Mean = 10.93/20 = 54.7%

During the summer of 2003, eleven students were interviewed in depth on seven questions on molecular shape from the ChCI Chemistry I Version B. Student interviews gave helpful insight into how students solve problems. For example, a question might be written to test one aspect of the topic, but students might solve it differently. They might use different reasoning that would lead to a correct answer. The item is therefore testing something other than the intended topic. Student interviews are useful for all of these reasons. For these interviews using Version B, students were chosen from those students currently taking the first semester of general chemistry. They were interviewed just after covering the relevant information in lecture. These interviews led to some unique findings in spatial understanding and misconceptions held by these students. They helped to validate the test, too.
The student interviews also helped modify some questions to make them clearer. In one question students had to determine the polarity of a molecule for which they were given a description. It reads:

Consider a molecule with the formula ZA2 where Z is the central atom and A and Z have different electro-negativities. In which of the following cases would this molecule always be nonpolar?

A. If A has 2 lone pairs and Z has no lone pairs.
B. If Z has 2 lone pairs and A has no lone pairs.
C. If Z has 1 lone pair and A has 3 lone pairs.
D. If A is drastically more electronegative than Z.
E. If Z is drastically more electronegative than A.

During the interviews it became obvious that the students were misinterpreting the statements that led to incorrect molecular drawings. The main confusion came from the number of lone pairs of electrons that should be on each of the A atoms. The purpose of the question is not to test whether or not students can draw a molecule from a written description, but whether they can determine the polarity of the molecule. It is not the intent to cause the students to miss the question because they cannot draw the molecule correctly. To clarify the question, the word “each” was inserted into three distracters so that the question now reads:

Consider a molecule with the formula ZA2, where Z is the central atom and A and Z have different electro-negativities. In which of the following cases would this molecule always be nonpolar?

A. If each A has 2 lone pairs and Z has no lone pairs.
B. If Z has 2 lone pairs and each A has no lone pairs.
C. If Z has 1 lone pair and each A has 3 lone pairs.
D. If A is drastically more electronegative than Z.
E. If Z is drastically more electronegative than A.

This new wording was tested on a group of incoming teaching assistants. They were asked to draw the molecule that distracters A, B, and C represented. In nearly every case the drawings that the TAs produced were the intended ones. From student interviews, questions were modified to make them clearer and to ensure that the inventory was testing what was intended.
Results from Version B were evaluated, and a C version of the ChCI was developed. During fall 2003, the Chemistry I and II ChCIs were administered to a large number of students. Available information from this version is in Table 4.

Table 4. Results of ChCI version C.

Chemistry I	Chemistry II
N = 845 Students	N = 845 Students
Alpha = .5541	Alpha = .4761
Pre-test Mean = 4.94/20 = 24.7%	Pre-test Mean = 6.51/20 = 33.6%

Two things should be noted about these results. First, the alphas for each test are low. This is expected because the information on the ChCI has not been covered. Once the post-test has been given, the alpha will be calculated with that data. This will give a more true reliability measure. Second, the results show that each group’s Pre-test Mean is consistent with guessing, which is also expected, given that the students are at the beginning of the semester. At the end of the fall semester, the ChCIs will be administered again as a post-test. The difference between the pre- and post-test scores will represent what was gained by being in the class. Results will be analyzed and used to explore the effect of teaching style on student learning. Results will be shared with faculty members to shape future curriculum modifications. Student beliefs will be further explored with additional interviews. The final table, Table 5, summarizes the steps taken to develop the ChCI.

Table 5. Steps of the development of the ChCI.

1. Pick topic areas to be covered.
2. Search the literature for research on misconceptions in those topic areas.
3. Determine subtopic areas.
4. Write at least three questions in each subtopic.
5. Administer these questions to students. (Version A)
6. Eliminate weak questions based on the discrimination index, difficulty index, and the coefficient alpha.
7. Administer the remaining questions to students. (Version B)
8. Interview students.
9. Modify questions based on results from test and student interviews.
10. Administer the modified questions. (Version C)
11. Repeat steps 8 through 10 until acceptable results are attained.