Introduction

The Accreditation Board for Engineering and Technology (ABET) establishes criteria for accrediting engineering, technology, and computer science programs. In its Engineering Criteria, ABET established a set of student outcomes in Criterion 3. Institutions seeking accreditation may create their own sets of student outcomes that are supersets of the ABET student outcomes. For the set of student outcomes, each program must have processes that demonstrate that (1) program performance with respect to its outcomes is being assessed, (2) results of program evaluation are being used to develop and improve the program, and (3) results and processes are being documented.

As a result, engineering faculty members must develop methodologies for assessing performance with respect to outcomes in addition to developing new curricula [1]. Need for these methodologies has created increased interest in developing and identifying relevant assessment instruments [2]. However, only a handful of tools and methodologies are publicly available [3,4]. Meeting ABET Engineering Criteria created significant challenges for almost every engineering program.

Discovery Project

The Foundation Coalition (FC), one of eight engineering education coalitions funded by the National Science Foundation, initiated a project to collect and organize materials on assessment and instruction related to the eleven student outcomes. Project team members included faculty and assessment directors from Arizona State University, the University of Alabama, Texas A&M University, and the University of Massachusetts Dartmouth. During the study, the project team attempted to answer the following research questions:

• Is there is a gap between demand and availability of materials to teach and assess each of the ABET a-k competencies?
• What instructional and assessment materials are being used and have been used in engineering programs?
• What instructional and assessment materials are available to engineering faculty members and programs?
• How may project teams characterize and organize available instructional and assessment materials?

The project found limited resources for both instruction and assessment of ABET a-k outcomes. In response, the FC is constructing a set of minidocuments related to assessment and instruction for the ABET student outcomes to assist individual and program efforts.

For each student outcome, engineering programs must address the following questions:

• What observable student performances would demonstrate competence in this particular area, i.e., what must students be able to do in order to satisfy the outcome?
• How might evidence of student performance with respect to the outcome, while the student is still on campus [5], be acquired and analyzed in order to evaluate a program?
• How might student performance with respect to the outcome be improved? That is, what types of instruction are likely to result in improved student performance and what meaningful learning experience can contribute to the development of these outcomes in undergraduate students [5]?

The preceding questions are addressed by presenting (1) learning objectives, (2) assessment approaches, and (3) instructional approaches. Brief descriptions of the three items are provided for readers who may not be familiar with the terminology used in this document.

Learning Objectives

ABET student outcomes do not describe observable behaviors. Data can only be collected on observable behaviors; therefore, learning objectives are formulated for each outcome in order to describe desired observable student performance related to each outcome. Each mini-document will offer sample objectives that might be associated with the outcome. Section III of each mini-document will provide examples of learning objectives that have been culled from reviews of the literature.

Assessment Approaches

Moving from learning objectives to judgments regarding the degree to which the program is achieving its learning objectives requires relevant, appropriate, and informative data upon which judgments can be based. Prus and Johnson [6] described 15 different assessment methodologies, together with strengths and weaknesses for each methodology. There is no perfect assessment methodology, and evaluators often select multiple assessment methodologies to balance their strengths and weaknesses. Choice of the appropriate methodologies depends on many factors, including the goals and scope of the evaluation. For example, faculty members are usually interested in assessment of the courses that they are teaching as well as assessment of the program to meet the ABET accreditation criteria. Assessment approaches for course and program levels may differ, although there may be overlap. For each of the objectives described in Section III, each mini-document will provide approaches to obtaining data relative to one or more objectives for both the course and program levels. This document will identify when approaches could be applied at course or program levels.

Outcome assessment is a method for determining whether students have learned, have retained, and can apply what they have been taught. Assessment plans have three components: a statement of educational goals, multiple measures of achievement of the goals, and use of the resulting information to improve the educational process. The results of outcomes assessment are part of a feedback loop in which faculty members are provided information that they can use to improve their teaching and student learning [7]. For example, after industry provides feedback on the co-op student or intern, faculty members and administrators can determine if their program and courses within the program are effectively teaching teaming skills and appropriately providing opportunities for students to practice teaming skills in class and on course projects.

Designing a program-level assessment, collecting assessment data on an outcome, and analyzing the results may be complex and less objective than technical research; however, the goal is clear: to determine as reliably as possible if the objectives have been met and, if not, to what should be done to improve each student's educational experience [8].

Instructional Approaches

The ABET a-k outcomes include technical and non-technical (or "soft") skills that faculty members are expected to teach and therefore measure. Improving performance with respect to skills, as opposed to transferring information, requires alternative approaches to instruction [9]. For example, research shows that students need to do more than take notes while listening in order to learn [10]. Woods et al. [11] showed that students do not develop problem-solving skills by (1) watching faculty members work problems, (2) watching other students work problems, or (3) working many problems (even open-ended problems) themselves. Instead, problem-solving skills are learned in a workshop environment. Seat and Lord [12] state that interaction skills (a subset of team skills) cannot be learned by osmosis or simply working in groups. Interaction skills must be taught explicitly. Students need opportunities to develop and practice soft skills. Student-student interaction is an effective way to learn and is often neglected in the traditional lecture course [13].

Teaching critical knowledge, skills, and attitudes required for outcomes a-k must be student centered, where the teaching faculty members are viewed as coaches, facilitators, and guides in the learning process. Learning activities that reflect real-world situations must engage students in individual and collaborative problem solving, analysis, synthesis, critical thinking, and reasoning. New teaching and learning approaches that heighten practical learning and allow students to demonstrate the application of their studies to real-world situations must be put to use [14]. For each learning objective described in Section III, this document suggests instructional approaches for improving student performance.

ABET Engineering Criteria Program Educational Outcomes

The Foundation Coalition offers resources for assessment and instruction related to each of the following outcomes.

• Outcome a: "an ability to apply knowledge of mathematics, science, and engineering"
• Outcome b: "an ability to design and conduct experiments, as well as to analyze and interpret data"
• Outcome c: "an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability"
• Outcome d: "an ability to function on multi-disciplinary teams"
• Outcome e: "an ability to identify, formulate, and solve engineering problems"
• Outcome f: "an understanding of professional and ethical responsibility"
• Outcome g: "an ability to communicate effectively"
• Outcome h: "the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context"
• Outcome i: "a recognition of the need for, and an ability to engage in life-long learning"
• Outcome j: "a knowledge of contemporary issues"
• Outcome k: "an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice"

References for Further Information

1. Accreditation Board for Engineering and Technology, 2005-06 Engineering Criteria, accessed September 2004
2. McCreanor, P.T. (2001) Quantitatively Assessing an Outcome on Designing and Conducting Experiments and Analyzing Data for ABET 2000, Proceedings, Frontiers in Education Conference, accessed November 2004

Abstract: The Mercer University School of Engineering (MUSE) identified eight outcomes to assess for the accreditation process. MUSE Outcome #4 stipulates that students should be able to design and conduct experiments and analyze data.

The committee charged with assessment of Outcome #4 identified four separate skills associated with this outcome; conducting experiments, analyzing experimental data, interpreting experimental data, and designing experiments. The committee determined that assessment of this outcome required documentation of the number of student experiences with each of the four skills and the overall student performance level on each of these skills. A skill assessment worksheet was developed for use in the grading of any activity related to Outcome #4. The worksheet quickly identifies which of the four skills the activity incorporates as well as the performance of the students on each of the individual skills. This worksheet was distributed to instructors teaching courses that contain a significant content related to this outcome.

Data collected from courses in Industrial Engineering, Biomedical Engineering, and Electrical Engineering taught during the Fall of Semester of 2000 suggests that MUSE has been successful at meeting Outcome #4. The data also indicates that the skill assessment worksheet was an efficient and accurate method for collecting quantitative data and identifying weakness in the assessment process. Modifications made to the worksheet by professors to accommodate their personal grading scheme demonstrates that the tool has enough flexibility to be used across multiple disciplines and grading styles while still providing the data required for assessment of Outcome #4.

This paper presents the skill assessment worksheet, data collected using the worksheet, and instructor comments on use of the worksheet.

3. Panitz, B. (1998). Student portfolios. In How Do You Measure Success? Designing Effective Processes for Assessing Engineering Education. Washington, D.C.: ASEE Professional Books, 49-56
4. Shuman, L., Besterfield-Sacre, M.E., Wolfe, et al (2000). Matching Assessment Methods to Outcomes: Definitions and Research Questions. Proceedings, ASEE Annual Conference, access February 2005
5. Briedis, D. (2002). Developing Effective Assessment of Student Professional Outcomes. International Journal of Engineering Education, 18:2, 208-216

Abstract: As engineering programs continue to prepare for evaluation under EC 2000, faculty members are experiencing concern over the less well-defined outcomes of Criterion 3 that address lifelong learning, the global and societal context of our profession, and contemporary issues. Designing and implementing assessment for these outcomes might appear to be a time-consuming and ill-define endeavor. This paper suggests several straightforward classroom strategies that faculty may use to begin to develop these outcomes in their students and describes an effective assessment method that may be realistically implemented and maintained for the long-term.

6. Prus, J., and Johnson, R. (1994). Assessment & Testing, Myths & Realities. in New Directions for Community Colleges, No. 88, Winter 1994
7. Shaeiwtz, J.A. (1996). Outcomes Assessment in Engineering Education. Journal of Engineering Education, 85:3, 239 - 246

   Abstract: Outcomes assessment is a method for determining whether students have learned, have retained, and can apply what they have been taught. Assessment plans have three components: a statement of educational goals, multiple measures of achievement of the goals, and use of the resulting information to improve the education process. The results of outcomes assessment are part of a feedback loop in which faculty are provided with information that they can use to improve both their teaching and student learning. The experience of the Department of Chemical Engineering at West Virginia University is used as an example of how an assessment plan is developed and implemented. Examples of multiple measures of student learning outcomes and how the resulting information is used are presented. The resulting feedback loop allows for corrections to be made in specific classes if deficiencies are found, and indicates when remedial action should be taken to ensure that students do not graduate until a minimum level of competency is achieved.

8. Olds, B., and Miller, R.L. (1998). An Assessment Matrix for Evaluating an Engineering Program. Journal of Engineering Education, 87:2, 173- 78

   Abstract: In this paper we describe the use of an assessment matrix to help faculty develop an assessment plan for their engineering program. Use of the matrix assures that each of the key steps in an effective assessment plan is addressed: setting goals and objectives; selecting performance criteria; planning an implementation strategy; choosing appropriate measures; setting a timeline; and providing timely feedback. The matrix has been used successfully to provide an assessment framework for engineering curricula, individual courses, and educational research projects.

9. Collins, N. F., and Davidson, D. E. (2002). From the margin to the mainstream: Innovative approaches to internationalizing education for a new century. Change, 34:5, 50-59
10. Chickering, A., and Gamson, Z. (1987) "Seven Principles for Good Practice," AAHE Bulletin, 39:3-7, ED 282 491, 6pp, MF-01; PC-01

    Good Practice Encourages Contacts Between Students and Faculty

    Good Practice Develops Reciprocity and Cooperation Among Students

    Good Practice Uses Active Learning Techniques

    Good Practice Gives Prompt Feedback

    Good Practice Emphasizes Time on Task

    Good Practice Communicates High Expectations

    Good Practice Respects Diverse Talents and Ways of Learning

11. Woods, D. et al (1997). Developing Problem Solving Skills: The McMaster Problem Solving Program, Journal of Engineering Education, 86:2, 75-91

    Abstract: This paper describes a 25-year project in which we defined problem solving, identified effective methods for developing students’ skill in problem solving, implemented a series of four required courses to develop the skill, and evaluated the effectiveness of the program. Four research projects are summarized in which we identified which teaching methods failed to develop problem solving skill and which methods were successful in developing the skills. We found that students need both comprehension of Chemical Engineering and what we call general problem solving skill to solve problems successfully. We identified 37 general problem solving skills. We use 120 hours of workshops spread over four required courses to develop the skills. Each skill is built (using content-independent activities), bridged (to apply the skill in the content-specific domain of Chemical Engineering) and extended (to use the skill in other contexts and contents and in everyday life). The tests and examinations of process skills, TEPS, that assess the degree to which the students can apply the skills are described. We illustrate how self-assessment was used.

12. Seat, E., and Lord, S. (1999). Enabling Effective Engineering Teams: A Program for Teaching Interaction Skills. Journal of Engineering Education, 88:4, 385-390

    Abstract: A program for teaching interaction skills to engineers and engineering students has been developed. Based on cognitive style theory, this customized program uses the typical engineer’s problem solving strengths to teach skills of interviewing, questioning, exchanging ideas, and managing conflict. The goal of this program is to enable these problem solvers to apply their technical skills more effectively by improving interpersonal interactions. The modular nature of the training program makes it easily transportable, and all or part of it can be used in courses that require students to work in teams. This paper discusses what makes this training “a good fit” with engineering students, the background for its content, and the program’s six modules. Personal experiences with teaching this material and recommendations for implementation are discussed. Similarities and differences between teaching the engineering professional and student, themes of student perceptions about the training, and future directions are also addressed.

13. Mourtos, N.J. (1997). The Nuts and Bolts of Cooperative Learning in Engineering. Journal of Engineering Education, 86:1, 35 -37

    Abstract: A great number of engineering students work alone most of the time. This is in sharp contrast with industry where most of the work is performed in teams. The ability to work in a team effectively is not acquired automatically. It takes interpersonal and social skills which need to be developed and practiced. In addition, research shows that the student-student interaction, often neglected in traditional ways of teaching, is a most effective way of learning. Thus, it is imperative that we encourage our students to work with each other in their efforts to achieve their educational goals. In this paper I discuss my experience with Cooperative Learning (CL) in a variety of engineering courses during the last four years. The discussion includes benefits and problems along with possible solutions. Lastly, I have made an effort to evaluate the impact of CL on student performance and attitude.

14. Meier, R.L., Williams, M.R., and Humphreys, M.A. (2000). Refocusing Our Efforts: Assessing Non-Technical Competency Gaps. Journal of Engineering Education, 89:3, 377-385.

    Abstract: This study reports the findings of a National Science Foundation-funded study* focused on providing solutions to the identified needs for curricular change in Advanced Technological Education programs. The purpose of this study was to explore the extent of competency gaps in science, mathematics, engineering, and technology (SMET) education graduates as perceived by business and industry leaders. Due to the nature of the research questions investigated in this study, the methodology was divided into three phases. Phase one employed a widely accepted multi-step, scale development procedure to determine the domain of the subject matter. Phase two validated survey items. Phase three comprised two parts; part one prioritized SMET competency gaps. Part two utilized Hoshin quality analysis techniques to group, identify, and sequence thematic content areas for curricular development. This study found that SMET programs must extend the boundaries of their traditional curricula to include competencies such as: customer expectations and satisfaction, commitment to doing one’s best, listening skills, sharing information and cooperating with co-workers, team working skills, adapting to changing work environments, customer orientation and focus, and ethical decision making and behavior.

Reports on Engineering Education

Engineering Deans Council and Corporate Roundtable of the American Society for Engineering Education (1994). Engineering Education for a Changing World

The Green Report recommended that engineering curricula work to develop the following outcomes in addition traditional emphasis on engineering science and design.

• Team skills, including collaborative, active learning;
  communication skills
• Leadership
• A systems perspective
• An understanding and appreciation of the diversity of students, faculty, and staff
• An appreciation of different cultures and business practices, and the understanding that the practice of engineering is now global
• Integration of knowledge throughout the curriculum;
  a multi-disciplinary perspective
• A commitment to quality, timeliness and continuous improvement
• Undergraduate research and engineering work experience;
  understanding of the societal, economic and environmental impacts of engineering decisions
• Ethics

Restructuring Engineering Education: A Focus on Change

NSF 95-65 Restructuring Engineering Education: A Focus on Change is a report from the Division of Undergraduate Education (DUE) of the Directorate of Education and Human Resources (EHR) of the National Science Foundation (NSF). It reports results from a workshop and offers four recommendations.

• Engineering education must encourage multiple thrusts for diversity.
• Engineering Education needs a new system of faculty rewards and incentives.
• Assessment and evaluation processes must encourage desired expectations for both faculty and students.
• The changes needed for engineering education require comprehensive change across the campus, not just in the engineering college.

Engineer of 2020

The Engineer of 2020 is an initiative by the National Academy of Engineering to define the attributes required for an engineer in 2020 and actions that may be taken to promote achievement of these attributes. The Phase 1 Report on the NAE vision for engineering graduates state that graduates:

• will possess strong analytical skills, like engineers of yesterday and today,
• will exhibit practical ingenuity
• will be creative
• will be good communicators
• will master the principles of good business and management
• will understand the principles of leadership and be able to practice these principles
• will have high ethical standards and a strong sense of professionalism
• will possess a complex attribute described as dynamism, agility, resilience, and flexibility
• will be life long learners.

In a nutshell, the NAE report states, "What attributes will the engineer of 2020 have? He or she will aspire to have the ingenuity of Lillian Gilbreth, the problem-solving capabilities of Gordon Moore, the scientific insight of Albert Einstein, the creativity of Pablo Picasso, the determination of the Wright brothers, the leadership capabilities of Bill Gates, the conscience of Eleanor Roosevelt, the vision of Martin Luther King, Jr., and the curiosity and wonder of our grandchildren."

Newport, C.L. and Elms. D.G. (1997). Effective Engineers. International Journal on Engineering Education, 13:5, 325-332

Abstract: The aim of engineering education is to produce effective engineers. Achieving this aim depends on knowing what an effective engineer is. The present research looks at engineers in the workplace to determine what qualities make some engineers more effective than others. Effective engineer qualities were collected from engineer employers then tested using questionnaires designed to measure the predominance of the qualities in engineering individuals. Qualities associated with mental agility, enterprise and interpersonal capability correlated most significantly with effectiveness. Effectiveness did not correlate with achievement in tertiary education. The results showed that many of the qualities associated with effective engineer behavior are learnable and can be taught within an education program.

Engineer Profile: Transferrable Integrated Design Engineering Education

The profile of an engineer characterizes the engineer who is productive after graduation and advancing rapidly in responsibility as a professional. These attributes span all of the ABET EC 3a-k outcomes plus additional abilities and attitudes important to the engineer’s working environment. Therefore, these attributes encompass the range of actions or attitudes desired in engineers at the time they graduate and others for which a bias toward learning is present at graduation. Many of these attributes are possible outcomes of capstone design projects that engineering students experience.

Engineering Education: Designing an Adaptive System

The report from the National Research Council calls for a flexible and adaptive education system in which self-assessment and evaluation at each institution spurs curricular innovation, experimentation, and systemic change.

Resources

National Study of Liberal Arts Education

The National Study of Liberal Arts Education (NSLAE) is a large-scale, longitudinal study to investigate critical factors that affect the outcomes of liberal arts education. The research is designed to help colleges and universities improve student learning and enhance the educational impact of their programs. This is one of the most comprehensive national studies of the effects of American higher education on student learning and development ever conducted. The seven outcomes

• Effective reasoning and problem solving
• Inclination to inquire and lifelong learning
• Integration of learning
• Intercultural effectiveness
• Leadership
• Moral character
• Well-being

are of interest to instruction and assessment of the eleven program educational outcomes. The project has identified a set of outcome assessment instruments that may also be of interest to faculty members working with the eleven program educational outcomes.

Oberst, B. S., and Jones, R. C. (2004). Canaries in the mineshaft: engineers in the global workplace. Proceedings, ASEE Annual Conference and Exposition, accessed 29 April 2005

Abstract: We need to get beyond the overheated rhetoric about the offshoring of jobs and look seriously at how engineers and the engineering profession want to live and act in society. This article outlines the current debate about the migration of jobs overseas and the dismemberment of engineering and technology jobs into commodifiable pieces. It is written so as to provide a cross-section of information sources for the reader interested in pursuing the topics further, but may also be read without attention to the footnotes.

Robinson, M. A., Sparrow, P. R., Clegg, C., and Birdi, K. (2005). Design engineering competencies: future requirements and predicted changes in the forthcoming decade. Design Studies, 26(2), 123-153

Abstract: This paper seeks to address omissions in previous research by identifying a future competency profile for design engineers. A three-phase methodology using both quantitative and qualitative methods was employed. A competency profile for the future design engineer, 10 years hence, was generated. The profile consisted of 42 competencies divided into the following six competency groups (in descending order of criticality): personal attributes, project management, cognitive strategies, cognitive abilities, technical ability, and communication. Furthermore, non-technical competencies were forecast to become increasingly important in the future. Results were discussed with reference to their implications for the design engineering industry.

Linking Student Learning Outcomes to Instructional Practices

Cupp, S., Moore, P. D., and Fortenberry, N. L. (2004). Linking Student Learning Outcomes to Instructional Practices – Phase I. Proceedings, ASEE Annual Conference and Exposition, Retrieved, 27 June 2005

Abstract: This paper begins to test the assumption that stakeholders in engineering education know what set of teaching and learning practices by faculty and students will lead to desired student learning outcomes. The work reported here seeks 1) to identify from published sources, a set of desired engineering student learning outcomes, and 2) to characterize and categorize teaching and learning practices. Desired student learning outcomes identified in published sources mirrored twelve of the engineering accreditation criteria supplemented by five additional outcomes not explicitly addressed within current accreditation criteria: a) multidisciplinary systems thinking, b) business aspects of engineering practice, c) appreciation for diversity, d) good work ethic and commitment to continuous quality improvement, and e) logical thought process. Sixty-one percent (11) of the learning outcomes are categorized as Technical, and 39% (7) are categorized as Social.

With respect to teaching and learning practices, an initial investigation uncovered six published sources that collectively identified 146 practices. It is noteworthy was that all of the identified practices were for actions by faculty and teachers – not students. We place the practices into a five-dimensional taxonomic structure. An effort to link “effective” practices to specific outcomes is suggested for future work.

Cupp, S., Moore, P. D., and Fortenberry, N. L. (2004). Linking Student Learning Outcomes to Instructional Practices – Phase II. Proceedings, Frontiers in Education Conference, Retrieved, 27 June 2005

Abstract: In previous work, we identified five student learning outcome areas that might productively augment the current engineering accreditation criteria. In this work we review the literature on a) how these outcomes might be assessed and b) what instructional practices may encourage their attainment. Multiple assessment instruments are identified for the five student learning outcome areas. We offer examples of instructional practices that appear to align with developing a) multidisciplinary systems perspectives, b) appreciation for diversity, and c) familiarity with business matters.
We see the research base underlying instructional practices as lacking adequate breadth and depth.

Bjorklund, S. A., and Fortenberry, N. L., (2005). Linking Student Learning Outcomes to Instructional Practices – Phase III. Proceeedings, ASEE Annual Conference and Exposition

Abstract: More than ever, today’s engineering colleges are concerned with and attuned to improving the processes and outcomes of educating tomorrow’s engineers. To that end, ABET’s “3a through k” criteria identified eleven learning outcomes expected of engineering graduates. Based on a rigorous review of the literature, the first phase of our work found four additional student outcomes desired by the engineering education community, and suggested that an engineering graduate also ought to demonstrate 1) ability to manage a project (including a familiarity with business, market-related, and financial matters), 2) a multidisciplinary systems perspective, 3) an understanding of and appreciation for the diversity of students, faculty, staff, colleagues, and customers, and 4) a strong work ethic. During Phase II of this project, we identified several assessment instruments that might measure those outcomes and began searching for instructional “best practices” thought to promote the 15 desired learning outcomes. This paper, based on Phase III of the project, provides empirical evidence from and identifies the gaps in higher education and engineering education journal articles that link instructional best practices with the 15 desired student outcomes in engineering education.