Today at ISU
Cyclone News
Admissions

Learning and Teaching > Program Assessment > Process to achieve outcomes

 

Engineering Career Services; College of Engineering Cooperative Education Engineering Internship and Engineering Summer Work Experience Program

Contact:
Larry F. Hanneman
301 Marston Hall
lfhannem@iastate.edu
294-2540

i. Organization - how are departments organized for continuous assessment/improvement of learning?

The three major subunits of Engineering Career Services (career services, experiential education and career self-management) are seamlessly integrated around the service delivery concepts of competency/ability-based career self-management and professional development. The programs and processes are completely aligned with ABET Criterion 3 (a-k) Outcomes. The career services and experiential education subunits deliver large volumes of direct, quantitative and indirect, quantitative and qualitative measurements to each academic program and to the college. The career self-management unit delivers directly to students, and through the programs, education support and learning tools. Working with the Deans Task Group for Cooperative Education and Internship and an Employer Advisory Board, ECS processes and disseminates to the programs large volumes of constituent feedback, program analysis and improvement support.

ii. Objectives/Outcomes - What processes do you use to develop and review program objectives and program/course outcomes?

Our efforts are directed specifically at supporting each individual accredited program’s efforts to address ABET Criterion 3 Outcomes (a-k). Support is also offered, upon request, to develop and deliver assessment technology and learning tools to any program-specific outcomes that may be defined using ability-based statements. ECS works directly with the individual programs or collectively through the Dean’s Task Force for Cooperative Education and Internship and its associated Employer Advisory Board. ECS will engage a consultant, Development Dimensions International, as appropriate. Metrics and learning tools are developed using DDI, large numbers of constituents (faculty, employers and students) and “critical incident” data gathering and analysis technologies.

iii. Assessment Measures - direct vs. indirect, qualitative vs. quantitative, what groups/constituencies, what rubrics

Direct Measures Indirect Measures
Ability-based student self-assessment At-graduation in-profession placement
Ability-based supervisor assessment Six-month follow-up in profession placement
Ability-based graduate self-assessment Student evaluation of work experience
Ability-based manager assessment (Written and focus groups)

Engineering Career Services provides each program with direct, quantitative measures of each of Criterion 3 (a-k) Outcomes. Data at the individual program level and at the college level are generated at the individual Key Action (65 dimensional), Competency (15 dimensional) and Outcome (a-k) (11 dimensional) levels; these data include experiential education student self-assessment and workplace supervisor comparisons at each level. These data are created and reported for each academic period. Data capture rates for engineering experiential education students and their supervisors exceed 90%. Assessment data are collected from over 500 students and 500 supervisors in over 300 companies each academic year.
The Competency to Outcome (a-k) rubric, complete with weighting factors for each Competency/Outcome relationship, was developed and validated in collaboration with approximately 200 constituents: faculty/staff, employers, students, parents and international colleagues. Similar post-graduation (e.g., 5-year follow-up) service is provided to the individual programs upon request. These data are collected using the web-based delivery tool Online Performance and Learning OPAL™. In addition to the assessment data gathering function, OPAL™ delivers to students and supervisors a large collection of fully integrated personal development and performance planning and management tools.

Additionally, Engineering Career Services provides program-specific and college level at-graduation, in-profession placement data. In-profession placement is defined to be employed in-profession, accepted to graduate school or professional (law, medicine, etc.) school, accepted military appointment, or returning to home country to one of these outcomes. Data collection rates exceed 98%. A follow-up data collection is completed at six months after graduation. Data are reported to each program and to the college for each academic period and each academic year.

iv. Continuous Improvement - What changes have been made in your program as result of your assessment feedback process?

Direct Measures

Ability-based assessment and learning tools are now delivered upon request of faculty into classrooms, learning communities, design classes and extra-curricular activities. Additionally, ECS is delivering these ability-based assessments for direct measurement of program objectives in the post-graduation workplace for one program.

Engineering Summer Work Experience was added to the academic program in each of the accredited programs and ability-based assessment (student and supervisor) included as a course requirement.

An additional Competency, Safety Awareness, has been identified to be added to the Competency to Outcomes (a-k) rubric. Efforts are underway with constituents to define the mapping of Safety Awareness to (a-k) and to establish the Outcome-specific weighting factors. Assessment of Safety Awareness has already begun in all experiential education workplace experiences.

Indirect Measures

Engineering Career Services expanded at-graduation, in-profession placement data collection to include Masters and Doctoral graduates and completed a data collection program to retrieve data for the past ten years (1993-2003) with an 85% recovery.

v. Documentation and Dissemination - This would be primarily the web record of our objectives, outcomes, processes, and results.

Detailed program-specific and college-wide at-graduation and six month post–graduation in-profession placement outcomes data are distributed electronically to the program chairs and college administration each academic period. Data are reported at the Bachelors, Masters and Doctoral levels. Annual academic calendar summary data are also distributed to these constituents as well as to the Provost office for institutional reporting activities. Summary placement data for each academic period for the past four years are also available on the Engineering Career Services web site at: HYPERLINK "http://www.eng.iastate.edu/ecs/employers/StatisticsSalary.html" http://www.eng.iastate.edu/ecs/employers/StatisticsSalary.html.

OPAL™ Detailed Reports at the individual Key Action and Competency levels and the Competency Rank Order Reports for each accredited program for each academic period are available at: HYPERLINK https://intranetapps2.eng.iastate.edu/assessment/reports/ https://intranetapps2.eng.iastate.edu/assessment/reports/. Graphical presentation of each individual ABET Criterion 3 (a-k) Outcome for each accredited program and for the college are in development and will be available at this site.

vi. College role - What could the college do to help with portfolios, out-year assessments, data analysis and tools, etc.?

Provide the resources to support the continued research, development and deployment of ability-based assessment and associated learning tools for student development. These efforts should focus on data analysis, visualization and interpretation across all programs and within individual programs, and should support development and delivery to constituents of learning tools aligned with pedagogical and curricular change. Support expanding delivery to classrooms, learning communities, extra- and co-curricular activities and post-graduation assessments addressing program objectives. Support expansion of the existing (a-k) rubric to address program-specific Outcomes, as appropriate. These services will be delivered via the web, using OPAL™.

Support the efforts to develop the integration of our two data systems, OPAL™ and ISUCMS (Symplicity™), for seamless, single-portal delivery to constituents.

Support the leveraging and expansion of these resources and efforts across the institution in support of other colleges’ needs for ability-based assessment, student development and continuous curriculum improvement processes for accreditation and licensure, such as The Higher Learning Commission and Iowa teacher licensure requirements.