Assessment Design in the Era of Generative AI

The emergence of generative artificial intelligence (AI) has presented opportunities and challenges for the education sector. Generative AI has demonstrated remarkable capabilities in various fields, including natural language processing (NLP). As technology continues to advance, it is crucial for educators and institutions to prioritize assessment design to take advantage of these new capabilities and to ensure academic integrity. This article explores how educators can thoughtfully design assessments to enhance teaching and learning methods with AI tools. 

Understanding Generative AI and Its Impact

Generative AI refers to the technology that can produce content, such as essays, articles, or even computer code, that mimics human-like quality. While these advancements offer innovative possibilities, they can also be exploited for unethical purposes, including academic dishonesty. Students may be tempted to utilize generative AI tools to generate essays, reports, or solutions to assignments, thus compromising the principles of fairness and personal academic growth. 

Importance of Assessment Design

  1. Authenticity: Effective assessment design focuses on how to measure a student’s understanding, skills, and critical thinking abilities. By deploying assessments that reflect real-world scenarios, we are preparing students for the type of work they will be doing in their respective fields. Additionally, authentic assessments encourage students to demonstrate their knowledge and reasoning, making it more difficult to submit AI-generated content as their own work. 
  2. Higher-order thinking: Assessments should focus on higher-order thinking skills, such as analysis, synthesis, and evaluation, which are difficult for AI to replicate accurately. Designing assessments that require deep comprehension, problem-solving, and application of knowledge encourages students to learn the course materials and engage with the course instructor and their fellow students.  
  3. Process-oriented assessments: Incorporating process-oriented assessments, such as presentations, oral exams, or portfolio-based evaluations, can also help deter the misuse of generative AI tools. These assessments require students to articulate their thought processes, present their work, and engage in dialogue with their instructors. These assessments provide opportunities for instructors to evaluate the authenticity of a student’s work and assess their understanding beyond just a final product. 
  4. Timely feedback and iterative assessment: By adopting iterative assessment practices, instructors can engage students in ongoing dialogue, offer guidance, and evaluate progress. This approach discourages students from relying solely on AI-generated content as it becomes evident that growth and improvement depend on active learning rather than quick fixes through AI tools. 

Strategies for Promoting Academic Honesty

  1. Set clear expectations: Establishing clear guidelines and expectations regarding academic integrity is essential. Instructors should communicate their policies for academic misconduct, emphasizing the importance of original work. For additional guidance, see the page AI Ethics & Privacy. 
  2. Diversify assessments: Employing various assessment types that allow students to demonstrate mastery of the learning outcomes helps mitigate the risks associated with generative AI.  
  3. Use technology detection tools: Utilizing plagiarism detection software or AI-based tools specifically designed to identify AI-generated content can be instrumental in maintaining academic integrity. These tools can help identify suspicious patterns, language inconsistencies, or text that closely matches AI-generated content, enabling educators to investigate further and take appropriate actions. See the AI Tools page for additional details.  
  4. Employ educational interventions: Educating students about the ethical implications of using generative AI tools and fostering a culture of academic integrity can have a lasting impact. By raising awareness, facilitating discussions, and promoting responsible use of technology, students gain a better understanding of the importance of their own intellectual growth and the significance of original work. 

Assessment Approaches

Example 1: Synchronous Assignments

An effective method of mitigating the use of AI tools like ChatGPT is to incorporate more in-class or synchronous activities and assignments—for example, active learning approaches, live presentations, and peer review and discussion.  

Example-in-Action

Students in a large class take notes during a lecture. The instructor pauses after a logical break in content and asks students to form groups of two to four students. They share notes within their group and summarize the lecture together based on their notes. They ask for any clarifications from the instructor during this time (if their group is unable to come to a consensus on a point or topic). The groups add their summaries to a collaborative document or, if there is time, they share out with the whole class. This activity can be done in a face-to-face or synchronous online classroom.  

Example 2: Group Work

Students can work together in groups to solve specific problems related to their assignments or engage in activities that require reflection on their assignment’s development. This approach enhances critical thinking and writing skills while fostering an environment that promotes academic integrity. Faculty members can provide feedback and support students during the development process, fostering engagement and collaboration.  

Example-in-Action

In an electrical engineering course, students complete a hands-on lab in groups connected to previous labs, and the groups are tasked to work together to create an extended product that stores and plays music. The lab assignment includes a graded group discussion and lab report. The lab consists of three tasks. Task 1 requires students to create a top-level block diagram that shows how a BRAM (block random access memory) interface can be constructed to input into the PWM (pulse-width modulation) generator. For Task 2, students design and build the BRAM component to support the music length requirement. The final task requires students to design an interface between a USB (universal serial bus music serial entity and the BRAM to load the data onto the BRAM and then play the music files that were transferred. 

Example 3: Alternative Assignments

Another approach to mitigate the use of generative AI tools is to design assessments that allow students to express their learning through methods other than traditional writing. For example, students can create mind maps, timelines, infographics, videos, or other creative outputs. Introducing alternative assessment approaches can provide multiple means for students to showcase their learning, supporting inclusive learning environments. 

Example-in-Action

In an oceanography course, students select and track an Argo Ocean float that collects ocean data at that location. They update and submit PowerPoint (PPT) presentations at multiple points in the course, as new data from the floats is added every 10 days. Students access the near-real-time data collection system of Argo floats through two different websites by choosing a map projection, e.g., Web Mercator, change the time range, pick a float and copy its number, then adopt a float and select one to track. They are challenged to add some creativity by personifying their float.   

They then create a PPT in which they discuss the float, its drift, and any analysis they draw from the data. The PPT includes the time history of the float for five weeks of the course. They post the PPT in the course and update it every time new data is added by their individual floats. The final presentation is due after five weeks of updates (at the end of the course). This assignment is hard to generate through AI because there is a combination of multiple submissions, data that change over time, and the PowerPoint presentation. 

Example 4: Analysis and Critique of AI-Generated Content

For written texts, students can analyze and critique the accuracy, strengths, and weaknesses of the AI-generated content. As a part of this assignment, students could also identify areas where additional information is needed, assess the alignment of summaries with the original texts, and examine the credibility of cited sources. These approaches help students refine their analytical and critical thinking skills and provide an opportunity for students and faculty members to engage in communication and collaborative idea development. 

Example-in-Action

In the Foundations of Software Engineering course, students experiment with using AI tools as they work on team project assignments. The main purpose of the project assignments is to gain experience with software engineering processes. Experimenting with AI tools allows students to learn how to write prompts that will help the AI generate the documents they need. Another reason the instructors include this in the assignment is that students treat AI as a tool to help make their jobs easier. In this aspect of the project, students document their experience with using AI in assignments. For each assignment, in addition to the work product, students write up a “lessons learned” document describing how they used the AI tools, evaluating how easy the tools were to use, and providing a tutorial for using the tools in the future.  

The instructors relay this explanation to their students:  

One thing we’ve noticed about many successful software engineers is that they are often motivated by a desire to make their own lives easier. They want to create software that simplifies complex tasks and saves time and effort in the long run. Right now, we have an opportunity to take advantage of some exciting new developments in the field of AI. AI tools can now do many of the tasks that software engineers used to have to do manually. AI can be used to generate the documents that are an important part of software engineering projects. By learning how to use these tools effectively, you can save a lot of time and effort on your assignments and projects. 

Stay Connected

Privacy Statement

Accessibility

University Policies

Copyright Policy

© 2021 Johns Hopkins University. All rights reserved.