CS 205 Research Instruments Guide

Pre/Post Concept Test & Survey Instruments for the AI-Assisted Teaching Study (Mixed Design: Quasi-Experimental + Crossover RCT)
Share with Rolf for review and coordination

Contents
  1. Part 1: Pre/Post Concept Test (both sections)
  2. Part 2: Pre-Survey (both sections ideally)
  3. Part 3: Post-Survey (both sections ideally)
  4. Part 4: Administration Guide

Part 1: Pre/Post Concept Test

Purpose

Measure conceptual understanding of data structures and algorithms at the start and end of the semester. The same test is given twice (pre and post) to both sections. This lets us calculate normalized learning gains and control for any baseline differences between sections.

Format: 20 multiple-choice questions, ~25 minutes. No code writing — pure conceptual reasoning.

Key principle: Questions should test understanding, not memorization. Students who memorized a definition but can't apply it should get the question wrong.

Important Notes for Rolf

Concept Test Design: 20 Questions Across 7 Topic Areas

Topic Area# QuestionsWhat It Tests
Algorithm Analysis (Big-O)3Reasoning about time/space complexity, growth rates
Linear Structures (arrays, linked lists, stacks, queues)4Operation costs, choosing the right structure, tracing operations
Recursion2Tracing recursive calls, understanding base cases
Trees (BST, heaps)4Traversals, insertion/deletion effects, heap properties
Hash Tables2Collision resolution, load factor, when to use hashing
Graphs3Traversal strategies, shortest paths, representation tradeoffs
Sorting2Algorithm behavior, stability, best/worst cases

Sample Questions (20 Items)

Below are ready-to-use questions. Each includes the answer and a rationale explaining why the correct answer is correct and why distractors are tempting.

Q1Big-O
What is the time complexity of the following code fragment?
for (int i = 1; i < n; i = i * 2) {
    System.out.println(i);
}
  1. O(n)
  2. O(n log n)
  3. O(log n)
  4. O(n²)

Answer: C — O(log n)

The loop variable doubles each iteration (i = i * 2), so it takes log₂(n) iterations to reach n. Distractor A is tempting because students often assume all single loops are O(n).

Q2Big-O
Consider two algorithms: Algorithm A runs in O(n²) time and Algorithm B runs in O(n log n) time. For which input sizes is Algorithm A guaranteed to be slower than Algorithm B?
  1. For all input sizes n > 0
  2. For sufficiently large n (beyond some threshold)
  3. Only when n > 1000
  4. Never — Big-O doesn't predict actual running time

Answer: B

Big-O describes asymptotic behavior — it guarantees the relationship holds for sufficiently large n, but not for all n (constant factors matter for small inputs). Distractor D is partially true but overstates the case.

Q3Big-O
What is the time complexity of searching for a value in a sorted array of n elements using the most efficient algorithm?
  1. O(1)
  2. O(log n)
  3. O(n)
  4. O(n log n)

Answer: B — O(log n)

Binary search on a sorted array runs in O(log n). Distractor C (linear search) ignores that the array is sorted. Distractor A would only apply to direct index access.

Q4Linear Structures
You need a data structure that supports efficient insertion at the front and removal from the front. Which is the best choice?
  1. ArrayList (dynamic array)
  2. Singly linked list with head pointer
  3. Sorted array
  4. Binary search tree

Answer: B

A singly linked list with a head pointer supports O(1) insert and remove at the front. An ArrayList requires O(n) shifting for front insertions. This tests whether students understand the cost tradeoffs, not just definitions.

Q5Linear Structures
What happens when you push elements 1, 2, 3, 4, 5 onto a stack (in that order) and then pop three times? What value is on top of the stack?
  1. 1
  2. 2
  3. 3
  4. 4

Answer: B

Stack is LIFO. After pushing 1-5, the stack top-to-bottom is 5,4,3,2,1. Three pops remove 5, 4, 3. Top is now 2. Tests basic LIFO understanding through tracing.

Q6Linear Structures
In a circular queue implemented with an array of size 5, the front index is 3 and the rear index is 1. How many elements are currently in the queue?
  1. 2
  2. 3
  3. 4
  4. 5

Answer: B — 3 elements

Elements are at indices 3, 4, 0 (wrapping around). Count = (1 - 3 + 5) % 5 = 3. This is a common point of confusion for students learning circular arrays.

Q7Linear Structures
Which operation is O(n) for a singly linked list but O(1) for a doubly linked list (both with head and tail pointers)?
  1. Insert at front
  2. Insert at back
  3. Remove from back
  4. Access the first element

Answer: C

Removing from the back of a singly linked list requires traversing to the second-to-last node (O(n)), since there's no previous pointer. A doubly linked list can follow tail.prev in O(1). Both A, B, and D are O(1) for both structures with head+tail pointers.

Q8Recursion
What does the following method return when called with mystery(5)?
int mystery(int n) {
    if (n <= 1) return 1;
    return n * mystery(n - 1);
}
  1. 5
  2. 15
  3. 120
  4. 25

Answer: C — 120

This computes n! (factorial). 5! = 5 × 4 × 3 × 2 × 1 = 120. Tests ability to trace recursion. Distractor B (15) might come from adding instead of multiplying.

Q9Recursion
A recursive method has no base case. What happens when it is called?
  1. It returns 0
  2. It runs forever without any error
  3. It causes a StackOverflowError
  4. The compiler rejects it

Answer: C

Each recursive call adds a frame to the call stack. Without a base case, calls never stop, and the stack eventually overflows. Distractor B is wrong because the stack is finite. Distractor D is wrong because missing a base case is a runtime error, not a compile error.

Q10Trees
What is the in-order traversal of this binary search tree?
        8
       / \
      3   10
     / \    \
    1   6    14
  1. 8, 3, 1, 6, 10, 14
  2. 1, 3, 6, 8, 10, 14
  3. 1, 6, 3, 14, 10, 8
  4. 8, 3, 10, 1, 6, 14

Answer: B — 1, 3, 6, 8, 10, 14

In-order traversal of a BST visits nodes in sorted order (left, root, right). Distractor A is pre-order. Distractor D is level-order. This tests both traversal knowledge and BST properties.

Q11Trees
You insert the value 5 into the BST shown above. Where does it go?
  1. Left child of 6
  2. Right child of 6
  3. Left child of 8
  4. Right child of 3

Answer: A — Left child of 6

5 < 8 (go left to 3), 5 > 3 (go right to 6), 5 < 6 (go left — empty slot). Tests BST insertion logic by tracing the path.

Q12Trees
What is the maximum number of nodes in a binary tree of height h?
  1. h + 1
  2. 2h
  3. 2h+1 - 1
  4. 2h

Answer: C — 2h+1 - 1

A complete binary tree of height h has 2h+1 - 1 nodes (assuming height of root = 0). Distractor A is the minimum for a "stick" tree. Tests understanding of tree structure bounds.

Q13Trees / Heaps
Which property distinguishes a min-heap from a binary search tree?
  1. In a min-heap, every node is smaller than its children; in a BST, left child < parent < right child
  2. A min-heap is always balanced; a BST is always balanced
  3. A min-heap supports O(1) search; a BST supports O(log n) search
  4. They are the same data structure with different names

Answer: A

The heap property (parent ≤ children) is different from the BST property (left < parent < right). Distractor B is wrong — BSTs are not always balanced. Distractor C is wrong — heap search is O(n). Tests understanding of structural vs. ordering properties.

Q14Hash Tables
A hash table with 10 slots uses the hash function h(k) = k % 10. Keys 12, 22, and 32 are inserted. With chaining, how many keys are stored at index 2?
  1. 1
  2. 2
  3. 3
  4. 0

Answer: C — 3

12 % 10 = 2, 22 % 10 = 2, 32 % 10 = 2. All three hash to index 2. With chaining, they form a linked list at that slot. Tests collision understanding.

Q15Hash Tables
The average-case time complexity for search in a hash table with a good hash function and low load factor is:
  1. O(1)
  2. O(log n)
  3. O(n)
  4. O(n log n)

Answer: A — O(1)

With a good hash function and low load factor, collisions are rare, giving average O(1) lookup. The question specifies "average-case" and "good hash function" to be precise. Distractor C is the worst case (all keys in one bucket).

Q16Graphs
Which traversal algorithm would you use to find the shortest path (fewest edges) from node A to node B in an unweighted graph?
  1. Depth-First Search (DFS)
  2. Breadth-First Search (BFS)
  3. Either DFS or BFS — both find shortest paths
  4. Neither — you need Dijkstra's algorithm

Answer: B — BFS

BFS explores nodes level by level, so the first time it reaches a node, that's the shortest path (in an unweighted graph). DFS may find a path, but not necessarily the shortest. Dijkstra is for weighted graphs. This is a critical conceptual distinction.

Q17Graphs
An undirected graph has 6 vertices and 15 edges. What kind of graph is this?
  1. A sparse graph
  2. A complete graph
  3. A tree
  4. An impossible graph

Answer: B — A complete graph

A complete undirected graph with n vertices has n(n-1)/2 edges = 6×5/2 = 15. A tree with 6 vertices would have exactly 5 edges. Tests knowledge of graph properties and formulas.

Q18Graphs
For a graph with V vertices and E edges, what is the space complexity of an adjacency matrix representation?
  1. O(V)
  2. O(E)
  3. O(V²)
  4. O(V + E)

Answer: C — O(V²)

An adjacency matrix is a V × V 2D array, regardless of how many edges exist. O(V + E) is the space for an adjacency list. Tests understanding of representation tradeoffs.

Q19Sorting
Which sorting algorithm has a worst-case time complexity of O(n log n)?
  1. QuickSort
  2. MergeSort
  3. Bubble Sort
  4. Insertion Sort

Answer: B — MergeSort

MergeSort is O(n log n) in all cases (best, average, worst). QuickSort's worst case is O(n²) — a critical distinction that students often miss. Bubble Sort and Insertion Sort are O(n²) worst case.

Q20Sorting
You have an array that is almost sorted — only a few elements are out of place. Which algorithm would perform best?
  1. MergeSort
  2. QuickSort (random pivot)
  3. Insertion Sort
  4. Selection Sort

Answer: C — Insertion Sort

Insertion Sort runs in O(n) on nearly-sorted data because almost no shifting is needed. MergeSort and QuickSort still do O(n log n) regardless. This tests understanding of best-case behavior, not just worst-case.

Part 2: Pre-Survey

Purpose

Measure students' baseline engagement, self-efficacy, and AI perceptions before the semester's instruction takes effect. Administered in both sections in the first few weeks.

Format: 24 Likert-scale items + 4 demographic questions. ~8 minutes to complete.

Scale: 1 = Strongly Disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly Agree

Section A: Demographics (4 items)

D1. What is your major? [CS / CE / IS / Math / Other: ___]

D2. What grade did you receive in the prerequisite programming course? [A / B / C / D / Other: ___]

D3. How many programming courses have you taken before this one? [0 / 1 / 2 / 3+]

D4. How often have you used AI tools (ChatGPT, Claude, Copilot, etc.) for coursework before this class? [Never / A few times / Monthly / Weekly / Daily]

Section B: Engagement & Motivation (8 items)

Adapted from the Student Course Engagement Questionnaire (Handelsman et al., 2005)

# Statement SD   D   N   A   SA
E1I look forward to attending this class.
12345
E2I actively participate in class discussions and activities.
12345
E3I find the topics in this course interesting.
12345
E4I put effort into understanding the material, not just passing.
12345
E5I feel motivated to study for this course outside of class.
12345
E6When I don't understand something, I seek help (office hours, peers, online).
12345
E7I pay attention during the entire lecture, not just parts of it.
12345
E8I connect what I learn in this class to real-world applications.
12345

Section C: Self-Efficacy in Data Structures (8 items)

Adapted from the Computer Science Self-Efficacy Scale (Ramalingam & Wiedenbeck, 1998)

# Statement: "I am confident that I can..." SD   D   N   A   SA
S1...understand the difference between arrays and linked lists and when to use each.
12345
S2...analyze the time complexity (Big-O) of a simple algorithm.
12345
S3...implement a stack or queue in Java.
12345
S4...trace through a recursive method and predict its output.
12345
S5...explain how a binary search tree insertion works.
12345
S6...choose the appropriate data structure for a given problem.
12345
S7...debug a program that uses data structures like trees or hash tables.
12345
S8...succeed in this course overall.
12345

Section D: Perceptions of AI in Learning (8 items)

Custom items informed by recent AI-in-education literature (Kasneci et al., 2023; Wang et al., 2023)

# Statement SD   D   N   A   SA
A1AI tools (like ChatGPT) can help me learn course material more effectively.
12345
A2I trust the accuracy of explanations provided by AI tools.
12345
A3Using AI tools for learning feels like cheating.
12345
A4I would prefer a class that integrates AI tools into instruction over one that does not.
12345
A5AI tools can replace the need for human tutors or teaching assistants.
12345
A6I worry that relying on AI tools will prevent me from truly learning the material.
12345
A7I feel comfortable using AI tools as part of my study routine.
12345
A8AI will play an important role in how computer science is taught in the future.
12345

Scoring Notes

Part 3: Post-Survey

Structure

The post-survey repeats all 24 items from the pre-survey (Sections B, C, D) — this enables pre-post comparison on each construct. Additionally, it includes a short Section E with experience-specific questions.

Demographics (Section A) are not repeated.

Section E: Course Experience (Post-Only, 6 items)

These items appear only on the post-survey. For the treatment section, items P3-P6 ask about the web app. For the control section, replace those items with P3c-P6c below.

# Statement (Treatment Section) SD   D   N   A   SA
P1This course helped me grow as a computer scientist.
12345
P2I would recommend this course to other students.
12345
P3The animations in the web app helped me understand data structure operations.
12345
P4The interactive exercises in the web app helped me learn actively during lectures.
12345
P5The AI hints helped me work through problems I was stuck on.
12345
P6I wish I could use this web app outside of class for self-study.
12345

Control Section Variants (replace P3-P6):

P3cThe lecture examples helped me understand data structure operations.
12345
P4cI felt actively engaged during lectures (not just passively listening).
12345
P5cWhen I was stuck on a problem, I had access to helpful resources.
12345
P6cI wish the course had more interactive or technology-enhanced learning tools.
12345

Open-Ended Questions (Post-Only, Both Sections)

O1. What aspect of this course helped you learn the most? Why?

O2. What would you change about how this course is taught?

O3. (Treatment section only) Describe a specific moment when the AI hints helped you (or didn't help you) understand something.

Part 4: Administration Guide

Checklist for Both Instructors

WhenWhatWho AdministersDuration
ASAP (this week!)Pre-Test (concept test, 20 MCQs)Both instructors, in class~25 min
ASAP (this week!)Pre-Survey (demographics + 24 Likert items)Both instructors, in class~8 min
Week 8-9Optional mid-semester pulse (E1-E8 only)Both instructors~3 min
Week 14-15Post-Test (same 20 MCQs)Both instructors, in class~25 min
Week 14-15Post-Survey (24 repeated + 6 new + 3 open)Both instructors, in class~12 min
Week 16Interviews (treatment section only)PI (or trained RA)20-30 min each

Tips for High Response Rates

For Rolf Specifically

Recommended Tools for Administration

Generated for CS 205 AI Education Research Study — Spring 2026
Share this document with all collaborators. Last updated: February 2026.