context
Professor Ian Schnee teaches philosophy at the University of Washington. In his Introduction to Logic class, he experimented with creating an AI tutor bot to help student learning using NebulaOne’s RAG.
notes
intersection of humanities, computing, and technology
- Exciting and impactful space to work and teach
- Huge societal impact from AI and technological change
- Growing awareness in all sectors:
- Values and ethics
- Students care deeply about AI’s influence
- Recognize AI is not value-neutral
- Want a voice in how it is applied, adopted, and how it affects their lives
opinion on AI
- AI is powerful and here to stay
- Goal: harness AI to enhance learning, not detract from it
- Prepare students to
- Enter the workforce
- Use AI responsibly (not cheat or become dependent)
- Excel at university work
- Prepare students to
- Adopted a collaborative attitude toward AI use in the classroom, providing students the opportunity to guide the direction of norms
why NebulaOne?
- Rooted in pre-existing teaching issues, post-covid challenges
- Committed to innovating to recover teaching effectiveness, not giving up
- Getting through to students and getting them excited about the material is what makes the job fun
post-COVID academic performance decline
- Issues
- Decline in
- Engagement
- Attendance
- Reading outside class
- Traditional exams no longer effective
- Decline in
- Learning outcomes clearly dropped
- With the same difficulty level, exams showed full-letter grade (~10%) decrease
- Possible influencing factors
- Students less familiar with in-person testing
- COVID changed grading practices in high school to be more effort-based
- GPA drop for the class as a whole can’t be used as a measure because of homework assignments and S/NS grades
rationale for AI tools
- Inequities and unfairness caused by AI tools
- Some students used AI regardless of whether it was allowed
- Some students could pay for more advanced models
- Goal: Provide a powerful, free, equitable AI tool
- 24/7 tutoring
- Personalized learning experience
- Grounded in solid pedagogy
- Partnered with UW to use the experimental NebulaOne infrastructure
how the tutor bot works
- Custom AI trained on course materials
- Textbook
- Problem sets
- Answer keys
- Past exams
- Slides
- YouTube video transcripts
- Uses Retrieval-Augmented Generation (RAG)
- Off-the-shelf engine that you provide reference materials to
- Pulls from uploaded materials before default model data
- Hoped that AI can help sift, prioritize, and contextualize learning material
student interaction
- Provided layered training for students
- How to find the public domain for the AI tool
- Strategies for effective prompting
- LLMs do not shine in domains such as math and logic
- Students need practice to figure out how to get it to work for this class
- Provided types of prompts that can help with different types of questions
- Focus on metacognition
- Students monitor whether AI is helping or subverting their learning
- Challenges
- Needed a lot of configuration
- Students taught to
- Judge answers critically
- View it as a collaborative tool, not an oracle
- Meaningful interaction with the bot supplements learning
- Specific application: helps with logical validity questions
- Students often struggle with examples
- Can have back and forth with tutor bot until they are fully convinced about the reasons they got a question wrong
performance and feedback
- Launched Winter 2025 in Introduction to Logic class (~300 students)
- Exam score comparisons:
- Pre-COVID avg: 82%
- Post-COVID avg: low 70s
- With tutor bot: 83% — restored performance
- Student usage
- 1/3 heavy use
- 1/3 limited use (concerned about quality)
- 1/3 disliked or distrusted it
- Students are free to abstain based on moral or practical concerns
- The presence of the tool creates an equitable playing ground in which students can make their own decisions about the resources they choose to use
limitations
- Experimental, not fully reliable
- Concerns about hallucination
- Students need to know the approximate answers in order to judge the accuracy of the tutor bot’s response
teaching with AI
- Combined AI use with 15-minute metacognitive segments each lecture
- Advocate for blend of both non-tech and tech-based solutions
- Pen and paper notes
- Active learning with interactivity
- Tutor bot is an out-of-class tool
- Key principles
- Motivation and engagement matter more than tools
- Support a strong community and care for learning
- Judicious integration of tech in the classroom
- Make class expectations clear
- This is how the class, this is how you will be examined, here are the tools, and here are the steps you can take
- Teach principles of deep processing rather than passive consumption
reflections and future ideas
- Excited by results
- Outperformed expectations
- Encourages others to try:
- Tools like RAG are available online
- DIY tutor bots are possible with basic tools (e.g. Google + LLMs + course content)
cognitive offloading
- Worried students may offload homework to AI
- How to combat
- Teach how learning works
- Emphasize understanding over completion
- Promote enhancement of learning rather than offloading
- Foster motivation
- Students need to see relevance
- Even with the tools, if students don’t care about learning materials, they will still offload their thinking if students
- Have no time
- Don’t understand learning goals
- Don’t think learning goals are relevant to them
- Intrinsic value matters to combat transactional approach learning
- Teach how learning works
raw notes
- Intersection of humanities, computing, and technology
- Most exciting places to work & teach
- Why? Huge impacts on society (cmpt and tech)
- Adoption of AI
- growing awareness from all sectors
- values
- ethical questions
- students vitally interested
- AI not value neutral
- want say in how it
- applies
- is adopted
- role in lives
- Thoughts about AI and where do they belong in terms of teaching and learning experience
- powerful and here to stay
- various perspectives on speed of adoption, cautious, fizzle out? impact
- wants to understand how to harness and improves tudent learning rather than detract
- prepare to be
- ready to enter workforce
- teach and make sure not just using to cheat or impede
- do jobs better at universities
- philosophical perspective
- what ethical questions to ask
- norms in classroom
- what ethical questions to ask
- collaborative attitude
- opportunity to guide direction of classroom and community
- involve students
- when deciding to try NebulaOne, what were u thinking, what made u decide to try, this is what im tryna learn
- guided by something that pre-existed AI
- problem in teaching in old ways
- post-covid
- education impacted
- return to classroom
- hard time achieving same learning outcomes with same exams
- harder to engage, keep attendance up, get them to read outside classroom
- could dumb down or give in
- committed to figuring out recovering effectiveness
- getting through to students and getting them excited about material is what makes his job fun
- post-covid
- attempts to solve unsuccessful
- constantly trying to innovate
- figure out effective ways to use AI
- solve problems of not knowing how to use AI, how to be effective
- Covid GPA average dropping
- achievement of learning goals dropped in classes
- intuitive sense
- really saw drop in person, compare to pre-covid tests
- isomorphic and equal difficulty questions
- full letter, 10% point drop in how they were doing
- factors
- not used to taking as many in person exams
- classes in hs graded by effort and not really rigourously by performance
- covid impact on students
- struggling in diff ways than 5-10 yrs ago
- don’t know if grade points dropped
- satisfactory grades
- may impact studying for final exam
- satisfactory grades
- move to remote learning
- take-home exams different and hard to compare to in person exams
- achievement of learning goals dropped in classes
- saw problem, saw issue, trying to address → AI, using new tools, what do i bring into classroom, why, how gonna put into action, tell about thinking process
- observed studnets alr using AI regardless of whether allowed
- subscription models more effective than older/free models
- if some students can pay, advantage, willing to use even if not allowed
- unfairness and equity issue
- wanted to provide best learning tool with no additional cost
- allows students to get promise some people hope AI can provide
- 24/7 tutoring, customizable learning environments
- supplement to teaching
- investigated using AI like this
- equitable
- grounded in solid pedagogy
- found UW experimenting with NebulaOne infrastructure
- reached out
- what did you create with this tool and how it works?
- tutor bot
- AI bot trained on course materials
- engine running in bg that can be chosen from variety of frontier engines
- follow up: did u train yourself and how it responds?
- nebula one allows you to create bots
- infrastructure: RAG
- instead of training LLM
- off the shelf engine
- reference materials
- used as primary source before going to original training data
- customized tutor-bot experience without heavy, costly, resource intensive training period
- tailored?
- yes, can adjust controls for how precisely/vaguely it’s stuck in classroom domain
- set to heavily focus on course materials
- uploaded everything he had
- textbook
- problems sets and answer keys
- past exams
- ppt slides
- yt video transcripts
- hope AI can sift through and prioritize info useful for students
- as students, how to interact with tutor bot?
- gave multi-levels of training
- how to find public domain
- available on internet
- how to use effectively
- logic is difficult
- LLMs don’t shine in domain, like math
- predicting next token
- not simplest way to implement math or logic
- needs practice to figure out how to get it to do work in class
- types of prompts that can help different types of questions
- troubleshooting
- beta testing
- needed a lot of configuration
- metacognition, how learnign works
- how to reflectively monitoring learning
- is interaction helping or subverting learning
- helping or hurting learning?
- does it improve as students ask qs?
- unknown
- can give feedback on quality of answers
- doesn’t know how information is getting used
- nebulaone? backend resource?
- feeding in?
- when first start? which class?
- winter 2025 formal logic, 300 person intro logic lecture
- what help students?
- logical validity
- premises, result guaranteed to be true
- students struggle with examples
- can be tricked
- follow logical fallacies
- invalid arguments
- equivalent arguments
- ask tutorbot why they got question wrong
- back and forth until they are fully convinced
- meaningful interaction supplements learning
- logical validity
- using in spring?
- such diff class
- existentialism and film
- teaching again remotely in summer
- now that saw and resuing, what surprised? what feedback from students?
- student feedback: excited to share opinions, love or hate
- how open students are to having conversations
- surveys
- 1/3 used a ton
- 1/3 used sparingly
- number 1 reason because quality not good enough, unsure whether could trust answers
- 1/3 hated it and didn’t want to touch it
- happy that useful
- no problem with against AI, moral objections, don’t want to force
- {using it to be equitable, level playing ground of learning in terms of access to resources}
- concern about hallucination
- upfront that it’s experiment, dk how effective
- not perfect reliable
- need to approx answers and judge tutor bot’s response
- get answer and have to think about it themselves
- collaborative
- not all have friends in class
- not equal
- but online resource simulates collaborative experience
- chat with neighbor
- collaborative
- usage breaking up into groups, seems students still have qs about AI do we believe this thing or not? engage in question and ans with tutorbot
- teaching students to monitor interactions with AI one of learning goals
- centerpiece of metacognitive aspect
- improve test scores not just ai bot, but paired with metacog lessons
- 15 min of every lecture period
- combo of those two that helsp students
- better learners
- learned how to learn better
- applied lessons immediately in class contents
- synthesis of two things
- teaching students to monitor interactions with AI one of learning goals
- tool, must make decisions on how best to deploy to do what it’s supposed to do how well can something like this be implemented? can it be used in other settings?
- easy app to class because concrete problem solving
- problem sets
- depends on content engaging
- difficult texts in other philosophy classes
- large amount of materials that get tutorbot to focus on the thing you want it to
- could be useful for all kinds of classes
- easy app to class because concrete problem solving
- friend who teaches math hates IT traditional teaching still hearing resistance? sense of discussion at univ
- some ways to innovate in classroom with no tech
- getting students to take note by hand
- concept mapping
- lecture
- walls of laptops
- barrier to meaningful interaction with instructors
- walls of laptops
- sympathize with people who don’t like what’s tech oding in classrooms
- useful to get out of classrooms in certain situations
- avoidign all tech total mistake
- logic textbook completely interactive
- active rather than passive learnign
- combine with lowtech tools
- seamlessly integrate low and high tech tools
- some ways to innovate in classroom with no tech
- in classroom, tutor bot is something outside of classroom, doens’t interfere with teaching
- prim role help students when working problem sets, exam, study group
- not for lecture time
- lecture tech
- polleverywhere
- complicated proposition
- studies: use of tech for polleverywhere is just or more effective as not allowing
- use judiciously adn integrate tech into classroom
- if not engaging students, and all loking at lapttops and not listening, failure
- tech can’t solve all problems
- work to find out efective ways to impelment tech
- motivate and enthusiastic, build community, care for learning
- better outcomes when using bot in last class?
- demonstrably better
- identical difficult exam to pre-covid as benchmark
- 82% before extra credit pre
- post: low 70s
- winter tutor: 83% — made up difference
- gave free use outside classroom, no restrictions
- could homework
- control on learning because exams in person, no tech available
- if just use for hw, hurt exams
- no surprises — don’t depend on it
- this how works, this how examined, here r tools, here r steps
- principle of deep processing
- dramatic, excited?
- super excited
- other ideas doesn’t move needle
- happy that outperformed
- other thoughts?
- if instructors are excited to try experience for stuents and tutor bots, can do with tools avialble with tools on internet
- RAG: retrieval augmented generation
- Google for free
- Notebook LLM
- can create own trained on class materials
- cognitive offloading
- expresses worry that students would use bot to do homework
- combat ways
- 1 give studnets tools to know what they ought to be doing: teach how learning actually works, if choose, can use for enhancement rather than offloading: must have knowledge — skill component
- 2 give motivation: even if have tools, if don’t care about learning material, will still do if no time, if no see learning goals, don’t know but don’t think applies to them, so transactional approach, only care about grade, no intrinsic value: need both skills and motivation