Asking AI Something I Already Know

Published: 2024-04-11 4:38 PM

Category: AI | Tags: llm, chatgpt, model, education, teaching, data, analysis

I saw a post, I don't remember where, about how to gauge the usefulness of a model. Instead of asking it something you don't know about (and, as a result, cannot verify the results), ask the model something about which you know a lot. Use your knowledge to judge the potential usefulness of a particular tool. I'm not interested in using an LLM to generate new information - I'm more curious about the LLM-as-assistant potential where it extends what I know and can do on my own, but prompts me to maybe think differently about the topic. I've already mentioned the potential of using models to brainstorm and today, I did some tinkering with using an LLM to highlight trends in small datasets.

I took some completely anonymous data from my standard-tracking database and decided to ask some questions of a model after loading in the prompt. First, I tried on Mistral 7B locally, but it took forever to process (like forever forever - see my last post on using a local model on a mid-level machine) and my CPU fan went from 0 to the moon almost immediately. I decided not to cook my machine.

I tried both ChatGPT and Google Gemini. ChatGPT was able to do some interesting things like the types of comments I leave for student work attached to a particular standard score:

Here are the distributions of different feedback comments based on the provided categories:

1. Understanding of Particle Behavior: Occurs in 34 instances.
2. Use of Scientific Vocabulary: Occurs in 24 instances.
3. Specificity of Explanations: Occurs in 27 instances.
4. Accuracy of Explanations: Occurs in 11 instances.
5. Mistakes or Misconceptions: Occurs in 22 instances.
6. Follow-up or Corrections: Occurs in 10 instances.

There were ~150 records in the set I pasted in and, looking at the same data in a CSV, I think it did a pretty good job of categorizing my specific feedback to students. It's helping me note that I'm recognizing what students can do (categories 1-3) but not always giving them next steps (category 6), which is a critical component to effective feedback.

The model also made the categories on its own - this was built with text I'd sent to students specifically, not pre-categorized and not copy & pasted out of a comment database.

I know I want this kind of information about the feedback I'm leaving and asking ChatGPT to do the summary for me was surprisingly insightful as I worked through my reflection.

Gemini was not able to do this - it gave me some ideas about patterns I could investigate, but did not do any of the interpretation as part of the session that ChatGPT was able to do.

In this case, I didn't know the patterns the chat session brought out, but I knew the data source for the discussion, making it a productive use of my time. I wasn't shooting in the dark to find or transform information already in my mind (like a brainstorming or question-writing session would be). I was asking specific questions about specific data I knew I wanted to use. In this case, I did feel more efficient and I was able to glean some insight into how I can improve my practice. I can also show this to students as part of our reflective and metacognitive processing work.

While I think this is a potential good use of time, I wish I could make this more possible on a local model so I'm not contributing to the energy requriements to run larger models, so this is not something I'm planning on doing on the regular. I think looking at trends at assessment points or using specific assignments as datasets to identify trends is more likely at this point.