Is Local AI the Way Forward for Education?

I downloaded a PrivateGPT model and dumped a pile of books on AI into it to see whether querying a narrow corpus of knowledge would improve output. So far the results are encouraging.

Anyone who’s been following my often random, occasionally successful experiments into creating AI bots (that do things like mark essays and pretend to be Elon Musk), may have also seen some criticism of what I’m doing (along the lines of ‘this is garbage, these bots are totally inaccurate/unhelpful/hallucinatory etc.). To be fair I’ve stress tested them all and don’t see it myself, but I guess there are some people who aren’t that keen on generative AI so they’re going to bash people who are playing around with it to see what works.

Thing is, I like the critics, as they force me to dig into why some people are so worried, and to think of ways to address their fears. And I may have found an answer. Well, not the full answer, as no one has that. But at least some response to the worry that, by pushing out responses into an AI trained on the whole of the internet (as most of the big players are) there’s always a chance that what it comes back with is nonsense.

And then I saw Bodo Hoenen’s brilliant work on using PrivateGPT to create AI ‘twins’ of his kids. I read his posts and it got me thinking. About how this might be applied in an educational context. To create expert databases that teachers and students can draw on in their teaching and learning. As always caveat emptor but I think PrivateGPT has a lot of potential.

Imagine having the most knowledgeable PhD beside you as you work. You can ask them any question and they’ll pretty well always have the answer. They’re not a polymath (like ChatGPT) by any means, but you could bring in other disciplines if you wished, in order to counterbalance too narrow a field of knowledge.

That’s kind of what you have with PrivateGPT. You download the AI, drop in a pile of books, papers or whatever, it reads the database and references it in response to queries.

What is PrivateGPT?

PrivateGPT is a local AI model you download onto your computer. There are several models available to download through the Private GPT application, that vary in number of parameters and related memory size requirements.

However, for the purposes of creating a private database of knowledge and enabling the AI to query it, you don’t actually need the PrivateGPT app. It took me a while to work this out, but I eventually found a video that helped.

You’re going to want to head to the PrivateGPT Github page, and have Visual Studio Code downloaded in order to follow along with the above.

You may need to play around with this a bit, as it can be a bit buggy. One of the best ways to debug is to copy paste the error codes into ChatGPT or Claude and ask it for suggestions. It usually helps.

Running PrivateGPT

Once you have it working, you need to set up your database. You have a subfolder inside the PrivateGPT folder called source_documents. All you need to do is drop your files into this folder and run the Ingest command. This splits the documents up into ‘ingestible’ chunks for the AI to read.

This is where it gets interesting. Because what you add to the source-documents folder will create the knowledge corpus on which the AI will base its responses. Are you thinking what I’m thinking? You could have a folder which contains as many books and web articles and reports and papers on a subject as you can get your hands on. For example I have a lot of ebooks on AI in various formats - PDFs, ePub and so on. I dropped them into this folder and hey presto! I was able to query this database in any way I wished.

I’ve yet to fully stress test it and it’s not perfect - I’m still trying to work out how to purge the database between uploads of files, as it seems to remember bits of data previously ingested and at times get a little confused. If anyone knows how to do this let me know!

Some interesting use cases?

But I can see some really interesting use cases which I’ll be exploring in the coming weeks once I’ve de-glitched it. Such as dropping in a knowledge corpus on two totally different fields, such as machine learning and Buddhism, to see whether any insights can be gained by throwing two radically different bodies of knowledge up against one another. It may end up being 75% garbage, but you only need one golden nugget for it to have been worthwhile.

The current AI model (called Groovy) is only 7B parameters which is pretty small, but once I have computer with a larger memory I want to experiment with larger 13B and above models. Maybe I have more of a Masters or MPhil level AI at the moment, and if I want Einstein mode I need to wait for bigger models to be available within this application.

Either way, we are so super early in on all this and I must say it’s a lot of fun to be experimenting in this way. Just don’t take any of it too seriously right now.