Keep LLM's local

A lot of the use cases for LLMs I’ve heard from people is along the lines of “Help me get started on this document, presentation, report or speech” or “Summarise this document”. Which are pretty simple requests, but it’s when people send data with no thought…that’s when I get frowny.

You could go to the big player and pay the monthly fee, take a risk on sending it data. But it’s a cost of living crisis and I like to keep your data safe.

I tried Ollama today.

I’m liking the open models. It’s open source and runs locally on your laptop/desktop. I used Llama 3 8B as the model. Small at 4GB, but it worked well. Quicker than I expected. Llama 3 70B is 20GB…I don’t need that right now.

It’s multi-modal so you can give it an image and ask questions about it, and it can also summarise documents. You could, if you were so minded, create your own models or refine existing models with your own data…but that’s not straightforward. At all.

All local, nothing shared, no cost.

It won’t flirt with you though.

And obviously it’s an LLM so can spew nonsense, don’t take outputs as gospel.

(I asked it to act as a Risk Director and define the differences between Risk and Uncertainity…the output wasn’t bad at all)

Some more details on it

What is Ollama? A shallow dive into running LLMs locally | Isaac Chung

Being able to run LLMs locally and easily is truly a game changer. I have heard about Ollama before and decided to take a look at it this past weekend.

Isaac ChungIsaac Chung

Ollama GitHub page

GitHub - ollama/ollama: Get up and running with Llama 3.2, Mistral, Gemma 2, and other large language models.

Get up and running with Llama 3.2, Mistral, Gemma 2, and other large language models. - ollama/ollama

GitHubollama