LLLM = DIY AI

You’ve probably read @gpentelie’s posts about AI. Recently, @CarolJoy brought up AI that runs completely within your walled-off computer. This intrigued me. First off, let’s be honest: considering the IT talent around here, I’ve got no standing to write this post. I graduated with an arts degree—53 years ago. Lately, though, I’ve seen tantalizing hints, personal accounts of this surprisingly accessible, strange new world of the mediated human-computer interface. Even more than the home computer boom of the 80s, artificial intelligence is electrifying businesses everywhere while inspiring a worldwide frenzy of millions of home experimenters in Asia, America and Europe. It’s like the radio boom of the 1920s, created by an ambitious young industry, but much of its excitement and, critically, political support was driven by amateurs and supported by professionals who correctly sensed big dollars in it.

Most of the public face of AI is the LLM—the Large Language Model that matches patterns and predicts output. A Substack article about a daily journal writer who set up his own local LLM got to the heart of what I’ve thought more about lately: the “why” of it. What, realistically, do I want out of it? What level of AI augmentation would we really be talking about? And how much of that AI could be done on a local LLM, located entirely on my kitchen table? LLLM = DIY AI. How much can it do? I want to find out.

Local LLMs are what CarolJoy’s comments on a “$400 data center” refer to. They’re fun and can do some useful work. But they can’t replace cloud-based LLMs. They lack depth of inference and reasoning. Yet I’m getting farther than I thought with my laptop. It can work AI as networked or local. Your computer can too. Details to follow.

When you go local, you are, in effect, homeschooling a computer, quite literally adjusting its values (in AI jargon, “weights”). Like a homeschooling parent, you have a choice of using training material that is non-mainstream. For example, if you don’t want to use graphics from the internet, you can have your own library of stock images. There are obvious limits and tradeoffs to doing it that way, and like homeschooling, setting yourself apart takes some work.

This field has a bewildering jargon, of metaphorical containers and volumes, languages, models, engines, layers, and intersections. My household goes back a long way with Nvidia GPUs (graphics processing units), since its chips were fundamental to making our Christmas 2005 Alienware computer something special. Back then, it was “merely” a well-heeled specialty manufacturer with a bright future, beloved in the “serious gamer” community. Now, it’s one of the most valuable companies in history, worth $5 trillion. Instead of buying that computer, I should have bought their stock

What made them so important? Nvidia’s prospects were changed forever in 2007 when it released CUDA, Compute Unified Device Architecture, a method of using the unique talents of its game-boosting graphics GPUs to perform conventional computation in a massively parallel way.

Instead of the classic Von Neumann model of a computer, with serial data running through a high-speed, high-bandwidth centralized CPU, an Nvidia GPU is a single large graphics chip that uses many sub-cores. Programmers figured out how to break jobs up so those hundreds of sub-processors could work on the job at the same time. To this day, a level of AI software is usually based on Nvidia’s CUDA architecture, whether the chip is Nvidia or not. That’s market clout.

Having looked at AI history and how it interacts with China’s, I brought my sights down to earth reading about local LLMs, their advantages and limitations. One issue is persistence; LLLMs tend towards short-term memory and are less likely to remember every stage of a project’s evolution. Memory utilization tools can enhance local LLMs with, in effect, short-term memory within the short-term memory. As any stoner would understand, better short-term memory helps keep reasoning on track.

AI in the cloud has its own issues, other than privacy. Tradeoffs must be made. There’s only so long that the cloud AI’s servers will retain your project in active memory; afterwards, it distills or condenses it for later retrieval, in effect needing to reload parts of it each time. As you might expect, the more you pay, the longer you stay.

Whether you go local or not, you can choose one “dashboard”, or control platform, and independently choose a “model” that it runs, or even a virtual harem of several competing AI models for doing core reasoning. Just jump in and try them, the free ones, anyway. They aren’t people; you aren’t hurting Sam Altman or Demis Hassabis’s feelings if you switch around. It’s as if you bought a car in two parts, the controls and the engine, both separate and independent of each other. Ollama is an example of an open-source control panel that can run various models of AI.

Google’s Gemma 4 12b level is a new one that's getting a lot of attention among Apple users. A technique called 4-bit quantization metaphorically lowers the resolution to fit comfortably within the Mac’s unified RAM. Google has a complete web showroom of control panels like Gemma’s, usable with innumerable flavors of open-source models that do the actual AI work.

There’s LM Studio. Anthropic has Claude Projects. Some are cloud, some are local, some allow both. All have their partisans.

Often an AI’s free tier is enough, but if you’re doing work worth paying for, an obvious factor is how integrated you want your workflow to be. Alphabet (Google), Microsoft (Copilot), and Meta (Facebook) take advantage of your familiarity with their earlier tools and make use of connections to other parts of their ecosystems, offering to integrate your emails, your taste in web surfing, YouTube selections, family photos. Grok (X) plays up its integration with hundreds of millions of Tweeters. Anthropic’s Claude is what Wall Street would call a standalone “pure play”.

On the consumer level, the software itself is usually free; it’s a classic razor-and-razor blades model. The use level that most hobbyists, enthusiasts and home businesses employ, roughly $20 a month, is being squeezed, the way cable and the internet jacked up prices once they’d turned into something you basically couldn’t do without.

None of this is likely to affect me very much if at all. I’m just trying things out, learning them for the fun of it. Aside from that hobby angle, I have listed a few specific projects that I’d like to see AI tackle. I don’t need Chat/GMcV. I want agentic AI to do tasks.

Evaluating any potential AI-related project involves benefits and effort. Does one justify the other? Tremendous benefits are worth tremendous work and computer time. Merely interesting, even tantalizing results don’t justify as much of a slog. A massive personal journal project, my ultimate AI goal, is technically straightforward but has lots of potential decision points. One (repeatedly) recommended data structure for this scale of text work is RAG, Retrieval Augmented Generation, which uses vectors to in effect prepare deep searches, relating them to each other nearly instantly.

On a more achievable scale, I could easily turn some of my Ricochet posts about film and television into an actual film, history, and culture book. Of course it would only be a digital book, but I could do it so simply; the subjects are already organized. I could give it away as a PDF or even make a (very) few instant-printed copies of it for family and friends.

Another AI project, at least potentially, that “lives” here on Ricochet: Ricochet Silent Radio fiction. I could take a handful of key RSR stories: The Legend of Ricochet Silent Radio, Arahant Begins, The Wire Men, Tomorrow Never Knows, The Judge Mental Christmas Special, Everything’s Better In Color, Lord Protector of the United States, Utah Wheels and Rails--and use AI to instantly format them, with occasional black and white illustrations, modeled on the graphics of 1950s pulp detective and mystery magazines (Alfred Hitchcock, Ellery Queen).

Or have it create a comic book/graphic novel of 2017’s Everything’s Better in Color. The Ricochet member I included in the story liked it so much that an artist friend of his was thinking about making a graphic novel of it; nine years later, maybe Mr. A.I. can handle the job.

A completely different, but seemingly logical problem for AI: automatically aligning pairs of stereoscopic slides. This bedeviled me in 2011- ‘13. I did the best I could. Today a computer could do it better and a lot faster. But can it run locally? Unknown.

Let’s look under the hood of what current higher prices mean for non-local, cloud-based AI customers, the vast majority, and why a growing minority are being driven to try going local. MindStudio had a good article on mainstream pricing. Current favorite Claude Fable 5 charges $10 per million tokens in, $50 per million tokens out. In real terms, a million tokens in equals about 1,500 pages of input. The output will almost always be much terser; you’re paying for a conclusion, and for the thinking time it took. If that’s too rich for you, Anthropic also markets a less dazzling but still amazing AI model, Sonnet, at only $3 per million tokens in and $15 per million out.

Which brings me all the way back around to local LLMs; you don’t pay for tokens. Once it’s running you don’t pay for anything but the electricity, and the software itself is usually free open source. Yes, local is less capable than cloud, and is not always easy to set up, so look for the easy stuff. Turns out there’s plenty of it.