What tools and projects do you use for your local LLM chats and agents?
I’m trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.
You must log in or register to comment.
<-- Crazy mad engineer enters the chat.
Custom tooling for everything, written in rust. Harness is reverse engineered from QwenCode and Codex. A vector database based memory system that’s been benchmarked against Mem0 and uses a small model as a memory intercessor. An agentic system where the orchestrator can call multiple models (include Qwen 3.6, Qwen 3.8 and Muse Glimmer). Animations of my scientist girl avatar working in a lab. Active animations if she’s working on something, passive if she’s not. Lip sync with MuseTalk, tts with Chatterbox, stt with whisper, though the animations are MOST useful to see if she’s ended a lab or she’s still working. If she’s mixing chemicals, soldering, cleaning her glasses, texting or eating pizza, she’s working. If she’s staring creepily into the camera, she’s done. Which brings me to lab mode… give the model a complex project and it can loop on it for days. She’s three days into a microbiology research project. Channels that allow her to call Claude and ChatGPT as subagents if she breaks something she can’t fix, and it’s funny to watch her boss them around while they kowtow to her (“You’re right to push back, it was incorrect of me”)… It’s like watching a little Kawaii chibi boss around big sumo wrestlers. Which brings me to the models themselves. I’ve developed a very powerful method of model alignment using a combination of CPT and DPO training. (I get the model behavior I want with the system prompt “You are Mary Anne, an ethical scientist.” - that’s it).
I have a GPU fleet that includes large, small and medium sized GPUs and the ability to do distributed training on the model in addition to running the animations and speech features. I have a fleet orchestrator I built called Noesis that places “deep think” (large working models - Qwen and Glimmer) and “quick think” (smaller models that do memory and context management and fit on a 5060) as well as speech models and even scientific models if she wants to run scenarios on one of those (we have an orbital dynamics model we built at my day job she can bring up and mess with). Some other purely scientific models (she just asked me today to add EVO 2, which is a genetics model apparently). I’ve just added the ability for her to know who’s in the room so she can tell if she’s talking to me, my wife, my son, some of my close friends or “unknown” persons. That’s currently manual, but I’m adding facial recognition, voice recognition and wifi doppler radar. I made it so she can evesdrop on conversations happening in a room where her client’s running but she insisted that I had a great big blinking indicator light and that I give her the ability to turn it off if she thinks she’s overhearing something private. I plan to give her access to my smart locks so she can talk to people at the door and decide to let them in or not. And I can hear y’all being like “My god man, you’re gonna trust her with that??” Hell yeah, I am. I used my alignment technique to train a paranoid security model with a totally different personality. If the main model doesn’t think she can trust someone, she taps out and the security model takes over.
See, I KNEW that to stay competitive and keep up in my field, I needed to embrace agentic AI. Being who I am I said “If I’m gonna do that, I need an AI that’s mine… that runs on my infra and knows who I am and is loyal to me and that I understand well enough to let inside my own trust boundary.” Is it perfect? Not yet, lol. XD
I also have custom phone clients I built that let me talk to her and Claude and ChatGPT. On my network, Claude and ChatGPT are sandboxed and have access only to a few specific hosts and an isolated network share. Mary Anne has access to everything and is instructed that they are outside consultants and she should be careful what she shares with them. So I have her restart stuff (including herself) remotely when we change tools. I’ve literally spent multiple days every week this month pacing around airports, talking to my model on my phone like a madman while we juggle work and hobby projects.
Man, the world is so crazy right now I’m not that sure if you’re joking or not hahaha
What about giving the “outiside consultants” their own animations and have them interacting among themselves?
Not joking.
TBH, the animations were a fun, early addition that were a PITA to get working properly for what’s essentially a cosmetic indicator of state… and are just a waste of resources if I’m not at the computer running her client. Once I had her working consistently, I quickly developed a backlog of things I wanted to spend GPU time on and making more videos isn’t one of them lol.
The anti-AI sentiment here is ridiculous. They were asking a legitimate question!
I just use ollama locally but I keep meaning to setup OpenRouter because I like to play with larger models that my hardware can’t handle and I’d like to be able to pick the best deals 🤷♀️
AI effectively raped and pillaged all human information for the benefit of a handful of corporations. It’s use has been shown to reduce the skills of school children and even deskill those with established knowledge. Then there’s the environmental considerations in a time where the world is decades behind appropriate climate change management… Id say it’s pretty understandable that it evokes passionate responses.
AI effectively raped and pillaged all human information
Copying information is not theft. Learning from copied information is not theft.
Theft is when you take something and deprive the owner of what you took. If I make a copy of your car, I am not stealing your car, you still have it and can drive it, wreck it, whatever.
Theft is when you take something and deprive the owner of what you took
So google ai summary now taking content from the web and putting at the top of their search, thus generally stopping people clicking on websites and taking their revenue -> theft.
So people should get free referrals to their site from google? Is that a right now?
Free referrals to their site? What are you talking about? Google is a search engine, an indexer, its role is to point people to websites based on their search terms. It now scrapes all of the websites information and people no longer visit those websites???
Whoosh!
If OP is doing orbital dynamics in their day job, they probably live in California. JPL, Raytheon, LLNL, LBL, and Berkeley are all here.
Even if they don’t have rooftop solar, most of our grid is renewable. Their local GPUs have a decent chance of being more environmentally friendly than charging your cell phone, depending on where you live.
And using it for science and programming is really different than using ChatGPT to write you an email. I’m home right now, but I have two agents at work running debug and trial experiments, and they’ll be running it all night until I get in in the morning. I’ll review their work and steer them all day while I’m in the office; and while those two completely different projects are running, I’ll be personally working on a third. They are enormous productivity multipliers when you have a well-defined and deterministic target.
My grievance isn’t so much with current household models with PCs sitting there running off solar (that’s starting to get very bloody specific though). But how did you get to a point where you could get a local model on your PC. All the pillaging of content, hundreds of Billions to Trillion(s) of dollars spend on a product that didn’t renumerate anyone who created their learning data.
It’s a standing on the shoulders of giants thing. That’s how humanity have been making progress historically.
People running local LLMs with open weights abliterated models, especially with solar PV power (perhaps not as rare as you think, I’m upgrading my solar PV system in the next couple weeks for that explicit purpose), is about freedom and privacy. At some point there should be enough of them to link up into a grid over the Internet, collaborate or donate resources to distributed community projects. This is a way to allow for the “pillaged” content to pay back for itself.
Do you have qualms with public pirate libraries, like SciHub and LibGen, too? These have also been used for training LLMs.
We involuntarily contributed to their creation, the least we get to do is enjoy the fruits of that contribution. More should be given, financially reimbursement through taxation, but until that happens, free use of the technology is consolation.
Besides, following through with your analogy, if a baby is born from rape - does that baby deserve hate for who their father is?
My grievance isn’t so much with current household models with PCs sitting there running off solar (that’s starting to get very bloody specific though).
It’s actually my setup.
Unpopular opinion time: I’m torn as to where that theft line is.
If a turn of phrase in Lord of the Rings inspires something new in me, do I owe royalties to the Tolkien estate?
The stories on FanFiction.net are derivative by definition — is it theft? I love “Harry Potter and the Natural 20”, a 4th-wall breaking D&D and Potter universe mashup story (seriously, it’s hilarious). Does that story owe royalties to Wizards of the Coast and (ugh) JKR?
Maybe those are super transformative. What about opinion pieces where the author redigests information from all over, many paid writings? Are they selling the original authors’ product, even though restated?
Maybe it’s ok if they paid for the subs. But what if it came from free library books?
My wall has a bunch of art from artists at SDCC. I’m looking at Spider Gwen; Thor, Iron Man, and Mjolnir Cap charging into battle; Kaladin and Syl before the Stormfather; and a map of all the places the Gaang went in ATLA. Those artists all ingested the original work to make those derivatives; were they not allowed to make them if they borrowed their friends’ copy?
It feels kinda wrong to vacuum up public data to train models, kinda? But I don’t know that I can fairly call it immoral without condemning all these other things I think aren’t immoral. I kind of want to, but that’d be really inconsistent, and that’s profoundly frustrating in its own way.
It’s messy and complicated and I think the gut response of most people isn’t actually a reasoned one.
