I've read stuff about this concept for a while and from a few sources. Essentially many consider getting good LLMs to run on the hardware that's already cheap and ubiquitous has been a major goal for a while, which makes sense.
there are tons of models you can run on just about any setup. They're not opus of course, but still enough to tinker and be useful.
I know, and I've been tinkering with those so far. I'd just like more fluid response ,which this seems to deliver.
arxiv.org
makes sense I think, "more" isn't necessarily always better since you're just giving more potentially false paths to run down. less context that's focused on the right things gives it less shit to trip over. I haven't gotten deep enough in it yet to really say that from practical experience, but when I lurk the ai reddits, most of the optimizations or tactics look like they're really boiling down to context management.I know, and I've been tinkering with those so far. I'd just like more fluid response ,which this seems to deliver.
Another topic for the day. Seems the more context we give LLMs the worse they hallucinate.
How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms
How much do large language models actually hallucinate when answering questions grounded in provided documents? Despite the critical importance of this question for enterprise AI deployments, reliable measurement has been hampered by benchmarks that rely on static datasets vulnerable to...
TLDR; - Under absolute optimal scenarios LLMs still give bad answers around 1.2% of the time. Under normal circumstances that jumps up to around 5%, expending context to "give it more data" actually drives the rate even higher.
I'm getting ready to get unpopular at my work (which is fully at the "We're attaching performance review metrics to how much you use AI" point) because I'm about to showcase this and a litany of the horrible things our non technical folks are doing with it. Such as putting together technical sales strategies for customers based on hallucinations about our products capabilities, then being surprised with actual engineering folks refuse to do meetings with them if that's what they're pushing.
For most of the basic tier uses I have for an LLM Gemini seems to work great. For me Grok is "advanced search built into x.com" I will admit I use almost no "Traditional search" where I have to then go click on links to figure out if they're a moderate match and rely more on the AI summaries. Which might be something I need to reconsider.
This is on my list to look into once I have openclaw/home erp a little farther along. Just checked and I’ve spent $50 on inference/api the last 2 weeks.
