This makes me deeply curious about how LLMs understand language. Do LLMs relate cognates more than words that are dissimilar in different languages? I wonder if that plays some role in the effectiveness of tokenization.
I have no idea if the similar spelling will somehow help - I used that mostly because it's a simple way if illustrating the close relationship, but I suspect you'd find that the meanings of closely related words are likely to more directly overlap.
The grammar is perhaps more likely to help. Similar word order etc. Even weirdness like German - my only top grade on a German essay in school was one where I on purpose ignored what I thought I knew about German and tried to evoke "old fashioned" Norwegian. The result was guessing at a bunch of grammatical structures that I didn't know if was valid German. Turned out I was right about most of it - century old Norwegian was far closer to century old Danish, was a lot closer to valid German, and enough so to impress my teacher enough to overlook a number of orthographic mistakes.
The same thing works for guessing German grammar from English. The farther back you go in English, the more its grammar resembles German.
"What sayest thou?" -> "Was sagst du?"
In fact, for the above, you don't even have to know a single German word. You just have to know what for question words, "wh" -> "w", that the English "y" at the end of a syllable usually comes from an older Germanic "g" sound, and that "th" was replaced by "d" in German. That gets you 90% of the way from early modern English to modern German in the above example.
> Isn't it just a matter of not changing the previous context?
Yes, but a lot of harnesses change previous context. E.g. the system prompt injects the current time/date, working directory, files in the working directory, etc. Compaction also changes the whole previous context. I _think_ changing the list of tools also invalidates cache, so invoking a subagent with different tools would invalidate the cache.
My vague impression is that it's in a similar vein to functional programming languages. It generally disallows doing things that lead to bugs (cache misses in this case), and presumably allows you to do those things in a way that makes it much clearer that this is likely to cause cache misses. I would guess that in this paradigm, you don't mutate your existing session, you derive a new session by mutating the prior context into a new context.
Cache is always there, it’s just that it only caches up to the point where an input token changes. So if the tools list is early in the prompt, changing it would limit cache for most of the prompt. If the tools list is the last thing, you could still get 99% cache hits even if it changes every turn.
After a couple of turns the system prompt is a small part of the context. Not changing the system prompt at all is key so that the rest of the history is itself part of the prefix.
Depends upon the service and how the harness is built, Some of the services allow for very few cache keys, so you won't necessarily get any cache if you edit recent messages as the cache is not per message, but big blocks of everything up to a cache key.
This was actually surprising to me when I learned about it as I have never worked with (or built) any cache working like that before.
China also does massive amounts of investment in new technologies and local companies. One of the reasons capitalism concentrates wealth these days is that it’s cheaper to generate enormous capex to prevent paying for labor than to deal with the opex of labor. Ie the road to wealth requires either wealth or investment from the wealthy.
In capitalism, the wealthy provide the capex and reap the profit. In China, the government provides a lot of the capex and claims a lot of the profit (either directly or indirectly). They can then re-invest that, use it to set salaries free from market forces, etc.
The US could do more of that without getting into the contentious issue of raising personal taxes. Ie the government could control the flow of GPUs into the country, asserting that it had the right of first purchase, distribute them to a tightly controlled private company (think power or water company, with margins set by law), and probably make a profit on the cheaper hardware because they’d be a massive purchasing block. We won’t, though, because Google, Amazon, Oracle, etc are currently posed to make a _ton_ of money off doing basically the exact same thing.
> The owner of FoodCo put in a lot of their money upfront to make the company happen in the first place. Your deli guy wouldn't have a job if the FoodCo guy hadn't done what he did.
This is blatantly untrue in the face of even a cursory thought. If the deli didn’t exist, that worker would have done any number of other jobs. Subsistence farming, doing deli stuff out of his house, maybe even community funding a coop grocery store.
That was how things worked at many points in human history. Instead of “private citizen makes $thing and collects rent” it was some variety of “municipality funds $thing for the community”. That just doesn’t extrapolate to a global economy well, for better or worse.
I think there’s a realistic chance consumer inference moves on-device. I think it really depends on marketing.
My non-tech friends and family would probably be served perfectly fine by local models today, if they had a working web search tool. Their queries are often “soft” and don’t have an exact answer. My mom and aunt used it to pick a hairstyle, my mom used it to get an image of what a room would look like with particular drapes in it, etc. Stuff I think mid-sized local models like Gemma or smaller Qwens could do without issue. They just don’t have a device that will run them.
Businesses won’t move. They need a huge context so they can stuff a bunch of Confluence pages in it and 300 tools and it needs to read an entire codebase and yada yada. The hardware depreciation and electricity will probably make it a net zero or even cost more than paying for API access.
My argument is predicated on the assumption that mainstream hardware manufacturers will copy the way Apple and Framework have made system memory usable for inference.
In that world, a) we are already at or close to having enough memory in local devices to do inference locally, and b) that memory isn't inference-specific and can be utilized for other things. Most devices come with enough memory to do some level of inference, and some come with plenty (eg a gaming desktop probably has 32GB+ of RAM in it).
You aren't going to run Kimi on it, but I think the reality for a lot of consumer inference is that it doesn't need to be. It's going to be a lot of things that are soft, and easily answered by a search API, so the LLM really just needs to be able to skim and summarize. Going a step further, we may even see some kind of hybrid approach where a local OpenRouter kind of thing decides whether the task is soft enough to do locally with models that fit in RAM or if it needs to be farmed out to a PaaS provider.
I strongly disagree here. On the technical side, I'm sure it works, I almost never hear about Nix not working.
On the practical side, "learn Nix" is a _massive_ onboarding task. Without Nix, I'd probably pick one up assuming I'll find something to do with it. With Nix, I'd wait until I have a project I know is worth figuring out Nix.
If this were my project, I'd probably go with the absolute most simple answer: multiple SD card readers. Install the base OS on one card, allow hot-swapping the other card, do some mount point stuff with the other card (like maybe it auto-mounts to /usr/local, and have packages install into /usr/local). Or maybe some kind of overlayfs with the other card. SD cards are cheap, and I'd rather glue an SD card holder to the back of a Flipper than learn Nix.
It is just a Linux device. Other people will install NixOS on it anyway, and use specializations if the whole idea of swapping device roles in-and-out is viable. I don't really understand why would the team that already got a full plate decide to also invent a whole new Linux system while they're creating their hardware device.
People that want NixOS should absolutely be free to install it, I just think making it the default makes the system dead on arrival. There are hordes of people using Arduino's editor on esp32 boards because they don't want to learn esp-idf (not a judgement, Arduino works fine enough for most uses).
> I don't really understand why would the team that already got a full plate decide to also invent a whole new Linux system while they're creating their hardware device.
Honestly, I just wouldn't solve that. Nix makes the product way harder to sell, and home-building a solution is either a) an entire product all by itself, or b) shitty, in the "it only works on very specific happy paths" sense.
I also frankly just don't think it's a feature worth as much to consumers as it costs to make. At worst, it's a minor inconvenience to reflash a Pi card. If I'm really lazy, I just disable systemd services for whatever was on it and layer the new stuff on top. It's like 5 commands to get it back to "close enough to fresh".
https://github.com/kstenerud/yoloai/blob/main/runtime/regist... <- `Register` embeds a copy of the code from `IsAvailable` because of the locking; that could be replaced with a private `isAvailable` that has no locking that both use (after doing their own locking)
Just out of curiosity, I enabled some other linters and it looks bad. Excluding test files, there are 110 functions with a cyclomatic complexity over 10 and 7 that are _over 50_. The worst is at 86, which is mind-boggling.
Could probably find more, but you get the drift. I'm sure it runs, but stylistically this is more along the lines of what I would expect an intern to do.
This is also sort of nit-picky, but like half the stuff in https://github.com/kstenerud/yoloai/blob/main/docs/dev/backe... isn't idiosyncratic, it's just the way those things work and a lot of them aren't even tricky. The one linked is particularly blatant; that's not limited to os.Stat that's literally just how permissions work. Denying permission on inodes is a property of the folder, not the file.
Seems like everyone skipped over this part, but optical controls for motorized wheelchairs is a cool idea (at least to me, maybe that's an old idea).
Full VR hasn't done well, but it does continue to make me wonder if there's a market for a stripped and slimmed device. I'd maybe be interested in a device that does optical controls if it fit in regular-sized glasses. I'd be super interested if it had a HUD system (even a super basic one that can only show a handful of symbols). Better still if it had some basic audio, but maintaining the "regular glasses" form factor is more important to me than the HUD or audio.
It's been done for a while - follow the links to who they reference. ie https://www.tolt.tech but it's their integration they've done into the OS is interesting.
Seems like a pretty strong indicator that AR glasses are still being worked on, this definitely feels like one of those features Apple ships to refine before the proper hardware is ready.
I lack that confidence here, because it doesn't appear to really involve AR/VR. Eye tracking is the only feature this really uses afaict. The AR seems like a net negative here, it's just the only device Apple has that has a consistent and remotely convenient view of your eyes.
The device is large and makes the user look weird and non-present, which are net negatives.
The only benefit of the AR is showing the directional arrows, but they could get the same thing with much less weird looking non-prescription glasses with arrows sharpied on them. More realistically, anyone really using for mobility probably develops muscle memory for which direction to look to go where and then they don't even need that. At that point it's just a really expensive, really clunky camera.
I dislike it here because I like Mullvad, but yes, I think it’s fair to go straight to public disclosure.
Someone with likely substantial qualifications put in time to find this. The company is in it for profit (at least partially). What’s fair for the company is fair for the individual. The company can either offer to pay for bugs under the terms they want, hire more security folks to find the bugs themselves, or just accept that researches get to do whatever they want with their findings.
I’d tell Mullvad, but there are companies I don’t respect enough to feel compelled to give them a heads up. Perhaps the author feels that way about Mullvad, it’s entirely within their right to use this to publicly shame Mullvad.
Those who do bug bounties full-time ignore programs with no rewards. Those who want to gain experience or pad their resume can submit reports to programs with no rewards because they are not as competitive as those with rewards.
I tried watching that but X either broke their video controls or disabled them so I can’t skip ahead and the first couple minutes are _slow_.
The whole bug bounty thing is a mess, admittedly, but lacking a bug bounty program entirely feels like immediately losing the moral high ground on “you should have told us first”. There’s a lively debate about what bugs are worth, but it’s objectively not $0 for many classes because a botnet developer will buy them for some amount.
Personally, a big part of my view is formed by the educated assumption that security practices will never improve unless poor security becomes a liability. That’s unlikely to happen with “responsible disclosure” because it gets swept under a rug. Immediate public disclosure changes that risk calculus a lot. I think wed see a lot more downward pressure from vendors to their suppliers if $RandomSaaS had to worry about losing their pants because Oracle had a vuln published.
No software is free from bugs. Category of software that undergo extensive verification like aerospace are priced far higher to accommodate the additional QA. If such extensive verification are added to average consumer or even business software, the massive costs will pass down to average users making it too expensive. Security practices need to improve but I don't think 0-day droppers are the answer. Not every threat actor is at the same skill-level. Immediate public disclosure provides them the opportunity to hit endpoints that they would not have hit coz of low skills.
Software is the only field where people will routinely argue producers can’t be expected to make a product that won’t harm its users and I don’t buy it.
The way your argument reads to me is “software as a category has such little utility that profit margins can only be derived from corner cutting”.
The reality of the landscape is that most companies don’t get hacked as the result of an incredible and novel Spectre-esque attack, it’s something bland and entirely preventable.
SAP got a CVE because they just flat out didn’t implement auth on an endpoint in an app architecture that will execute files just for being in a certain directory, and also didn’t prevent writing files to executable paths (or maybe that’s how the feature works, not a SAP person). For every 0 day with a novel root, there are like a thousand that are some kind of humdrum “didn’t enforce auth/SQL sanitation/XSS/other well known exploit with comprehensive solutions”.
I do think there are good reasons to withhold some classes of exploit. If a hacker writes a 14 page proof on how to beat some encryption we had no idea was vulnerable, that’s one thing. Getting owned for making an insecure architecture and then not even putting auth over it is a whole other issue.
Now that I've thought more about it, I agree with you. Most companies fall prey to well known exploits that are not that expensive to mitigate.
I think it's mostly ship product faster > secure product first that leads to such insecure architecture. Ideally, security should be incorporated early in the software development life cycle but most start-ups rarely hire a security guy in the initial phases. https://www.reddit.com/r/indianstartups/comments/1r6zwbg/why... They expect the software devs to have that knowledge. But security hardening is a skill that takes time to develop so most devs just focus on feature development.
Will immediate public disclosures change the mindset of top leadership regarding security? For some, yes but most will not change because breaches have become too common. They reason if top tech firms like Microsoft or GitHub can suffer breaches and come out on the other side unscathed, they too can survive a major security incident.
reply