News on model binarization (well actually -1,0,1).

News on model binarization (well actually -1,0,1). If briefly:
- the paper
https://huggingface.co/papers/2402.17764
- they say the transformer itself gives the same quality (wow)
- obviously a whole playground for hardware optimization for these needs (Nvidia is gonna go down by the end of the year)

Why am I writing about all this. Understanding LLMs and other generative models and keeping an eye on where everything’s moving will let you better understand use cases and which devices and hardware will win from all this движ. It’s obvious this year will be the year of optimizations and by the end of the year pretty much everything will run on device in one quality or another, which of course opens up all kinds of opportunities. But you can’t forget:
- all local models will most likely work only on the latest device models
- it still won’t be GPT-4 level quality, but quick evaluation and summaries right inside the keyboard will be there
- it’s still bad with facts and personal data (vector search loses way too much information along the way — though there are some tricks there)

I’m sure the window of opportunity to jump into this world will close by the end of the year. It’s the perfect time to do something :)