This is so interesting. How does one go about setting the constraints you see here? Like how do you decide right in the beginning to limit the number of parameters? Or that it's going to run well on smartphone devices? Or that it's going to hallucinate less?
Are these all covered in Karpathy's LLM videos? I haven't seen them, but am growing more curious by the day of how these models are actually built (and seeing that it may not be that inaccessible given there are so many out there now)
This is so interesting. How does one go about setting the constraints you see here? Like how do you decide right in the beginning to limit the number of parameters? Or that it's going to run well on smartphone devices? Or that it's going to hallucinate less?
Are these all covered in Karpathy's LLM videos? I haven't seen them, but am growing more curious by the day of how these models are actually built (and seeing that it may not be that inaccessible given there are so many out there now)
We fine-tuned existing models, so we have an understanding of their performance before hand.
For hallucination- we specifically built our own dataset called SynthQA that focuses on hallucinations in RAG systems.