Introducing Refact Code LLM: 1.6B State-of-the-Artwork LLM for Code that Reaches 32% HumanEval
September 4, 2023
by Sergey Vakhreev, Oleg Klimov
Right now we’re introducing Refact LLM: 1.6B code mannequin with infill real-time code completion (together with fill-in-the-middle(FIM) functionality) and chat.
Refact LLM achieves the state-of-the-art efficiency among the many code LLMs, coming nearer to HumanEval as Starcoder, being 10x smaller in dimension, and it beats different code fashions equivalent to StableCode, CodeGen and ReplitCode on HumanEval metric.
Abstract:
- 1.6b parameters
- 20 programming languages
- 4096 tokens context
- code completion and chat capabilities
- SoTA on HumanEval benchmark amongst comparable code fashions
- pre-trained on permissive licensed code and obtainable for industrial use
| Mannequin | Mannequin Measurement | HumanEval cross@1 |
|---|---|---|
| DeciCoder-1b | 1b | 19.1% |
| Refact-1.6-fim | 1.6b | 32.0% |
| StableCode | 3b | 20.2% |
| ReplitCode v1 | 3b | 21.9% |
| CodeGen2.5-multi | 7b | 28.4% |
| CodeLlama | 7b | 33.5% |
| StarCoder | 15b | 33.6% |
The bottom mannequin was skilled on our personal set of code with permissive licenses solely and open textual content datasets (the textual content to code ratio was 50:50). In complete, we skilled our base mannequin on 1.2T tokens of code on our cluster.
The mannequin was then fine-tuned with open code instruction-following datasets filtered for high quality and an artificial dataset primarily based on The Stack dedup v1.1 to enhance FIM and boosting the bottom mannequin efficiency.
You may learn extra concerning the structure selections that we made within the blog post.
We intention for the mannequin to be accessible to everybody, we’re releasing the mannequin for industrial use underneath BigScience OpenRAIL-M license and making the burden obtainable on HuggingFace.
Whereas the pattern just lately was for the mannequin sizes to get greater, we needed to decrease boundaries to entry and make it a flexible software for builders with various {hardware} setups. With the smaller dimension, operating the mannequin is far quicker and reasonably priced than ever: the mannequin may be served on most of all trendy GPUs requiring simply 3Gb RAM and works nice for real-time code completion duties.
Refact LLM may be simply built-in into current builders workflows with an open-source docker container and VS Code and JetBrains plugins. With Refact’s intuitive person interface, builders can make the most of the mannequin simply for quite a lot of coding duties. Finetune is offered within the self-hosting (docker) and Enterprise variations, making ideas extra related in your personal codebase.
Refact 1.6B LLM is the third mannequin within the household of our code fashions, with CodeContrast 3b and CodeContrast 0.3b launched beforehand. We intention to proceed with our analysis and future updates to enhance the LLM’s efficiency and capabilities. We’d like to get neighborhood contributions and suggestions to reinforce the mannequin additional. For any questions and concepts, please go to our Discord.