Assess
Docker has released Model Runner, a CLI tool that standardizes local inference of AI models. (Any maybe also for production in the future)
With a single docker model run <<modelname>> "Hi" command, developers can launch a model as simple as launching a container.
Simelar to Ollama, it uses llama.cpp
Model Runner is ideal for:
- Local testing before deployment.
- Working with OpenAI-compatible API.