Operating large language models in-house

At Home

Conclusions

An in-house AI server offers corporate users considerable advantages, especially in terms of data protection and control over confidential information. In addition to improved data sovereignty, an in-house AI server also gives you better performance. Because data processing takes place locally, latency times are minimized, which can be particularly beneficial for computationally intensive tasks, such as image processing or speech analysis. Organizations can also work independently of an Internet connection and services offered by external providers. Another benefit is the ability to customize the hardware and software to meet specific requirements, which can also be more cost effective in the long term.

Open source platforms such as Ollama and the Open WebUI web interface let you provision LLMs efficiently, offering not only more flexibility in terms of model selection, but also the ability to process all data locally without having to rely on third-party services. Moreover, the two tools facilitate integration with existing networks and offer a user-friendly interface that can be customized.

Infos

Ollama: https://ollama.com
Open WebUI: https://openwebui.com
OAuth for Open WebUI: https://docs.openwebui.com/features/sso/
LLMs for Ollama: https://github.com/fmaclen/hollama
Hardware requirements for Llama 3.1: https://llamaimodel.com/requirements/

The Author

Thomas Joos is a freelance IT consultant and has been working in IT for more than 20 years. In addition, he writes hands-on books and papers on Windows and other Microsoft topics. Online you can meet him on http://thomasjoos.spaces.live.com.

« Previous 1 2 3