User's company is exploring building a Retrieval-Augmented Generation (RAG) system for internal documentation and onboarding materials, focusing on data privacy and self-hosting.
My company is exploring building a RAG system for internal company documentation and onboarding materials. One of the main questions that came up is data privacy. Ideally, we don't want to send internal documents to external APIs. Because of that, we're considering self-hosting an LLM instead of using something like OpenAI or Anthropic. Our company is pretty small, we are roughly 12 people. Has anyone implemented a similar setup (RAG + self-hosted LLM) in a company environment? Was it worth the effort in terms of performance, maintenance, and cost? I'd really appreciate hearing about real experiences or lessons learned. Thanks!