Most organisations have people who want to use generative AI on real work: contracts, customer cases, product documentation, financial figures. The question that stops many initiatives is not whether the technology works. It is where the data goes when someone types a prompt, who can see it afterwards, and whether it ends up training someone else’s model.
Private AI is the umbrella term for approaches that let you use AI while keeping control of that data. It is not one product or one architecture but a spectrum of options, each with a different balance of capability, cost, operational effort and control. Choosing well means matching the option to the use case, rather than picking one model for everything.
The spectrum of options
It helps to think of four broad options, ordered from least to most control over where data is processed.
- Public AI tools: consumer or freely available chat services used through a browser. They are capable and quick to try, but terms differ by service and plan, and depending on the settings, inputs may be retained or used to improve the provider’s models. Without an agreement and admin controls, you have little say over what happens to the data.
- Enterprise AI within your existing tenant: assistants built into platforms you already use, under your existing enterprise agreement. Microsoft 365 Copilot is the best-known example. Microsoft states that it operates within the Microsoft 365 service boundary, honours the access controls already in place, and that prompts, responses and data accessed through Microsoft Graph are not used to train foundation models. For EU users, Microsoft also commits to keeping traffic within its EU Data Boundary.
- EU-hosted and sovereign cloud models: language models consumed as a service from a cloud provider and deployed in European regions, with contractual commitments on data residency and use. Some providers also offer sovereign set-ups with extra controls over operations and administrative access. You call the model from your own applications, so you design the data flows yourself.
- Private deployment of open-weight models: models whose weights can be downloaded and run on your own servers, in your data centre or a private cloud. Data never leaves infrastructure you control. In exchange, you take on hardware, operations, security and model updates, and you need to check each model’s licence terms, which vary.
The trade-offs to weigh
No option wins on every dimension. The useful conversation is about which trade-offs you can accept for a given use case.
- Capability: the largest commercial models are generally strongest on complex reasoning and broad knowledge. Open-weight models are often good enough for well-defined tasks such as summarisation, classification and answering questions from your own documents, especially when the task is narrow and the retrieval behind it is good.
- Cost: services are paid per user or per use, which is easy to start with but grows with volume. Private deployments need upfront investment in GPU capacity and skilled people, and pay off mainly with steady, substantial usage or a firm requirement for control.
- Operations: with a service, the provider patches, scales and upgrades the model. With a private deployment, that is your job, including monitoring, capacity planning and replacing models as better ones appear.
- Security: keeping data inside your network removes a class of risk, but a self-hosted model is only as secure as the platform around it. Identity, access control, logging, network segmentation and patching still matter.
- Compliance: GDPR, sector rules and customer contracts may limit where certain data can be processed. Some data may only be acceptable on infrastructure you control, while much of it will be fine in an enterprise service with the right agreement.
When each option makes sense
Most organisations end up with a mix. Public tools suit low-risk tasks with no confidential content, such as drafting generic text or brainstorming, provided there is an acceptable-use policy that staff understand. Enterprise AI within your tenant is often the natural default for everyday productivity, because it builds on identity, permissions and compliance controls you already run.
EU-hosted models fit custom applications where you need more design freedom than an off-the-shelf assistant offers, but do not want to run infrastructure. A private deployment makes sense for the most sensitive material, such as trade secrets, health data or defence-related information, for workloads where contracts rule out third-party processing, and for high, predictable volumes where owning the capacity costs less over time. Running models on your own hardware also ties in with decisions about your data centre and hybrid infrastructure, so involve the infrastructure team early.
How retrieval over company data works
Most useful business applications do not rely on what a model learned in training. They use retrieval-augmented generation (RAG): when a user asks a question, the system searches your own documents, passes the most relevant passages to the model, and the model writes an answer based on them, ideally with references to the sources. The model itself is not retrained on your data, and you can update the content without touching the model.
The hard part is permissions. A retrieval system that indexes everything and returns anything becomes a very efficient way of leaking information. A few principles keep it under control:
- Carry permissions into the index: store each document’s access rights alongside its content, and keep them in step with the source system when they change.
- Filter by identity at query time: the search runs as the user, so passages they cannot open in the source system never reach the model.
- Clean up before you index: if file shares and sites are overshared, the AI will expose that faster. Fix the permissions first.
- Separate sensitive collections: HR cases or board material may warrant a separate index with tighter controls, or no indexing at all.
- Log and review: record what was asked, what was retrieved and what was answered, so you can investigate issues and improve quality.
These principles apply whichever option you choose. Enterprise assistants such as Microsoft 365 Copilot handle permission trimming for content in their own platform, while custom and private solutions need it designed in from the start.
Questions to settle before you choose
- What data will the use case touch, and how is it classified?
- Which legal, contractual or customer requirements limit where that data may be processed?
- How much model capability does the task really need?
- What volume do you expect, and how predictable is it?
- Who will run, monitor and update the solution once it is live?
How Altechy can help
Altechy helps you choose the right option for each use case and then build it. Through Generative & Agentic AI we deliver assistants and knowledge search on your own data, either with ready-to-use tools or as a customised version running inside your own infrastructure, and our AI Assistant Pilot puts a working solution in the hands of real users. If you first need to decide where AI pays off and which data may go where, the AI Value Assessment within AI Strategy & Governance gives you a prioritised shortlist with a recommended deployment approach per use case.
We bring in the right specialists from our partner network, from infrastructure to data protection, and remain your single point of contact. To talk through your options, book a free 60-minute idea session.