Anthropomorphising machine learning models has long been a trend that creates confusion and misunderstanding. A recent article on the BBC news website describes Generative AI (GenAI) as “taking in information from its surroundings, and deciding the best response based on what it learns.” This sounds like they are talking about a human, not a computer algorithm that isn’t at all “aware” of the outside world.
However, when it comes to your security plans, considering your GenAI solution as a human actor can be a productive thought experiment. It leads to questions like: if our GenAI solution were a member of staff, which level of security clearance should they have?
In most organisations there are multiple levels of security clearance, to control access to both data and information, as well as permissions for carrying out certain actions. These controls might be in place to protect customer privacy, prevent fraud, respect regulations or safeguard commercially sensitive information, to name a few.
While most organisations are not building their own GenAI solutions and hence have zero control over the core training data, many are using or considering Retrieval Augmented Generation (RAG). RAG is a technique where more specific and timely information can be provided to a GenAI solution alongside a specific prompt. The goal is to generate answers that avoid out of date information and that are enriched with specific data related to the problem at hand.
So, when building a customer-facing GenAI solution, organisations might be tempted to give it unrestricted access to as much information as possible. This is something we would never allow for a human, so is it safe to do this for GenAI? In the following, I illustrate the problems of GenAI security through some real-world examples and hope to present some actionable solutions.
Let’s start by imagining a fictitious bank looking to use GenAI to enhance customer service and to provide support for internal colleagues – and decide how much power we should give to our automated solution.
Customer support chatbot
The first initiative the bank wants to try is a very common one: using GenAI to provide an enhanced response to online enquiries, the ubiquitous chatbot. Our bank plans to make use of RAG by using existing public documents (such as Terms and Conditions, up-to-date interest rate, etc.) to help the GenAI solution create more helpful and relevant answers specific to a potential customer’s needs.
If these documents are all public and available on the bank’s website without authentication there’s little problem with security here: we are providing no more information than what is already open to everyone. So, in this instance, it initially seems like our GenAI solution would need no higher security clearance than a member of the public.
However, since the chatbot might end up interacting with one of the bank’s existing customers, and not just a member of the public with a general query, there is a risk that the customer themselves might start disclosing sensitive information. This can have regulatory and privacy implications, especially if the GenAI is hosted outside of the bank. Thus, we’ll need a way for our chatbot to identify when this happens and ensure customer privacy is respected, perhaps by stopping the chat and redirecting a customer to a more secure channel.
Enhancing our chatbot
What if we wanted to allow our chatbot to support existing customers? Perhaps by tailoring the response to, for example, provide information on their recent transactions, account balance and so on? This could again be done using RAG, however alongside our public documents we’d also pass along some customer-specific information obtained by querying the bank’s internal systems.
Obviously, this “version” of the chatbot should only be available to authenticated customers. Clearly, at this stage we also need to increase the security clearance of our GenAI solution, but by exactly how much depends on the details of authentication.
When dealing with sensitive information, we must always authenticate the identity of a customer before sharing information – hence, customers need to log in with their credentials before seeing bank account details. We might also require additional authentication steps when a user transitions from using the bank’s on-line solution to, say, interacting with a member of staff directly. In this case, staff might ask some additional security questions before any interactions take place.
Assuming we find a way to replicate the above steps, when our GenAI assistant interacts with an authenticated customer it will now effectively needs some of the same security clearance as human members of the customer support team who have access to some amount of client information. In the above example, it would be “read-only” access to the account without the ability to make changes or act on behalf of the customer.
Real world training
At this stage, it’s probably worth pausing and reflecting on the training that human customer assistants at the bank receive, especially around security and protecting customers’ information. If we plan to give the GenAI assistant the same level of clearance, does it need to undergo the same training?
If so, we could include some of the training documents as part of our RAG repository. We might also need additional development effort to vet the GenAI responses to ensure they meet the expectations we have for dealing with clients, for example not commenting on spending habits or disclosing more information than is needed to answer a customer’s question. This could be done through prompt engineering which I return to below.
Clearly, when we increase the security clearance of our GenAI chatbot solution we must consider what other needs and responsibilities this creates, and looking at how we deal with this for human members of staff is a useful starting point.
Internal summarisation assistant
What about using GenAI to support business operations? At our made-up bank, some members of staff spend a lot of time reading and summarising data for leadership and regulators. Reporting on and summarising large amounts of information are challenges for any organisation, and GenAI can definitely be of great help here (at Softwire we’ve built several solutions to do exactly this).
Let’s assume that the bank already has a repository of documents and data available in a “shared drive”. We could just give the GenAI solution access to all these documents to use during RAG. In fact, several organisations advocate exactly this approach: just plumb GenAI via RAG into your existing document repositories.
However, when we ask the question “what level of security clearance did we just give our GenAI solution”, we can see the obvious flaw in just opening access to all documents. Very few normal members of staff within a bank have access to every single document and piece of data, and in fact in many cases regulations prevent this. Giving GenAI these access rights would give it a very high-level of security clearance, likely much more so than any human employee.
One way to address this challenge might be via prompt engineering.
Prompt engineering
When we use GenAI solutions, a “hidden” prompt is sent before the actual end user’s query. Put very simply, this is done to direct the way the GenAI solution interacts with the end user. For example, a prompt might include instructions to “please be polite, avoid politics and don’t suggest financially irresponsible actions”. There is a lot more to prompt engineering, and a recent article by my colleague Ant Pengelly shines some light on how to write good prompts.
When using RAG with GenAI, we provide all relevant documents, the prompt and finally the user’s input. So, could we construct a prompt that asks the model to respect the security clearance of the current user and not to provide any document or data they are not authorised to see? In this scenario, the GenAI model has the highest possible clearance (access to all documents) and we rely on it to only provide data and information the end user should be able to see. A bit like a librarian who could potentially read every book in the library but can only loan out books that the customer is “cleared” to read.
Prompt attacks
Granting GenAI this kind of “super clearance” should already raise alarm bells, but when we also consider prompt attacks, this scenario looks even more dangerous. Prompt attacks attempt to subvert the prompt provided to the GenAI solution and “trick” it into stepping outside of the intended bounds.
These attacks can range from amusing to disturbing, but in this case, there is an obvious security risk: it might be possible, using a prompt attack, to “persuade” the model to provide information I am not authorised to see. In our library analogy, this might be someone using social engineering to trick the librarian into giving them a book they are not allowed to read (social engineering uses manipulation to persuade someone to carry out actions or disclose information they shouldn’t).
In practice, we hope no one would build a real solution that simply uses RAG with open access to every single document in an organisation. However, this scenario does highlight the difficulty in creating a single solution that can be equally helpful for people with different levels of security clearance.
Compartmentalisation
One solution to the above challenge could involve creating different “repositories” of knowledge grouped together by the same security classification. RAG-based solutions are suited to this by design, with their information often coming from specialised data stores (i.e. a vector database) that have been pre-populated with chosen documents, as this allows faster and more accurate retrieval for large amounts of data.
Given this, we could create different repositories for our different levels of security clearance. Care is needed here, for example, if one of these repositories is pre-filled with “top secret” information we’d need the appropriate access controls and protection in place for “human’ access, even before we start using it as part our RAG process.
Once we have specific data repositories in place, the GenAI solution could pull information from those as appropriate based on the current user’s security clearance.
Whichever approach is used care is needed to ensure the right store is used for RAG, as well as ensuring security classifications remain up to date as source documents change or get deleted.
Customer chatbot upselling
As a final scenario for our bank, let’s assume we want to enhance our GenAI chatbot solution to suggest some additional products and services our customers might want to consider after the initial query has been dealt with. For human support staff, there would be additional training documents that provide guidance on this, including ensuring they do “up-sell” in a way that meets regulatory guidance. These documents likely contain some commercially sensitive information including prices and discounts.
To enable GenAI to up-sell, the simplest solution might be to include these additional documents when doing RAG alongside instructions in our prompt not to disclose anything sensitive. However, given what we know about prompt engineering we still risk that elements of these commercially sensitive documents might be disclosed.
Does the same risk of disclosure exist with a human assistant? Well, yes, and again this could happen via social engineering. However, right now it is still probably easier for a human to spot someone attempting to trick them into disclosing information than this would be for a GenAI based solution.
It’s easy to get this wrong, but it’s not hard to get it right
The current generation of GenAI solutions can bring real benefits, but like any new technology they need to fit into an existing set of security controls and constraints.
Special care is needed when using RAG, with attention paid to the sensitivity of source documents and data. At our fictitious bank, a good solution would be to create specific levels of clearance for the various GenAI solutions as just copying existing ones for humans (e.g. ones who can act on behalf of customers) might not be a good fit and could add unnecessary security risks. Allowing the GenAI solution “super user” access to everything clearly isn’t at all safe.
I’ve highlighted just one dimension of GenAI security here (privacy and data protection), but hopefully this is enough to show that you can’t just drop these solutions into your IT estate and throw open the doors to everything. Care is needed with at least a full review of the security implications, including threat modelling.
In addition to the more novel security implications of GenAI solutions it’s also worth remembering that all the usual good practices around security and regulatory compliance still apply. Specifically, when using GenAI we are likely making use of 3rd party APIs and solutions, often hosted externally and maybe even in different countries.
So, what lessons to take away from all this? While it’s easy to make mistakes if you rush into things, carefully thinking about the privacy and security implications of your GenAI setup will allow you to build valuable, safe and trusted customer experiences.
