AI and threat modeling for secure architecture

What is prompt engineering?

Prompt engineering is putting prompts into the model, either in order to get an output from it, or it can also be part of your training, so you can train the model as the prompt. But essentially, if you're using an LLM or a chat bot, think that the prompt is what you're entering into the command line interface in order for it to give you a response.

What role does this play for building secure architecture? 

There are several potential roles for using AI. We will explore three of them in particular below; using LLM as the training model, prompt variations, and blank page syndrome:

LLM Training Model 

Firstly, the part that's about training the model itself. So making sure that the model is trained properly in order to give the responses that would be helpful. 

Use case: Say you need reporting for architects to use, you'd want to do the prompt engineering to train it, to behave or to respond accurately. And then, importantly, how the engineers that are interacting or the analysts are interacting with the system. What they're going to ask. So it'd be questions that they could ask that would help them with security architecture. It could be anything from asking them about threat modeling. Could you help me do a threat model, or I've done a threat model? Can you review what the prompt could be. Here's the threat model I've completed. Could you review it? And did I miss anything? Are there aspects that I might, or questions that weren't asked. That would be one way to be used.

Prompt Variation

A piece of prompt engineering which is absolutely crucial is not just using the model and putting in a prompt, but varying the prompts and seeing what happens as you do so.

That's the engineering part of prompt engineering. As otherwise the outputs can be so convincing, that the user may forget to further engineer or tweak the outputs and responses.

Use case: Place the same prompt repeatedly into the same engine and look at how it varies the output. If you evolve your prompt. Does it get smarter at answering your question? If you go from engine to engine, does it get better?

Blank Page Syndrome

Arguably it is easier to edit than it is to write. And so if an individual is struggling with this blank page syndrome, using prompt engineering to get some form of response, even if it isn’t very good, can be adapted and improved further with this engineering approach to AI prompting. 

Use case: Begin with your question or query, get the initial response, and tailor it further. Following a similar style to the Prompt Variation recommendations above, will continue with the potential outcomes and information you may be looking for. 

Breaking down bias and building better prompts

Any of us can start from a point of bias. Even in a team brainstorming workshops this can happen accidentally, and then edit further but still retaining that point of bias for that product you are discussing. It can be difficult to break out of that myopic view. The same can be said when doing prompt engineering, and getting biased output back out from the LLM. 

Repetition is crucial

As stated earlier, try the same prompts more than once, don’t pass at the first attempt. The LLM may then give you differing answers for the same thing. See if you can ask the same question or query but in a different way. Otherwise, if you ask it once, to generate 10 ways or ten ideas to write a specific code, you could possibly get back the same error or bias in ten different ways.

Human collaboration 

One thing that can help on the human side is to not just be going at this alone. How do we eliminate bias in real life? We ask each other, we brainstorm and consider each other's views and ideas. You can also brainstorm with each other in terms of prompts. People will think of different prompts which can help quite a lot.

Recency bias

This is real, and can be based upon the most recent information you placed into your prompt. The context, type and amount of data you add into your prompt can change the output and its bias. Of course, it is no guarantee you will be able to get a useful architectural response from that LLM. But always base the context on what you feel comfortable giving it. And incorporate content into each prompt where possible. 

Share results 

Within your company, you could have five, 10 or even 20 people doing similar prompts. But the temperature of the response is changing. How do you determine if you’ve done a good job, or if you’ve captured the best prompts or responses from your LLM? LLMs are inconsistent, which most people are relatively accepting of, however, sharing best practice, or as close to that as possible, can support with reducing bias, and sharing results from the best, refined engineered prompts. 

Incorporate prompt chaining 

You can have a really big prompt which is confusing the model because there are a lot of things, a lot of commands, orders, and information within the middle of the prompt. So one thing that you can do is to prompt one thing, wait for the answer, and then prompt another thing. This is

because the second prompt can confuse the LLM. If you give a really large prompt instead of chaining the prompts, it can confuse the first request. 

Using prompt engineering for threat modeling 

A way to start or progress your threat modeling journey 

You can use AI prompt engineering to aid you in identifying different ways of architecting what you’re working on. For example, ‘I am thinking of building product X that achieves business case Y, tell me several different ways I can architect this.’ You can then look at the security implications of those outputs, it can allow you to narrow down your choices and iterate faster. 

One big question organizations ask is ‘how do we scale through threat modeling?’ This is usually due to limited people with security expertise. Using LLMs as part of building a threat model can help, by asking the right questions, fine tuning the prompts and responses, which helps to get ideas flowing, and to get you started. Doing a complete threat model isn’t possible with an LLM, but it can aid at the beginning. 

And if you are brand new to threat modeling, using LLMs to have a starting point, and the context, it allows someone like a developer to take that output from the LLM to the security expert who has the knowledge and critical thinking to guide the threat model process further. 

Create prompt templates 

The more we interact with LLMs, the more we mature in our prompts. So, if you can create templates where a certain engineer may have worked out a great basis, with the kind of information back that is wanted, it is going to help others within the organization. In some companies this is hard and takes time to get great prompt engineering skills, but getting a template from it is beneficial to everyone longer term. 

Is there an ideal framework for AI prompt engineering? 

You may wonder if there is a specific framework for prompt engineering separate from other best practice, guidance or other pre-existing frameworks.

We see huge value in the aforementioned prompt templates to identify positive patterns that work for what you need. It is a great way to record what problem you are looking to solve, what sort of information you may want back, or to even act like a specific role within your prompt - such as thinking like a threat model expert. Of course the overarching cybersecurity frameworks and use of AI should be referenced as normal, alongside use of artificial intelligence. As use of LLMs continues to evolve, more guidance and frameworks may begin to enter the market. We may even see prompt engineering evolve too. We need code that tests code that talks to LLMs!

What next for AI and threat modeling?

You can take a look at this eBook that may be helpful, ‘Threat Modeling the AI Pipeline’. Or even visit all of our resources that relate to artificial intelligence and machine learning in our blog here. We even have a playlist on our YouTube Channel, for all things AI and threat modeling. 

In addition, ProtectAI has excellent information and research that covers MLSecOps, prompt injections and much more if you would like to consider further reading. Finally, for threat modeling specifically, we highly recommend Adam Shostack’s book, Threat Modeling: Designing for Security.