Did we do a good job in our threat modeling program?
Congratulations, you have gone through the process of an entire threat modeling program! But, the work is not yet over. Threat modeling programs should be iterative and evolve over time. Assessing the results is crucial, or else how will you know if you did a good job? Did the program deliver against the objectives? What data do you have to hand that you can assess and learn from for next time?
This blog takes you through multiple aspects of the results from your program, and if you desire you can also watch the full episode about this topic below:
What can be signals of success?
Results tell us where we are and where we are likely headed. We will always be adapting, changing, and improving our programs. Whether it’s training, support, tooling, process, reporting we will have to keep up with the business’s speed of delivery.
This area is important because if you didn't set out with clear objectives, it will be difficult to decide if the program achieved the goals you set out to accomplish. But when you set up good foundations at the beginning of your program, and know what the outcomes need to be, the process begins of measuring the success against those. In addition, beyond your teams who were involved, leadership is likely to ask you, "How well is this program performing, and should we continue investing in it?" Measuring the success of your threat modeling program not only demonstrates its value but also helps refine and evolve its impact over time.
These metrics can start off smaller and more attainable, it is better to measure a few areas very well, than many done poorly. Perhaps you are wondering what metrics in particular matter most - here are some areas to consider:
Adoption Across Teams - Are business units engaging? If so which departments (development, security, architecture teams)
Outputs Driving Decisions - Have threat model outputs driven changes in architecture, security controls or something else to make a positive difference? This can be within any department.
Improved Security Posture - Successful programs can result in security changes that become more effective for the product and the organization.
Reduction of Rework - Rework should certainly decrease compared to when threat modeling was not in place. Fewer last minute changes or late-stage findings can be supporting evidence of this benefit.
Repeatability of Process - A successful program enables repeatable processes making it easier for teams to model consistently and effectively.
Growth in Knowledge and Collaboration - Effective programs should encourage knowledge-sharing and collaboration across teams. Aiding improved communication and an evolved manner of working together.
Using data to drive decisions
Program and modeling measurements provide a lot of data, too much most of the time. Find ways to narrow down your data to what is important for you and your objectives. Only collect what you need.
Threat modeling generates a wealth of data, but it’s crucial to focus on what aligns with your objectives. Narrow down your data collection to what’s necessary and impactful to you, your organization and your unique objectives. Here are some considerations.
Some key data points within your threat models are:
- Identified threats and associated countermeasures/controls
- Architectural designs and product components
- Network integrations and third-party dependencies
- Data flows and classifications
- Application purposes and user types, including permission levels
Program-level data may include a combination of the following:
- Number of models created
- Applications and business lines represented
- Findings and remediation efforts
- Participation by teams and individuals (e.g., developers, security, architects)
- Threat landscape trends
- Training metrics and resource costs
- Adoption rates of modeling tools and processes
Utilizing reports for maximum impact
Reports that centralize and visualize data like the above, help stakeholders at varying levels to understand progress and make informed decisions. Ensure any reports use clear but critical insights in a format that is easy to understand and interpret.
Key principles for impactful reporting include:
Centralization: Ensure reports are easily accessible and shareable.
Clarity: Focus on concise, valuable data that is relevant to your objectives and organization.
Actionable Insights: Showcase adoption rates, actions taken, and overall program success.
Conclusion
Threat modeling programs must continuously iterate and this can only happen if you review your data, take informed decisions, and periodically improve. By focusing on measurable, actionable insights, you can demonstrate the program’s value, deliver ongoing improvements, and align it closely with broader organizational goals. Programs need to adapt for growth and scale over time, making repeatability and consistency crucial for now and into the future. This evolutionary approach not only secures your systems but also embeds security as a natural part of your organizational culture.
