@Stuart Mitchell, Founder of security recruiting firm Hampton North, recently posted about the types of candidates he hasn’t placed in his security recruiting career. Unfortunately most Security Engineering roles are not entry level and most hiring managers expect candidates to have experience.
It is even more difficult than that, not only is the expectation that Security Engineers hit the ground running, hiring managers are also looking for Security Engineers that have many capabilities beyond Security Engineering.
The Future of AppSec Engineers
Last year, I talked to Chris Romeo and Robert Hurlbut about The Future of Application Security Engineers and what it will take to succeed in our industry. In my career, I have hired a number of these “futuristic” application security engineers and they have greatly helped scale my AppSec programs and provide the business considerable security value.
Before we dive into the future, let’s talk about the average profile of today’s application security engineers. Most organizations hire application security engineers that have the following skillsets:
- Strong vulnerability knowledge and has the ability to threat model complex functionality
- Deep knowledge in authentication, authorization and logging domains
- Individuals that can perform secure code reviews and run a bug bounty programs
- Folks that can provide guidance on software architecture and how to embed security into it
- People that can review and and integrate security tools into their ecosystem
While those skills are important and “Future Application Security Engineers” require those skills as well, these skills don’t scale. The above skillsets are manual and they are extremely difficult to automate. To match the growth of your Engineering org, you would have to scale by hiring more people, which isn’t an option in many organizations and which is why I focus on hiring “futuristic” security engineers.
The four skillsets that I look for in a Futuristic Application Security Engineers are:
- Application Security
- Software Development
- Influence Skills
- Program Management
Application Security
Clearly for an application security role, you will need individuals that have application security skills, this is a no-brainer, but what type of AppSec skills? What do I look at when hiring? What specific AppSec skills help you scale?
When hiring, I ask a lot of questions in the following areas
- Threat Modeling
- Secure Design
- Bug bounty
- Secure Code Review
- Security Tooling
- (and many more)
Let’s dive a bit deeper into a few of these capabilities.
Threat Modeling
Threat Modeling is a key capability that I expect most candidates to have. When it comes to threat modeling, I want to understand how the candidate reviews a new feature, how they break it down and how they approach the problem.
During the interview, I want them to:
- Ask clarifying questions before they do their assessment
- Systematically go through the feature and point out the major risks
- Provide solutions to those risk and ideally provide multiple solutions and the benefits and detriments for each
- Prioritize their findings and suggest which risks they would fix immediately and which would be ok to fix at a later date
Asking clarifying questions, being systematic, understanding the pros/cons of each solution and prioritizing their findings shows considerable experience and maturity. In real life, you can’t fix all of the things and risks have to be weighed against feature development.
The threat modeling interview provides me some of the best signals for leveling a candidate and let’s me know the breath and depth of their skills.
Bug bounty
I don’t always get a chance to talk about bug bounties, but when I do, I love talking philosophies and thoughts around bug bounties and building communities. Bug bounty discussions help me understand how the candidate thinks about the researcher community and how they motivate the researchers to find more and higher severity vulnerabilities. This is a very difficult task, especially given the fact that good researchers can join any bug bounty program.
Building communities is a unique skillset and it requires a lot of empathy and patience. I have discovered that folks who have built up bug bounty communities tend to do well with building security knowledge within the engineering organization. There are a lot of similarities between those two functions. It is also interesting to understand the problems that candidates run into with researchers and how they have resolved them. It gives a lot of insight into how they may resolve issues internally with engineers.
Security Tooling
I don’t specifically ask questions around Security Tooling, but the topic does come up often when I have an open discussion with the candidate. I want to understand the candidate’s opinion on security tooling and specifically how they have scaled those programs.
Security tooling is easy when you have a small engineering org, but what do you do when you have millions of findings and thousands of developers. How do you remediate classes of vulnerabilities at scale? How do you ensure that the engineering org is fixing vulnerabilities and not ignoring them? How do you use a carrot or a stick?
All these questions are philosophical in nature, but it helps ensure that the candidate’s thinking is aligned to my current organization and that they are thinking about scale.
Software development
Software development is becoming a required skill for all security engineers. It doesn’t matter if you are an AppSec Engineer or a CloudSec Engineer or a Detection Engineer, more and more companies are expecting all Security Engineers to have a software development background.
In the past three years, I have only hired one candidate that didn’t have a software development background, they had other unique skills that were desirable, but every single other individual that I have hired was previously a software developer before they switched to security engineering. I need my security engineers to build complex automations or security features/services, everyone has to be able to code to scale our security engineering program.
Why is it so important these days? As mentioned, engineering teams grow at a much faster pace than security and the only way for Security Engineering to keep up is to build tooling and automations. Within AppSec we need to reduce the manual work and build programs that will scale to any sized organization. Security Engineering teams cannot build programs that scale by hiring more people, we need to build automation to ensure we don’t hire more security engineers or worst case scenario, rely on the Engineering to build solutions for our team.
I do not expect all security engineers to write production-grade code, while that is desirable, it isn’t required. Production grade code should be reliable, performant, maintainable, scaleable, etc. Security Automations will not have the same number of users, nor will it require the same level of uptime. With that in mind, my software interview loops are not as intense as being hired as a Software Engineer.
For those Security Engineers that are able to write production-grade code, I have embedded more and more security engineers into engineering teams to reduce specific risks or build security services. I have seen great results with this collaboration and I have seen other organizations also take this path. Not only do security engineers need to have strong software development skills, they also need to be good enough to deploy code to the production environment.
Strong opinions around software development
I do not believe in hiring entry-level folks for security roles and I have a few reasons as to why. I should preface this, when I talk about entry-level, I am referring to folks that have zero experience within the software development world. I am not talking about software developers making the switch to security engineering.
Unfortunately, most security teams are lean and the business limits their overall investment into security. With limited resources, most security leaders will focus their energy on hiring folks that will be productive from day one. Therefore there is very little time or incentive to hire and train junior engineers.
Entry-level security engineers don’t have software experience, which means that they are a liability when they interact with engineers. They are not able to provide engineers specific remediation guidelines and may not understand the complexity of rolling out changes. On the other hand, entry level folks will trust Engineers because they don’t know where and when to pushback.
The Security Engineering roles that I have opened have not been for entry-level folks and I see that happening more and more in the industry.
Influence skills
In The Application Security Podcast, I talked about Security Engineers being teachers and building the security knowledge and capabilities within an organization. Since that podcast, my views have evolved and not only do I believe that security engineers need to be teachers, they also need to be influencers.
One of the most important skills that a security engineer can have is their ability to communicate with all different types of stakeholders. If you were to talk about a specific vulnerability, you would deliver the information differently to an engineer, to their VP, to the Product Manager, to an exec. Each of those personas requires different information about the vulnerability and an understanding of what is important to that persona and how to communicate and influence them effectively.
How do I gain influence skills?
A quick reminder, influence is about having strong communication skills and you don’t necessarily have to gain it at your workplace. You can practice strong communication skills anywhere.
Get involved with your local security community (OWASP, BSides, ISACA, etc). Go early (or stay late) and network with other folks at the event. Be able to explain to them what you do, learn about what they do. Continue to refine how you speak about yourself and the more you do it, the more comfortable you will become.
Be a speaker at those security events, it will force you to think about messaging and telling a story. Here are all of the things that you have to consider when speaking in front of others:
- Who is your audience? How do you message your content so that your audience understands the subject matter?
- How do you structure your presentation to make sense?
- How do you engage your audience? The right amount of engagement makes the presentation more interesting and the people will remember the content.
- What type of questions will the audience ask?
I would also recommend that you shouldn’t limit yourself to the security community. I felt that I gained a lot of effective communication skills when I started to volunteer my time outside the security community. You will talk to all sorts of people and most of whom are not tech savvy, which is a good thing. Be outside your comfort zone and learn effective communication techniques.
Influence skills assessment during an interview
It is really difficult to test for effective influence skills during an interview, but as stated previously a lot of influence skills goes back to being an effective communicator.
I typically have a role play question where I pretend to be a technical software engineer, but also pretend to have limited knowledge on the security-side. The question will be around a specific vulnerability or a secure design and the candidate will provide their feedback. As a non-security individual, I would ask really basic questions to better understand the security requirements.
Influence within the Security org isn’t just about selling your ideas, it is also about being authentic. Security is synonymous with Trust and if your engineering peers can’t trust you, they will not take your input. Interview candidates should be authentic, honest and transparent. They shouldn’t be afraid to say that they don’t know the answer or when I ask about difficult working situations, be honest and talk about their failures and how they have grown from them. All of us fail, and I personally have learned the most from my failures vs successes. I need to be able to trust that the candidate is genuine and trustworthy, otherwise it might impact the team and our security program.
Program Management
Security teams are stretched thin, typically that means each security engineer runs a full program on their own. Some security teams are lucky to have a (Technical) Program Manager within their org, but only the highest priority project will get their support. Each security engineer needs to have program management skills and needs to be able to take a project from ideation to execution to maintenance mode.
Program Management (PM) skills are more important within the higher seniority levels. At Senior, Staff or Principal levels, these security engineers are expected to deliver projects that are at least six months long, but can be multi-year projects. Projects of that size are complicated and cross-team, which will require PM skills.
For multi-quarter or multi-year long projects, I expect security engineers to:
- Be able to create the necessary documentation to define the project
- Within the documentation, there should be a work breakdown structure, where I can understand the entire scope of work and the tasks/phases within the project
- Clear milestones and delivery dates for those milestones
- Provide risks to a successful delivery of the project and plans to derisk the project
- Create a communication plan, how often will stakeholders get status updates?
- Plan for cross-functional collaboration and how it would work, how often are folks working on the plan meeting?
- Escalation plan, when things are going off track, when do you plan to escalate to the appropriate folks?
I wouldn’t suggest spending 85+% time on the planning phase, but I would recommend that each security engineer invest a good portion of their time to ensure that they properly program managed the project and there is little risk of going off course.
Program Management skills assessment during an interview
This particular skill is very hard to assess during an interview, but I have asked candidates about their largest projects and specifically what went wrong (and what they learned from it). I have also done the reverse where I give a candidate a 3-5 year long (fake) project and they walk me through their thinking about the problem and how to tackle it.
Questions that may be asked during an interview
- Can you walk me through your thought process for planning a multi-quarter project?
- How do you assess and prioritize risks when starting a new project?
- How do you ensure that your multiple stakeholders are aligned with the same goals, especially when there are competing priorities?How do you handle conflict or disagreements, particularly regarding technical decisions for a large project?
- Can you give an example of how you managed project updates and reporting for a cross-org security initiative?
Have I hired Future Security Engineers?
Yes, I have hired many folks in the last three years that fall into the “Future Security Engineer” category. Those individuals have skills in all four domains and the more seniority they have, the more domains I expect them to excel within.
These individuals are great at tackling complex problems at scale. They understood the business objective, but more importantly how the solution needs to look like and how to align stakeholders to execute and deliver on the solution. The projects were multi-quarter or multi-year long and several of them went off-track, but these individuals had influence and program management skills to ensure that deadlines were not greatly missed.
Every Security team is unique and you don’t need a full squad of future security engineers to be successful. As mentioned earlier, I have hired folks that don’t necessarily have all future security engineering capabilities, but they were so talented in other disciplines that they helped move the AppSec program forward. Build the team that your organization needs.
5 Future Security Engineering Takeaways
As Security Engineers, here are some items to think about for the future:
- All future security engineering roles will require software development skills
- To build communication skills, don’t be afraid to branch outside of the tech industry, go volunteer at local centers
- AppSec skills will continue to be required for all application security roles
- Security teams will always be lean, learn how to deploy major cross-org changes
- The only successful security engineer is the one that can adapt and change with the times, these “future” requirements will change too
Podcast – Link
If you are bored or have a lot of time on your hands, watch the podcast below. I have mentioned that my views have evolved, but this is a good video to get a baseline of how I think about the future.