I switched from being a penetration tester after nearly 7 years at Security Innovation to a security engineer at 98point6, an exciting healthcare startup. Given the fact, that all of my work in the security industry (nearly 15 years now) has been on the offensive side - I felt that this move would force me to think differently. As in, it would force me to evaluate whether all the recommendations I've given to clients over the years actually made sense - when one thought of them from a defensive perspective. Also, I wanted to learn what it was to work as a product security engineer on the inside of a product company - rather than from the outside, as I'd always done. So I made the leap about a year ago :). And predictably, I've learnt a few things, that I want to jot down here for anyone reading this and thinking about making a similar switch:
- Time = money. The more time you spend doing work that will earn you money, the more likely it is that you will eventually be a profit making company. So at a startup, its been very interesting for me to see that even a simple low risk vulnerability fix is almost instantly mapped to a question along the lines of - "Can we accept this risk?".
If, after evaluating all existing mitigations it's found that the vulnerability is hard to exploit, it's not getting fixed. Well, not immediately anyway. Engineers will much rather spend time creating new features or scaling the environment or improving performance. Because that's going to earn $$, much quicker than fixing a largely theoretical Low-Medium risk vulnerability.
So, in short prepare to do a lot of talking trying to convince people to see why something is a security risk that needs to be fixed at the earliest. Or soon...
- Pulled in multiple directions. If you're joining a company with a large, already existing security team - this is unlikely to happen and your role is likely to be well defined. If not though, and its a smaller team, multiple teams will all want you to chime in and provide security related guidance fairly quickly. All of this has to be done within a reasonable time interval (often
24-48 hours) so there is pressure to give quality answers in a short span of time. Which can be tough.
So, if you are not good at saying no to overly optimistic timelines and learning to work in a sustainably intense manner - you will be burnt out very soon. - Technology overload. In a sense, joining an internal team is good - as you can focus on a single product and learn it well. The problem though, is that you're not doing only software code review where you are reading the code line by line and have 2 months to master every flow - that has taken years to build. Nope. What will instead happen that you will juggle a lot of very different asks at the same time.
So the mental context switching, as you switch between vendor analysis, nodeJS code review, writing modular code to scan AWS infrastructure, provide advice on how to deal with phishing attacks, explaining a security concept to a junior team member, handling a huge ever growing JIRA backlog, creating an entire Wiki from scratch that is often out of date and trying to deal with an incident at 5pm on Friday evening, just as you wanted to log off can be very very exhausting. It's fun for a while as you think you're making giant strides across multiple muscles across the company - but it quickly gets very tiring, and you're not as efficient as you want to be. - Feeling of having achieved nothing. This is just because there is so much to do, and as the product improves there is more stuff to secure. As the team size increases, there is more educating to be done and more code to be reviewed. So however hard you push, however organized and modular you are - there is always something that can be done better. And if you're one of those grumpy security people who see edge cases all the time, you'll always end up feeling that there are a million potential vulnerabilities that should be fixed - but cannot be (justifiably at times) fixed.
So if I look back at what I've done in a year, I'm sure that I've contributed to improving the overall security posture and made it somewhat easier for future security engineers - but there is always something to do, and I can constantly criticize my own work and feel I can do better. - Learning something new all the time. Each organization does its development work differently and this has been no different. It has been quite challenging to learn exactly how dev pipelines are set up and which parts need to be explored more deeply - specially if you don't come from a development background. It's all incredibly interesting, and I'm happy to learn all these new skills - but you have to learn a lot, ask just the right amount of questions and keep things secure. I'm used to this from my red team work but it doesn't make it any less challenging just because there is so much to do - in red team work you learn about 1 thing for 2-3 weeks and throw it away until you need it again.
There's probably many more things that are very different in blue team work but these are some of the main points that I have found challenging and very different. There's more time for sure and I'm happy I made a switch as I could learn a new way of doing things and from multiple new teams - but it's not been a smooth ride.
As long as you embrace the uncertainty of a startup, where you are a major contributor towards setting up security engineering from scratch - a blue team can be fun. It is very different to working on a red team - which is much faster and certainly more glamorous. However, a blue team does give you the feeling that you are creating a much more solid foundation for a team and its product - something a red team does not always give you.
