Red to blue team switch

I switched from being a penetration tester after nearly 7 years at Security Innovation to a security engineer at 98point6, an exciting healthcare startup. Given the fact, that all of my work in the security industry (nearly 15 years now) has been on the offensive side - I felt that this move would force me to think differently. As in, it would force me to evaluate whether all the recommendations I've given to clients over the years actually made sense - when one thought of them from a defensive perspective. Also, I wanted to learn what it was to work as a product security engineer on the inside of a product company - rather than from the outside, as I'd always done. So I made the leap about a year ago :). And predictably, I've learnt a few things, that I want to jot down here for anyone reading this and thinking about making a similar switch:

1. Time = money. The more time you spend doing work that will earn you money, the more likely it is that you will eventually be a profit making company. So at a startup, its been very interesting for me to see that even a simple low risk vulnerability fix is almost instantly mapped to a question along the lines of - "Can we accept this risk?".
   
   If, after evaluating all existing mitigations it's found that the vulnerability is hard to exploit, it's not getting fixed. Well, not immediately anyway. Engineers will much rather spend time creating new features or scaling the environment or improving performance. Because that's going to earn $$, much quicker than fixing a largely theoretical Low-Medium risk vulnerability.
   
   So, in short prepare to do a lot of talking trying to convince people to see why something is a security risk that needs to be fixed at the earliest. Or soon...
   
   
2. Pulled in multiple directions. If you're joining a company with a large, already existing security team - this is unlikely to happen and your role is likely to be well defined. If not though, and its a smaller team, multiple teams will all want you to chime in and provide security related guidance fairly quickly. All of this has to be done within a reasonable time interval (often 24-48 hours) so there is pressure to give quality answers in a short span of time. Which can be tough.
   
   So, if you are not good at saying no to overly optimistic timelines and learning to work in a sustainably intense manner - you will be burnt out very soon.
   
   
3. Technology overload. In a sense, joining an internal team is good - as you can focus on a single product and learn it well. The problem though, is that you're not doing only software code review where you are reading the code line by line and have 2 months to master every flow - that has taken years to build. Nope. What will instead happen that you will juggle a lot of very different asks at the same time.
   
   So the mental context switching, as you switch between vendor analysis, nodeJS code review,  writing modular code to scan AWS infrastructure, provide advice on how to deal with phishing attacks, explaining a security concept to a junior team member, handling a huge ever growing JIRA backlog, creating an entire Wiki from scratch that is often out of date and trying to deal with an incident at 5pm on Friday evening, just as you wanted to log off can be very very exhausting. It's fun for a while as you think you're making giant strides across multiple muscles across the company - but it quickly gets very tiring, and you're not as efficient as you want to be.
   
   
4. Feeling of having achieved nothing. This is just because there is so much to do, and as the product improves there is more stuff to secure. As the team size increases, there is more educating to be done and more code to be reviewed. So however hard you push, however organized and modular you are - there is always something that can be done better. And if you're one of those grumpy security people who see edge cases all the time, you'll always end up feeling that there are a million potential vulnerabilities that should be fixed - but cannot be (justifiably at times) fixed.
   
   So if I look back at what I've done in  a year, I'm sure that I've contributed to improving the overall security posture and made it somewhat easier for future security engineers - but there is always something to do, and I can constantly criticize my own work and feel I can do better.
   
   
5. Learning something new all the time. Each organization does its development work differently and this has been no different. It has been quite challenging to learn exactly how dev pipelines are set up and which parts need to be explored more deeply - specially if you don't come from a development background. It's all incredibly interesting, and I'm happy to learn all these new skills - but you have to learn a lot, ask just the right amount of questions and keep things secure. I'm used to this from my red team work but it doesn't make it any less challenging just because there is so much to do - in red team work you learn about 1 thing for 2-3 weeks and throw it away until you need it again.

There's probably many more things that are very different in blue team work but these are some of the main points that I have found challenging and very different. There's more time for sure and I'm happy I made a switch as I could learn a new way of doing things and from multiple new teams - but it's not been a smooth ride.

As long as you embrace the uncertainty of a startup, where you are a major contributor towards setting up security engineering from scratch - a blue team can be fun. It is very different to working on a red team - which is much faster and certainly more glamorous. However, a blue team does give you the feeling that you are creating a much more solid foundation for a team and its product - something a red team does not always give you.

IAM Least Privilege Permissions

There are multiple parts to writing IAM least privilege policies. This document attempts to touch on the best ways to do each of these tasks. The assumption is that you’ve never written an IAM policy before, which is probably false but starting there is probably useful for a newbie developer.

Know what you need

Only you as a dev will know what data your code needs access to. For example: At a minimum you’ll need to know that your new feature plans to read patient records, identify the non-encrypted ones, encrypt them, delete the old records and store the new encrypted records, while maintaining an audit trail. No tool can tell you this, right?

Also, you’ll need to know which AWS service can do all those things. This too is not something any tool can automatically tell you. You need to go Google and read all AWS documentation to understand what tools AWS offers to do the stuff above. So then you get to a point where you say:

• My records are in DynamoDB. I need read access to DynamoDB.
• I need to look at the metadata of each record to see if it's encrypted. So I need read access to all metadata.
• Then I need to encrypt them. I need a key to do this. I need to be able to use an existing KMS key OR be able to create a new one to encrypt my data.
• I need to delete the old records. I need write access to DynamoDB.
• I need to store the new records. I don’t need any more access, as I already got write access in the previous step.
• Oh wait. Not all tables - just Table A :)
• I need to log all my actions somewhere. I need write access to Cloudtrail.

How do I implement the above?

• Now you know you’re going to probably have a StepFunction or a Lambda function where that code will live. Which means there’s an execution role that comes into play. It’s a good idea to use a new role for each function coz:
  
• It’s cleaner and you know exactly what that function and only that function needs
  
• It’s easier to change and not worry what else will break in some other function that shares that role
  
• Security will whine less as you’ll probably follow least privilege when you assign permissions
  
• Okay good. You create a new execution role and now need to assign permissions. You need to map all those Read/Write statements that made sense intuitively into an IAM policy, which is what controls access to all the roles.
  
• You add a lot of permissions, see that it works, dial down a bit, see it still works, dial down some more and see it break and repeat this increasingly painful and irritating exercise till everything works. Of course then security complains that it's not least privilege, gives you a link to what’s expected and rejects the request. The link’s somewhat helpful but not a lot, and you end up repeating all that crap again till they’re happy.
  
• You get better at this over time but at one point feel you’re wasting a lot of time tweaking policies, when you should be writing code instead. And then wonder if there is a better way to write IAM policies.
  

Some ideas to get better

• Definitely use a PolicyGenerator. Don’t write a policy by hand.
  
• AWS Console: You need to know exactly what you want. If you do, it works great
  
• AWS PolicyGen: Older version of Console but with a simpler interface. Not sure its maintained
  
• PolicySentry: This looks like a fantastic tool that does a lot of the policy generation for you. It is worth spending the time learning all its detailed options.
  
• If you know which resource you want to control access to, use a CRUD template
  
• If you know which actions you need, use an Actions template
  
• To clean up policies generated by PolicySentry, remember what you really need and refer to the AWS docs. You’ll need to do this as you almost certainly do not need everything PolicySentry gives you. Pay heed to the Dependent Actions column as well and don’t forget to grant access to those actions. Here is a sample table for CloudTrail.
  
• Create as many templates as possible for services that you often use and make them easily reusable or to reference, across your engineering team.
  
• Use a linter such as Parliament to check your policy once it’s generated for obvious typos as well as some security misconfigurations. Integrate it into your pipeline.
  
• Use the AWS policy simulator to verify if your execution role has and doesn’t have access to the expected resources. Think of this as a confirmation of your existing policies. The good thing with this is it doesn’t make any changes to your actual stack.

Examples for tools above:

Policy Sentry templates

CRUD:

policy_sentry create-template --name TestRole --output-file crud.yml --template-type crud

Edit the YML template that gets generated, tweaking it to remove whatever you don't want.

Policy Sentry query commands

policy_sentry query action-table --service cloudwatch --wildcard-only

policy_sentry query arn-table --service dynamodb

policy_sentry query condition-table --service lambda

Parliament sample bad policies

Blank resource field where a "*" or an ARN is needed:

Mismatched condition where the condition is not valid for the chosen action: