Respect for developers

At work, we tend to spend time that we’re not on projects however we want. Ideally, however meaning - technically however – not being on Facebook 8 hours a day. But jokes apart, usually someone picks up a skill that he/she isn’t familiar with and tries to learn the same when not on projects. Since most of my professional life in information security has been to break things, I’ve always been curious on how things are on the other side – to build things. What challenges does a developer face? Why is there still such a lot of truly awful insecure code still out there? This despite there being tons of resources to learn from? After all, as a hacker that’s how I’ve learnt all my life – don’t know how to hack a new technology? Go read. Learn. Master. Hack. Why should it be any different from developers?

And so, I decided to fix a ton of bugs in Whetstone, our internal appraisal system which our boss wrote while he was on paternity leave. He takes his vacations seriously, as you can see. Whetstone was written in Ruby on Rails – which is one of my favorite languages for web development – just because of how easy it is to get started. So I go, ooh nice – how hard can it be? And that’s where it all began going wrong…

So obviously, I can’t gut the whole thing and start re-writing the entire system. I have to fix bugs that currently exist. I launch GitHub and see 78 existing bugs. So now I have to prioritize which ones to fix first. I start the process. I pick a simple one to start off with and quickly fix it. I’m excited. I want my change to be visible immediately. Oh but wait, that’s not how it goes. Apparently, git has something called branches that I first need to learn about – so as to not mess with other people pushing code back as well. That puts and end to my coding for the present, and I spend 2 hours reading about how Git handles versions and branches and why its better than SVN and so on. Then I push my code and see it appear on Github. But then someone has to “pull” my changes or accept them. And clearly, while I have permission – it makes no sense to approve my own code – that defeats the purpose. And boss is on leave. No push. No warm fuzzy feeling of first ever fix. Process problems.

Anyway, I’m now feeling better and pick up a few moderately complex issues. It seems I know how to fix it. Code. Compile. Fail. WTF. Google. Stack Overflow. Fail. As it turns out, I’ve upgraded Rails to 4.3 or something and all the fixes on the Internet which have up-votes on Stack Overflow all fail. Somehow after a lot of trial and error I find a fix, but it has taken way way longer than I’d ever expect it to take. And note, this is a simple internal application. If I fixed bugs at that speed in production, as a developer – I’d probably be fired in a week. Skill problems.

Then, I get put on a billable project and forget all about this for a bit. Boss comes back, my 4 changes are accepted. Am I having fun, he asks? Yes yes of course I say, drunk with the success of my 4 massive quashed bugs. Okay, have fun – fix some more says boss. So 2 months later, I pick it up again. So let’s start now. Er, why did I write this code this way? F***, it was so long ago – I should have commented. Lets look at some old code, maybe that explains it. Er no, no comments there either. Just some fancy one-liner SQL looking query. Ha, I’ll just comment the old code then and write new fresh code! That’ll fix it. For sure…… 2 hours later. Undo. Undo. Undo. Okay that broke everything :| and clearly I can’t code. I suck.

And now, after all the undoing, something else isn’t working too. Screw it. Let me revert to a clean state. Let me just delete that stupid whetstone_old directory that I created. And I’ll rebuild everything from Git again – from my old state. Delete. Rebuild. Ugh. Read Me isn’t good. How the hell did I build it last time? What does this error even mean? Note here, that I have NOT fixed 1 single bug yet… 1 hour later – AH so all you needed to do was change the config file entry? Okay. Anyway, let me first collect all my old notes from whetstone_old before I forget stuff like this. Don’t want to waste time.

Eh. Where’s whetstone_old?? Oh no!!! Don’t don’t tell me it was in the directory I deleted #-o. Yes. It WAS in the directory I deleted. Woo Hoo. All gone. I’m screwed. Backups are important. There’s a reason you backed up to whetstone_old. Why would you delete it?

Another day wasted then. Okay okay, now lets fix bugs. Ah, here’s an easy bug – “Display last login time for a user”. 10 minutes. I got this. Just print a date out in the view. Adds date to view. But this is just printing the current date each time ffs :-o. We want it for each user. Oh. That means a new column. In the user database. I know a little MySQL though, from all my SQL injections over years gone by, so this should still be quick….

Error. Can’t connect to MySQL database. Eh? Different port? No. 15 minutes. Can’t figure out where the DB is. RTFM. Sqlite. NOT MySQL. Now if I want to debug anything, I need to learn Sqlite querying. Learn how to use a database. NO ffs I have still not fixed the bug. Okay, now I understand sqlite. But how do I add that column? Learn Rails migrations. Wow. Another half an hour. Add column. Finally fix bug. It lunch time. I should stick to hacking things – I clearly am not good at this stuff. But wait, now I got it.. I know how the DB works, so things should now work out...

Okay, lets login and see if our date prints right. Just to check lol. Obviously nothing can go wrong. Hmm, looks okay. Oh wait, we need to do some stuff with the Date. Why do none of the in-built functions work? Oh no, its not a varchar – its DateTime – only then will some functions work. Or I’ll have to write my own functions. Which would be stupid for such a trivial task. So I have to change the data type in the DB. Learn how to change Rails migrations. And while that looked simple, it didn’t work. The accepted solution was to “DROP TABLE” and “Recreate TABLE”. Ugh.

Now remember, I hadn’t created the table in the first *^%$#@! place. So I now have to find out the structure of every single column of that table and re-create it. Screw it. I don’t want a fancy date. I’ll just leave it as it is and go on… But now that failed migration attempt has screwed something else up – and all my normal code that I never never touched is not working. Fantastic. Things you never coded can break despite you never touching it.

Meaning, I now have to learn the entire old schema. Luckily logging was turned on by default, and the old old old log file was never truncated. Extract the old old query. Convert it into a valid Sqlite query. Drop old table. Create new table. Okay error gone. But now no date display. WTF!!! That’s cause recreating the table destroyed my Rails migration. So do that again. Finally done. In short, playing with the DB isn’t particularly fun, and any re-architecting brings up weird weird problems.

And after all this, all I’ve done is fix one lousy feature request that displays the last login time. Okay let’s move on to the next bug. It’s a security issue now, user IDs are generated sequentially and that’s a problem – a hacker could enumerate all valid numbers. So I need to randomize stuff. Meaning I have to change the data type of a column. Right now, the id column is an integer, I need to convert it to a GUID. Looks very similar to the previous DateTime problem haha. I got this one for sure. Migrations. Change. Blah. . Er. Why did something else break again? Why is nothing working now? Why can’t I even login? Panic. Turns out that id was the Primary Key for the DB. And messing around with the primary key is a bad bad idea. Even worse than messing around with a database. Don’t do it. Just just don’t do it. Deny that feature request.  Learn how to change a primary key.

This is terrible. Is this what fixing other people’s code is like? And this is for a small internal application which is relatively well written, compared to some of the bloated code that I’ve seen over the years. Anyway, more hours wasted – I figure out what to do. But now, for some reason I can’t find out where the hell to put the ‘randomizing code logic’. A few more hours… oh wtf.. you know by now that I’m spending more time fixing stupid knowledge gaps than actual bugs.

As it turns out, my boss (correctly) decided to not write a line of code for authentication, and relied on a third party Ruby Gem instead. Well whoopee doo doo. #-o Why am I sarcastic, you say. It means that I now have to hack on some third party module code instead. Start learning third party module code now. That’s another fail, and I have to contact the module owners who patiently explain what to do. Eventually it all works.

Lets look at Bug 3 now. All you need to do is to make a really simple UI change. This CANT be hard ffs. How hard can it be? But by now, I’m in a numb state – coz I’m fairly sure something will go wrong. Oh look. Its CSS. And CoffeeScript. Do I know either? NO. Learn CSS. Learn Bootstrap. Learn Coffee Script. The saddest part is I just need enough to fix a tiny bug, but without knowing the basics – I can’t do it. More time gone.

And this pattern repeats.. again and again and again. Eventually of course, as expected (else I won’t have a job) I improve and start fixing things quicker. But something or the other always always breaks – when you least expect it to. And when its not your code, its harder to fix it. And now I think of those massive banking applications, when the chief architect and the lead developer quit a while ago, and there are 5 new developers. Shudder.

And I still have tons of bugs left. I find my mind thinking of shortcuts and dirty ways of solving problems all the time. I find myself thinking of how to somehow get that number of bugs down. Somehow. Quickly. So I can get back to actually writing new cool stuff. It isn’t easy to maintain that mental discipline. Honestly isn’t. And this is for an internal app, which isn’t critical and with no deadlines except those that I set for myself. I have a new found additional respect for developers.

I always knew a developer’s job is trickier than that of a hacker – and I’d like to honestly say I’ve always been respectful to every dev I have met. But I’ve never sympathized with one. I’ve always felt that it is yet another job that one learns to do well over time. And that’s true, sure – but they do work under greater pressures than I do. There’s other skills in my job as a white-hat hacker, other traits that maybe developers don’t need to learn – but whatever a developer does need to do – isn’t easy. IF IF you want to do it well.

Lastly, I want to tell every white-hat hacker to put on the defensive hat once. Actually code. Code things that people will look at and break and tell you how awful your hacks are. And how you should test more… before releasing to production. See how it feels. :) If not anything else, it’ll make you respect the developer community more than you already do. And that’s how it should be.