Have you ever had a typo break everything, similar to instances where DNS configurations have brought down the cloud or Chrome pointing entire workflows to some other website?asked by China Shop Rodeo
TL;DRAhmed Shaker recalls an extra space in a URL inside a large config file that broke authentication in staging, took a while to find because the error was unclear, and never reached production.
I did have one of these. Every one of us has had one of these. One time... so the way we deploy our environments is very well known in the cloud and DevOps industry as blue-green deployments: we would have a full environment running, whether it's PTU or live, players can see it, and then we stage a shadow environment, complete and full, and then at the moment, we just switch the load balancer and the players get moved. I can't remember, was it 2.2? And I had to edit very, very large files; I mean, we are an alpha project. So sometimes we broke our own rules, like, I'm going to test it right before I do it. Yeah, we have no time. The thing that we have in DevOps, and I think we have in Austin, is we deliver whatever the company needs to be delivered as quick as possible. So yeah, it was one of these things where you hack something really quick, and I had an extra space in a URL that goes to the platform APIs in a very gigantic dictionary that generates a lot of configuration that goes to different services. And this space led to an authentication issue. So QA was testing, trying to get in, and nothing was working. And it took a while, because the error message wasn't very, very clear. So yeah, these things always happen. The good thing is we did not do it in production. We had it in the staging one, so we got to fix it and see it. I'm sure every person running live production servers all the time gets to see some of these human errors.