I thought this was a good example of how STEM communication can work well. The author introduces concrete, everyday problems then provides the math and logic to work through them. That is surprisingly rare in STEM comms.
The core ideas of this book are very interesting but it feels very much like a research thesis stretched to book-length. The repetitive examples of how prosperity did or did not develop in different areas of the world are supporting evidence of the thesis, but become a bit of a slog. Definitely worth reading, but I’d recommend selective skimming once you get a handle on the core of it.
Cheever’s short stories are strange in that they feel both very foreign and very familiar. The style and melancholy are modern and relatable. The settings and context… aren’t, at least to me. These are East Coast stories that take place in the 20’s-50’s. They reference household servants, wet bars, and console radio sets. Those things aren’t strange by any means, but it’s a testament to the quality of the writing that even set 70-100 years in the past, the stories themselves feel contemporary.
43. “Remote” – Jason Fried & David Heinemeier Hansson
This was a re-read. I’d read it when it first came out a few years ago. It’s one of those business books that companies buy for all their employees to try to pitch a new philosophy. That doesn’t mean it’s bad, just that is it fairly light. It’s meant as a sales pitch for remote work. I dove back in to pull out some items relevant to a job I recently started.
I started this book knowing almost nothing about James Garfield and came away with a lot of respect for him. Had he survived his assassination, I don’t know that he would have been a “great” president, but he appears to have been an incredibly decent man. His death helped bring the country together and heal some of the wounds from the Civil War. It also, at least in part, inspired Chester A. Arthur to completely turn his life around and reject the Tammany Hall corruption he had spent his career soaked in. This is a super-engaging historical narrative.
This is by no means a flawless book, but it is the most comprehensive history of Hitler’s empire that I have read. Granted, it is 1300ish pages. Given the subject matter, it’s not a book you can read without feeling something. The Nazis’ crimes are almost impossible to process – too terrible to fit wholly in your brain. There is so much in the history of the Third Reich to be angry about, things we should remain angry about and watchful of for as long as our species exists.
I needed something a little lighter to recover emotionally from the previous book. The core idea here is that your time and attention are invaluable and you should viciously protect them. Also, that open offices for “collaboration” are a sham, which is no surprise to anyone who has ever worked in one.
Read this on the recommendation of a friend and enjoyed it, only minimally due to it being about my people. The author goes into detail about how class has been used as a weapon in the U.S. to control and divide. The history covered adds to the mountain of evidence that Southern Secession was never about states’ rights and that the possibility of upward mobility is no where close to evenly distributed.
I really enjoyed this book. It’s basically “if a botanist wrote sci-fi” and felt very fresh as I haven’t read anything in that vein before. It does suffer somewhat from what I call Clarke-Doctorow syndrome – a condition in which the quality of the ideas presented vastly exceeds the quality of writing.
I normally don’t care for ‘business’ books but this one was fairly interesting. Dalio presents a unique take on what a data-driven culture looks like. Working at his firm, Bridgewater, doesn’t sound like much fun to me, personally, but they do seem to have achieved as mature an analytical culture as I have seen. I didn’t particularly care for the structure of the book as it’s setup as part-autobiography, part reference book, but it makes sense in the context of the ideas presented.
This is such a weirdly wonderful book. Fortey weaves his personal history as a paleontologist through a history of life on Earth. He balances passages on the evolution of trilobites with anecdotes about laying under the stars in the Australian Outback. This is a type of book we need more of as it humanizes science while broaching big, hard-to-comprehend ideas.
I wanted to like this more than I did. “Binti” is the first of Okorafor’s books that I have read and I will attempt another as there were the seeds of many interesting things here, but this novella read like a first draft for a novel. I normally like terse writing, but this stretched that. It’s the skeleton of a story without much of the meat.
I liked the other Jemison books I’ve read this year so decided to try one of her other series. The Dreamblood duology was tight and well done. It’s heavily influenced by Egyptian mysticism, which was a nice break from the standard fantasy tropes of European heritage.
This is a really solid, if dated, primer on statistical misrepresentation. It was written in the 1950s so you have to keep reminding yourself of the context around any dollar amounts thrown out. The companies used in many of the examples might as well be made up names given how few of them exist anymore, but that’s fun in it’s own way, inspiring some Wikipedia searches to find out what happened to them. Aside from that, the tactics described are timeless and I could think of countless contemporary examples of their use. If you like charts and graphs, this is a good, short read.
I had to read this book in short bursts. It’s written in the gonzo-journalism style of the late 60’s/early 70s (think Hunter S. Thompson) in which the narrator is definitely unreliable. It’s a fever dream of the Vietnam War that is disorienting and easy to get lost in, definitely not something I could follow when I was tired. The second half of the book is an easier read, more grounded.
Reading this at the same time I was dipping into “Dispatches” helped me stay grounded in reality. It’s one of the better O’Reilly books I’ve read, being easy to read and covering enough detail to be useful. The author presents a sane approach to thinking about microservices architectures and how to go about chopping up a monolithic application into microservices.
I hadn’t read anything of Le Guin’s since the Earthsea trilogy in middle school, which I remember liking but feeling that it wasn’t very substantial. This left me with a different impression of the author. There’s a lot packed into its 300 or so pages with much of the weight given to how gender norms shape society.
This is a decent run-through of Docker and how to do some basic troubleshooting. I’ve run through better tutorials online that were more up-to-date but lacked some of the context the book provides. It’s a good addendum, but I probably wouldn’t recommend it as a place to start with Docker.
It’s an OK desk reference for different SRE topics, but “The Practice of Cloud Systems Administration” is a far better resource. I discovered after I purchased it that it’s available online for free and would have preferred to have gone that route as it’s basically a collection of blog posts by Google alumni.
This was an odd case of being familiar with a lot of the author’s research from other sources and thus finding the actual source material a little less engaging than I might have otherwise. If you are new to behavioral economics, meta-cognition, or the study of biases, this is an excellent place to start.
This is perhaps my least favorite of Mieville’s books, perhaps because of my lack of familiarity with French surrealism. It’s only a little longer than novella length and may have benefited from some more meat on its bones to help add context for those not intimately familiar with the vernacular (or all the French for that matter).
Some dumb algorithm recommended this to me based on my like of China Mieville. Scott Lynch is good at setting pace but that’s pretty much it. Tone is all over the place. None of the characters are believable and there is a significant amount of repetition and deus ex machina.
This is more a history of finance than of money. Ferguson is very much in the conservative camp of economists so I found myself challenged by some of his views. The last quarter of the book covering modern issues seems a bit tacked on and rushed. Overall, it’s a good read, but heavily biased towards a Western, conservative economic view point.
Microservices can be awesome. Splitting up your monolith to scale services independently, accelerate change, and increase resiliency can reap huge returns. The agility that well-implemented microservices provide is one of those things you look back on and think “How did we manage this before?”
Microservices can also be terrible. If implemented poorly, they can cause more problems than they fix and create unmanageable chaos.
I’ve seen both scenarios, but failed microservice projects outnumber the successful ones I’ve come across. It’s getting better as the ideas around microservices mature and people gain more experience implementing them, but there’s still a long way to go.
Failure is where we learn. In thinking through the failed microservice implementations I’ve seen (Granted, this is from an SRE/ops perspective.), they all share similar traits.
1. They were split off too early
A well designed microservice requires a solid understanding of the service boundaries and context you are trying to carve out. If you’ve just built your MVP, you likely don’t have that understanding. If you’re trying to start your app with microservices, you definitely don’t have that understanding.
The rationale that folks tend to lean on when they don’t understand their app’s boundaries well is to split things out by data model rather than behavior. If you split your app into something like “users”, “orders”, & “items”, you’ll end up with three CRUD microservices and then a whole other set of microservices (or a seperate monolith) that handle business logic between the first services you created. You’re basically just abstracting your database in that scenario.
Instead, if you did something like “auth”, “order”, & “stockcheck”, where each service is defined by its behavior instead of its data, you end up with fewer, smarter services, intuitive logic flows, and a clear idea of how and when to scale each piece.
Stick with a monolithic app as long as you can. Let the app and your understanding of the business processes you are building mature. Then, when and where it makes sense, start splitting off microservices.
2. They were tightly coupled
A business process may rely on multiple microservices working together, but each of those services should not rely on another to function. Otherwise, you don’t have microservices. You have a distributed monolith that contains the worst of both worlds.
The worst example I’ve seen of this was an app that had three versions of a microservice-delivered API. Versioning your APIs is legit, there’s nothing wrong with that. But what these folks had done was make each version synchronously reliant on the version before it. V3 referenced v2 and v1, v2 referenced v1.
I have seen the inverse as a migration tactic, where devs implement a new version of an API, keep v1 up and change the logic of v1 to forward requests to v2. That feels reasonable. Coupling dependencies backward made no sense. It meant they could never get rid of their older APIs and troubleshooting was a nightmare.
In this same stack, almost all of the services were reliant on complex logic that took place within the messaging system (RabbitMQ in this case). This resulted in frequent cascading failures that affected all services.
3. They were orchestrated
There’s a microservice (and SOA) concept of “choreography over orchestration“, meaning choreographed services can function independently, whereas orchestrated services require a central conductor.
I’ve seen a pretty common scenario where microservices are implemented and then driven by an enterprise service bus (ESB), like JBoss Fuse. All requests and core business logic have to go through the ESB, which is likely the hardest component to scale in the entire app, usually due to licensing and technology limitations around state management. Because all of that logic is centralized in the ESB, the microservices around the spoke don’t know what to do unless the ESB tells them what to do. This is bad if you’re trying to build a resilient system.
Again, this would put you back in distributed monolith territory. ESBs are inherently complex and often fragile. So you have a complex, fragile, hard-to-scale single-point-of-failure bottlenecking all of your services. There is little chance this won’t cause you pain. That’s not to say that ESBs don’t have their place, just that they don’t line up well with microservice architectures.
API-gateways seem to be becoming a new form of ESB, so it’s important to watch for similar problems there and keep your API-gateways as dumb as possible (basically just using them to consolidate requests or add auth).
Be patient & thoughtful
Well-architected microservices require a lot of dedicated thought and research. This is an area you don’t want to launch into after reading your first blog post on the topic. Having to re-scope service-boundaries and re-implement services is one of the more painful engineering exercises I’ve been through. I hope to save you from that pain.
Luckily, there are some great resources to help. I’d recommend the following:
Don’t pursue microservices because Netflix or Google or Facebook uses them. Use them when they make sense for your app and be OK with the idea that they might not make sense for you at all.
Microservices are not “better” than monolithic architectures. They just solve a different set of problems that mostly have to do with scale and the way a particular business operates. Part of being a good engineer is using the right tool to solve the right problem. Before you jump into microservices, pause, and make sure that’s what you’re doing.
Because spinning up VMs in the cloud is so easy, it’s equally easy for your monthly bill to scale up as well. What may have started as a few hundred dollar-a-month charge for a few VMs can quickly ratchet up into tens of thousands of dollars.
In larger businesses, I’ve seen this get so bad that management freaks out and moves everything back on-premises, even when the growth was legitimate and not the result of waste or mismanagement.
There is also a tendency for devs and ops folks to treat the financial aspects of their infrastructure and applications as “not my job”, which is an idea that makes my brain hurt from all the angry it creates.
Three tables flipped, and I still can’t write about the “not my job” angle. It will have to wait for another blog post. For now, let’s just assume that you, dear reader, are a minimally-functional adult who understands the concept of owning the things you build.
As you’re building your infrastructure, it’s always a good idea to bake in instrumentation to understand utilization. Most people view this as a must-have for troubleshooting and monitoring, but there’s also a cost angle.
Assuming you already have infrastructure, the first step in lowering your AWS EC2 cost is identifying what you actually need.
You can either roll your own solution for right-sizing your infrastructure using the performance and inventory data you should already be collecting with AWS (or your cloud provider of choice) or you could use a service like CloudHealth Tech or CloudCheckr that are purpose-built for cost optimization.
The SaaS solutions do a lot of stuff related to cost optimization, but one of the main things they provide is utilization reporting and recommendations.
Maybe you thought you needed 50 m4.2xlarges for your app. Maybe what you really need are 25 t2.mediums (a $14k a month difference). I have seen gaps that large revealed when running utilization reports for the first time.
You may be thinking “someone who understands their app would never let that happen”, and that is true most of the time. The problem is:
Very few people actually understand the apps they run.
Very few people consider the cost of running those apps as being something they own.
Related to #1, many people don’t have the luxury of running apps that they have built themselves. They are handed someone else’s shitty app and told to keep the lights on.
Getting a good grip on actual utilization tends to also help with conversations like “maybe we should spend the time to make our app stateless so we can auto-scale servers” or “maybe it’s time to convert this app to PaaS.”
Note: If you do end up switching a bunch of servers to t2-tier, make sure you monitor CPU credits. This tier has a CPU-usage quota and if you go over that, your server is throttled until the credits rebuild. I’ve seen people flip over to t2s thinking “it’s got the same specs as my m4/5” and then wonder why their high-CPU-usage app is crawling a few hours later.
Buy a baseline
An argument that is normally used to justify hybrid cloud is “we have a baseline on-prem, so we’ll just use the public cloud for burst elasticity”, which is reasonable. However, you can do the same thing in-cloud.
Once you get a grip on your baseline needs, reserved instances (RIs) become an option. It is entirely realistic to cut costs in half with RIs.
RIs take away some flexibility in return for lower instance prices and are similar to financial instruments you see elsewhere in the world, like cell phone plans.
“You can pay month-to-month with no contract, or you can save X% by signing up for a minimum term.”
Unlike a cell phone plan, there is an RI market where you can resell your RIs if you need to get out of them, and there are also lots of options around converting them different instance sizes. Converting RIs to different instance families is also possible, but a bit trickier.
Note: There are limits around selling your RIs and the marketplace doesn’t have a high volume in many regions, so selling opportunities may be limited. So don’t purchase RIs with 100% certainty that you’ll be able to resell them.
RI discounts are primarily tied to term (the length of the contract) and percentage paid up front (none, partial, or full). Three-year full-upfront RIs are going to be the cheapest option.
A trick I’ve learned in the past year is that many resellers like SHI and CDW offer financing for AWS RIs. So instead of paying for 1 or 3 years fully upfront, you can amortize that “upfront” cost over a term with your reseller. Obviously, you’re paying interest as part of the financing, but if the decision was between a 10%-off no-upfront RI purchase and a 60%-off full-upfront purchase, paying a few points in interest is a no-brainier.
Fun fact: RIs can also be purchased for RDS instances and this is almost always an easy win because of how “permanent” your database servers likely are in comparison to the rest of your infrastructure.
Spot instances are your friend
Assuming you’ve purchased a baseline of RIs, any additional instances above that will be either at the “on-demand’ or “spot instance” price.
With spot instances, you are effectively bidding on AWS’ excess capacity, so pricing is highly dynamic based on demand. Because ‘spots’ are excess, they will be pulled back into AWS’ inventory when that capacity is needed by other users paying the on-demand or RI rate.
Spot instances are awesome when they’re feasible for what you’re doing (test environments, fault-tolerant apps, EMR processing, like Spark). Where you might get an RI for 50% off, I’ve seen spot instance discounts as high as 80-90%.
Some folks have done an amazing job of analyzing historical spot prices for the types of instances they need and have purchasing algorithms that help them both track price trends and buy & run their instances when it’s cheapest to do so.
Note: A word of caution here – spot price purchasing is something you want to keep an eye on. Because it’s demand-driven, spot pricing can sometimes (though it’s not often) spike above the price of on-demand pricing. So you’ll want to make sure you’ve accounted for that in any automation you build.
You’re not stuck
A common assumption that people make with their EC2 environments is that the cost “is-what-it-is” until the apps running there can be sunset or migrated to PaaS. Fortunately, that is rarely true. By leveraging some financial tools in your config, it’s highly likely you’ll be able to bring the cost of your EC2 environment down.
If you’re working in enterprise, this stuff will make the biz folks love you, and they’re likely who control your salary.
If you’re running a startup, this stuff can make the difference between you being able to hire more people, or even stay in business at all.
None of it’s hard, it just needs to be accounted for.
I’ve been reading more than writing so far this year. I tend to go in bursts like that – lots of writing, lots of reading, lots of Netflix, repeat. I’m not sure why, it just happens.
So far this year, I’ve read 18 books, a mix of fiction and non-fiction. I’m putting together notes on what I’ve read so I can come back to them and so they might be useful to others. I guess they’re sort of lazy mini-reviews. None of these are affiliate links, because I don’t care.
This is book one in “The Broken Earth” trilogy and is the first book of the author’s I have read. The story takes place in a fantasy future-Earth setting – think Shannara, but not shitty. I enjoyed Jemisin’s style and perspective. Her world is logically consistent and filled with truthful subtext.
It’s a book that has made all sorts of lists and is very much in the vein of “could have been a blog post instead of a book.” If you’ve read Malcolm Gladwell’s “The Tipping Point” you can skip this one
This is probably the shortest book Tad Williams has ever written, although “Child of an Ancient City” may tie it. I’m too lazy to check. It’s effectively a long prologue that ties Williams’ “Memory, Sorrow, and Thorn” trilogy (or quadrilogy if you read it in paperback) together with his new “The Last King of Osten Ard” series. I enjoyed the brevity, and that there were a limited number of subplots, which is rare for Williams.
Another “on all the lists” and “could have been a blog post” book. It was significantly better than “Made to Stick” in that it broke more original ground and is better written. The basic premise is “Don’t follow your dreams. Get really good at something and opportunities will present themselves.”
I enjoyed Bacigalupi’s “The Windup Girl” and hoped to enjoy this. It’s not as good as that book, but the story is compellingly told. The almost gleeful descriptions of torture and abuse that bothered me in “The Windup Girl” are cranked up to the point that it gets in the way of the story and you start to worry about the author (and yourself) a little bit. I probably won’t read his next book.
Related to item 3 above. Williams has gotten so much better over the years. His pacing is still molasses slow-build up to frenetic chaos, but he’s done a better job of culling boring subplots and characters. That’s not to say this doesn’t sprawl, but his writing feels more purposeful.
The curse of knowledge ruined this book for me. It is likely a very solid layman’s primer for near-term technological trends. But, being in the industry, my response to the info presented was “yep, and…?” That’s no dig at the author, I’m just not the target audience.
This is marketed as a novella but is barely that at 81 pages. It’s a solid aside set in the Expanse universe and serves as an acceptable fix to carry oneself over until the next release in that series. It sets up a premise that could be its own set of novels but that’s a lot of things in the Expanse series, which is expansive.
Finishes the “Broken Earth” series. It left me thinking a lot about coded language as well as motherhood and mother-daughter things that are completely foreign to me and that I will likely never fully understand. Thus is the power of reading books by authors who aren’t hetero white men.
I’m very interested in anti-fragility and resiliency, so this book was fun for me. It fed into a lot of ideas I’ve been thinking about related to chaos engineering. Taleb’s ideas are powerful and feel so uncomfortably true that it’s easy to look past what appears to be massive ego and condescension towards his peers. I actually like this book better than his “The Black Swan”, but that may just be because it lines up well with current thoughts.
Magical realism tends to go one of two ways for me. 1. This is fucking stupid. 2. This is amazing. This book was the latter. The setting of the story is a post-apocalyptic city inhabited by a giant flying bear. It’s completely bonkers but works wonderfully and inspires me a little in a “this is the kind of book I want to write” way.
This is a book about a possible theory of mind and sentience for cephalopods. I would summarize it with “Octopodes are super smart in ways that are completely foreign to us.” I’d given up on eating octopus a few years ago after watching videos of them using tools and this made that decision concrete for me (in addition to expanding that food-ban to squid). I probably need to be careful reading similar books as I may end up not being comfortable eating anything.
Given our current situation, this book made me both sad and grateful. One thing it really hammered home for me was how broad and open Obama’s world view is, more so than any president we’ve ever had. Similar to other black American memoirs I’ve read, there is much I couldn’t truly understand but is still ugly and painful.
This is the least weird of Mieville’s books I’ve read. I think he wrote it for his mother or some such. I can’t say that I enjoyed it as much as “Embassytown” or “Perdido Street Station”, but it was good.
I normally struggle a bit with “future” books, but O’Reilly has good credentials and writes with an authority that lends weight to his vision. Some of the first chapters felt a bit “look at me! I invented the internet!”, which is related to the previous statement. All in all, his is a compassionate take on a potential path for the future of work and business. I wished more startups would heed his advice.
To be honest, this is the first book of Hodgeman’s that I’ve actually liked. His fake trivia books hinted at something more interesting, but never really got there for me. This book is conversational storytelling and succeeds as such. It’s definitely in the vein of “NPR humor”, so your mileage may vary depending on your appetite for that.
Leaving a job is hard, and oddly enough, it gets harder each time you do it.
When you’re new to the workforce, you don’t know enough to appreciate what you might be leaving behind or worry about what you might be going to.
As you get older, that changes. You know how hard it is to build relationships and find your place. You know how stressed out you’ll be for the first six months in the new job, fighting against imposter syndrome and the uncertainty of new territorial politics and personalities.
If you’ve been a manager, you know the heavy feeling in your gut when someone gives you their notice and it’s impossible not to think about making someone else feel that.
You’ll lay awake at night trying to navigate to the “right” decision. You’ll worry about being happy, about your family being happy, about your coworkers and their families being happy. You’ll worry and stress over things you’ve never even thought of before.
It’s painful. It sucks, but hopefully, it makes you handle things the right way. It may even change how you approach jobs altogether because leaving a job the right way starts on your first day of work.
Say “no” to ego
I know people who have given months’ worth of notice when quitting. Most of the time, this was coming from a good place. They cared about how their coworkers would be affected and wanted to make sure business could go on as usual with minimal pain.
It’s impossible for me to not see the role of ego in giving that much notice though. Even if it’s coming from a good place, there’s an element of “The work I do is so important/complex/special/whatever, that it requires weeks/months of additional work to get someone else even minimally ready to take over.”
I say this as someone who has given a month’s notice before. If I’m honest about it in hindsight, that’s exactly the reason I gave so much notice. Well, that, and I hadn’t done the right things to prepare for me leaving along the way.
Generally, if you leaving a job requires weeks worth of knowledge transfer sessions and documentation work, you have done something wrong. You haven’t made yourself dispensable.
Prepare to leave every day
Don’t wait until two weeks before you leave to start removing yourself as a critical path to getting things done. Even if you never plan to leave, do you really want that anyway? The only award you can count on getting from being indispensable is being permanently on-call.
When you design something, document the high-level design and keep those docs updated when you make changes.
When you write code, write it as if you are immediately handing it over to someone else to maintain. And then actually hand it off to someone else instead of making it your baby.
When you setup apps and infrastructure, and anything else that has “access”, make sure someone other than you has access too.
Communicate. Make sure the team knows what you’re working on and how it works. You’re not bragging, you’re keeping them informed. They’ll appreciate it when they get tasked with taking on the things you’ve worked on.
If you’re worried about job security, focus on being good at what you do, not keeping secrets. Being “good” is way more effective at securing your job than acting like a hermit wizard.
The last one is a big one. When you do eventually leave, what feels better? That others miss your input and capabilities, or that they think “I sure wish they would have written more stuff down”?
Where you’ll end up
Doing these things will make your day-to-day job better (you’ll have more time to work on interesting things instead of solving the same problems) and it will make your eventual departure a lot more pleasant.
Imagine a scenario where you give two weeks notice and aside from having meetings with managers who don’t want you to leave, you spend the rest of your time just doing your regular job.
You’re not doing knowledge transfer. You’re not scrambling to document things. You’ve already done all that. So, instead, you get to write a little more code and focus on saying goodbye.
This may not leave you feeling completely at peace with leaving for a new job, but it will take away a lot of anxiety. You’ll leave feeling accomplished and confident that you handled things “the right way”. You’ll have maintained good relationships with your coworkers and bosses. You may even have built a safety net, a place you could go back to if you needed or wanted to.
I’ve never been at O’Hare for what anyone would consider a layover. It’s always been more a “desperate sprint from one far-flung terminal to another”. Wandering through terminal C, looking for somewhere to grab a drink and food, therefore, feels novel.
The high arched ceiling of the main corridor alludes to gothic cathedrals. The niches and warrens that house shops, bars, and cafes strengthen that tie. Unfortunately, the limited space also makes it hard to find somewhere to sit. So I stand awkwardly behind a row of occupied barstools sipping a $10 pint of $3 beer.
Table churn is slow. Food will have to be found elsewhere. The co-worker I’m traveling with gets a salad from the food court. Possessing an iron stomach and the culinary discretion of a raccoon, I opt for the “Chinese” buffet. Bellies full and wallets lighter, we board our flight to Frankfurt.
Upon hearing several mentions of it being a full flight, the thought briefly crosses my mind that this being United, I may be at risk of being assaulted and pulled from my seat. But then I remember that I am a straight white man and go back to reading my book.
I read for most of the flight, attempting to sleep a few times and failing until 30 minutes before touchdown.
Frankfurt is the most German of German airports, all clean lines and gloss. The path to our connecting terminal is more byzantine than German though. It snakes through multiple buildings and seems at times as if it is taking us back to where we arrived. Eventually, it deposits us out at an unexpected security checkpoint.
German TSA appears very concerned about the electric razor in my bag. The x-ray agent studies it in a similar fashion to the apes approaching the monolith in 2001: A Space Oddessy, flicking the power button and jumping in surprise when it starts buzzing. Given that it is a Brauhn, a German brand that should be familiar, I am a bit confused by the reaction.
The x-ray agent waves an assault-rifle-toting polezei over to discuss the situation. She flips through my passport and confirms my travel plan with me. After a back and forth with the x-ray agent that overlaps with none of the small amount of Deutch I know, she returns my passport, says that everything is fine, and wishes me safe travels.
Four hours later we are on our way to Bangalore, having failed to acquire better seats because of a lack of integration between United and Lufthansa’s booking systems. Being three rows from the back of the 777 makes for a take off that feels on par with a turboprop hitting jet-wash – that is to say, very bouncy with a lot of harsh lateral movement. I put on my headphones and eye mask and doze off and on for the majority of the flight.
As she’s serving breakfast (chicken shiha, the beginning of this trips’ Indian food), the flight attendant jokes that she has not seen me the entire flight, given that I slept through all the previous meal/snack services. She also mentions that my eye mask looks like a bra, which is true, but it is excellent and I do not care.
Bangalore’s airport is not as busy as I expected (in fact, it’s almost empty), but it is 1 AM. Due to very ambiguous signage, we stand in the wrong immigration line but are soon re-directed to the correct line.
The security stance appears to be the exact opposite of Frankfurt. I could have handed the immigration official any piece of paper in my possession and received an equal level of interest. The customs agent pushes my bags through the x-ray machine, waves me through after I set off the metal detector, then proceeds to go back to sleep.
Our driver is waiting for us and guides us out to a parking area for hired cars. Cool night air filled with the scent of curry powder and chai from the airport’s outdoor vendors replaces the previous 24 hours of recycled airplane stench (John Roderick’s airplanes as “fart-tubes filled with long pigs” remains one of the most accurate descriptors of the modern age.)
Our hired minivan careens down highways and side roads at double the speed limit. Lane discipline, turn signals, and other familiar Western driving concepts are not Indian thought technologies. Our driver takes us flying over speed bumps and potholes, honking at cars, tuk-tuks, cyclists, pedestrians, and stray dogs to warn them of our approach down the center of the road.
We weave around large trucks for which different signaling rules seem to be needed, flashing the high beams instead of the horn. There does seem to be a method to the chaos, but I can’t fully decipher its rules.
An hour later we arrive at the hotel, which is heavily guarded – high fences, bomb-sniffing dogs, and x-ray machines. The staff is cordial and helpful in a style that is almost uncomfortable. “Namaste. Namaste. Namaste. Thank you. Thank you. Thank you. No seriously, I can carry my bag.” Whereas an American hotel might have just one or two staff in the lobby at 3 AM, this Marriott is swarming with receptionists, bellboys, waiters, and valets.
We are offered a selection of juice-filled tumblers. I take one that turns out to be lychee juice and am guided by a hostess to my room where she painstakingly shows me how to use all the different room controls and light switches. I do my best to be polite, but shuffle her out the door as quickly as possible. I want more than anything is to go to sleep.
I set my alarm with the intention of getting breakfast, but hitting snooze multiple times takes me past the hotel restaurant’s breakfast hours. I drink Nescafe and eat a bag of masala-flavored Lays for breakfast instead. The masala wakes me up more than the coffee.
As I consume my non-coffee and spicy chips, I notice a woman hanging laundry on the rooftop of an office building across from my hotel in what looks like a scene from Ip Man. Still sleep-drunk, I imagine an army of Wing Chun practitioners sparring between the clotheslines.
The daytime traffic is fascinating to watch. Motorcycles and tuk-tuks swarm around larger vehicles while pedestrians dart across the road. Occasionally, a vehicle will pull into a drive, turn around and then drive the wrong way back up the road to the turn they missed. The ebb and flow are hypnotic.
The office address we were given shows as being a half mile away. The sidewalk along the route is in process of being demolished to make way for a new metro line, making it a bit challenging to navigate.
A valet at the hotel tells us the project is scheduled to take three years, but will likely take five. Barefoot workers chop at the dirt with assorted shovels and pickaxes, scooping debris into shallow buckets that look like poorly treated woks. Five years seems ambitious.
We arrive at our destination to find that it is the wrong building. After struggling to communicate with the security guard, I make a call and one of our local contacts comes to fetch us. He chauffers us to a building that is just a few blocks from our hotel.
The parking lot and garage are filled with more motorcycles and scooters than I have seen in aggregate during my life – Hondas and Royal Enfields as far as the eye can see.
Seemingly dozens of people introduce themselves and I struggle to remember their names. We spend most of the afternoon comparing notes and coordinating meetings for the days ahead.
Our walk back to the hotel is much shorter than the trip to the wrong building but feels more dangerous due to the narrow street and high volume of traffic. There is more sidewalk, but most of it is taken up by parked motorcycles and vendors selling an indecipherable selection of food, which I ask one of our local contacts about.
“It’s nothing you would want. Most of us wouldn’t even eat it.” I’d already assumed as much due to the unappealing smell – a mix of spices and fruit gone/going bad.
Hungry and tired, we drop our bags and head to the hotel’s main restaurant. The menu is a mixture of traditional western dishes and westernized Indian food as well as some items that seem entirely random. I order “Kung Pao” (their quotes) chicken.
It turns out to be good but incredibly spicy. I have to shoo the waitstaff away as they don’t seem to believe I can be trusted to scoop rice and chicken onto my own plate. One waiter, in particular, is genuinely stressed out about me not letting him help.
“Sir please, allow me…”
“I’m good. It’s OK.” I shovel a spoonful of peanuts onto my chicken.
“Sir…” He’s sweating.
“Thank you. I’m OK serving myself.”
“Sir…” He slowly backs a few feet away but continues to watch me. Eventually, he finds someone else to help, but I notice him eying our table for the entire time we’re there, even as other staff re-fill drinks and clear plates. I feel a little bad for not letting him help.
I’m asleep by 8 PM but wake at 1 AM. Luckily, I manage to go back to sleep until a more reasonable hour. We eat breakfast and successfully navigate our way back to the correct office without getting run over.
The morning is full of meetings after which we are shuffled up to a cafeteria on the 8th floor.
“Do you want a meal or a sandwich?”
“What are the meals?”
We’re shown to several stations serving what all seem to be the same stew in slightly different colors.
“What would you recommend?”
“The meals aren’t any good, but some of the sandwiches are.”
“OK, which one?”
“Corn masala sandwich is good.”
Corn masala sandwich is not good. It is an unpleasantly spicy mixture of sweet corn, onions, and tomatoes – mostly onions. I manage to eat half of it in an effort not to be rude while making jokes at my own expense about not being able to handle so much spice. Everyone laughs and nods.
I’m also served “buttermilk” accompanied by the statement that it is “good for digestion” which doesn’t inspire confidence. “Buttermilk” turns out to be warmish milk curd spiced with chilis and cilantro leaves. It isn’t bad and online research reveals that 1.) the drink is more appropriately known as chaas or mor depending on where you’re at in India, and 2.) it can be very good if properly prepared.
After lunch, I comment that I would like to buy my wife a saree (which seems like the most stereotypical Indian thing possible) while in the country and ask if the clothing sizes are similar to US or British scales. This produces a confused look followed by laughter.
“Sarees are all the same size, six yards of fabric.”
“Oh. So it’s basically just a long blanket?”
“Yes.” With ‘stupid American’ remaining kindly unspoken.
We end our day and wander back to the hotel.
Its elevators are plastered with advertisements for barbeque at one of the hotel’s restaurants. Overwhelmed with curiosity for what constitutes a pulled-pork sandwich in India, we make our way to the second floor and find the Whitefield Bar and Grill located pool-side. It is entirely empty.
A sigh, then a look back into the restaurant. “We can seat you on the patio, I guess.”
I order a Moscow mule and the pulled-pork sandwich, which turns out to be something like a sloppy-joe topped with coleslaw on a black bun. It’s not bad, but not barbeque.
I fight it, but again, I’m asleep by 8.
The next morning, we try a shortcut someone at the office recommended. My coworker decides to film the walk with his GoPro. Possibly because of being self-conscious about looking like tourists taking video, this is the first time I realize that we are and have been the only white people walking on the streets, even though white Westerners make up the entire residency of the hotel.
When we watch the video later, it’s startling how big a difference there is between inside and outside the office park. Outside is chaos – mounds of trash, noisy traffic, throngs of day laborers rushing past one another trying to find work.
Inside, workers sweep every inch of sidewalk and floor, the constant honking of tuk-tuks and cars is muffled by tall concrete walls, and rich-in-comparison IT workers buzzing around offices adorned with the logos of Western technology giants. Bangalore is rising to what San Francisco is descending to. Hopefully, when they meet, Bangalore will keep climbing.
At the office, I get a call from the coworkers I had asked about sarees. They ask for my wife’s t-shirt size. An hour later, they show up with a shopping bag containing a blue kurti and leggings. This seems to be a standard, modern outfit for Indian women who aren’t wearing Western-style clothing.
They also give me a couple of Diwali (which just ended) related gifts, a steel platter full of mixed nuts and a clay candle cup.
I can’t help but think this was partially driven by “he’ll never figure this out on his own”, which is likely true. Whatever the case, its a much-appreciated kindness.
Throughout our meetings, there are several mentions of a Friday outing. I eventually gather that it is to “a resort”, which contains “a water park, but not like an American water park”. We are invited.
The cycle of wake up, eat, walk to the office, hold meetings, walk to the hotel, eat, sleep continues for the next two days, over which I manage to find out the resort we’re scheduled to visit is called Club Cabana, there will be “team building” activities, and we’re to be picked up at 7 AM.
From their website:
Club Cabana offers hospitality that symbolizes Indian traditions and culture with a blend of modern preferences. A welcome as warm as this ancient land will enfold you as soon as you step through our doors. Be prepared for a standard of splendour that you thought was long past.
During breakfast on Friday, the coworker I’m traveling with complains of feeling ill but still plans to visit the resort. This remains the plan until our ride arrives at which point he bails to go back to his room.
Recent rains have washed out many of the roads, resulting in giant potholes that in-turn create stop-start traffic and jarring, metal-on-asphalt bounces. My sick coworker’s choice to stay back at the hotel proves to be a wise one.
We pick up a local coworker, then drive to the apartment complex of yet another coworker, where we trade the compact four-seater we’ve been riding in for a larger SUV. Taylor Swift blasts out of the stereo as soon as the driver starts the car and we proceed to listen to a soundtrack of American pop against the backdrop of small Indian towns and countryside.
Several wrong turns are made as a result of conflicting GPS directions and local geographical opinion. We arrive just as breakfast is ending and in time for a “state of the business” presentation. The room has a high metal ceiling and the PA isn’t properly tuned so I do my best to focus on what’s being said but most of it is indecipherable. I just clap when everyone else claps.
After the meeting, there are “team building exercises”, which turn out to be a task relay and a game where we are blindfolded and asked to make animal sounds to find others on our team. Everyone seems focused on making sure I am enjoying myself and I have to reassure them that “Yes, this is fun.” Given the rigid hierarchical structure that most Indian businesses adhere to, it’s nice to see the managers making clowns of themselves and committing to their roles in the games.
Games complete, we are released to the resort’s facilities, which include the water park, a cricket/football pitch, a tennis court, an archery range, and a bowling alley. I didn’t pack any swim trunks for the trip but find out they can be purchased onsite for 200 rupees (about $3 USD). I buy these shorts sight unseen and on unwrapping the packaging find that they are somewhere between a speedo and boxer briefs in size. I will be a pale ghost in tiny pants for the day.
Out in the wave pool, I discover that most of the resort visitors don’t know how to swim and definitely don’t know how to float. So I become a minor celebrity as I float through each wave. I am asked several times how I am floating and tell them that Americans are made of 50% fat, so we float easily.
I spend the afternoon switching between the slides and the wave pool, sticking mostly with the recent college graduates in the group who want to ask me about American movies and comic books.
“Marvel or DC?”
Around four I swap back into dry clothes and visit the archery range. Archery has a long history in India, so I expect that I will be embarrassed by the others outshooting me. It turns out to be the opposite and I’m able to cluster most of my arrows on top of each other in the bull’s eye and in a group to the far right that contains all the arrows with missing feathers. Given that I haven’t shot a bow in 15 years this can be attributed to luck and a limited memory of how to aim.
Before leaving we settle in for tea and snacks. Knowing that the ride back will be rough, I limit myself to a few bites of fried zucchini.
The return trip actually turns out to be faster and less severe due to taking a different route that is mostly highway. Along the way, the Indians I’m traveling with tell me about their trips to America and how hard it was for them to not honk the horns on their rental cars when they visited.
One was warned while in Houston that honking at the wrong driver may result in him getting shot.
“Is this true or was he joking at me?”
“Unfortunately it is.”
It’s fully dark when I’m dropped off at the hotel and I stop at the restaurant to order another round of “Kung Pao” chicken. It doesn’t seem nearly as hot this time.
The next morning we coordinate with some of our local coworkers to have lunch and go shopping. The hotel provides a driver who takes us to Mahatma Gandhi Road further into Bangalore. Along the way he stops several times, wanting us to get out and shop at what are obviously his friends’ shops. We have to tell him repeatedly that we don’t have time to stop because we are meeting people.
Even with the asides, we make it to our destination a little early and have time to buy some knick-knacks from a store with a needlessly complex checkout process.
Pick out purchases.
Go to the clerk in that particular area of the store and trade your selected items for a checkout ticket.
Once all items have been selected, go to a central purchase counter, turn in your tickets and pay.
Take your receipts to a “delivery” area and trade them for your purchased items.
This is what I imagine visiting a JC Penney in 1930 was like.
While picking out items, our local coworkers seem surprised by my knowledge of Hindu mythology, which is honestly very limited. I am able to point out a few major members of the pantheon from the shelves of statues but have to ask about most. I end up with a Ganesha:
and a Shiva depicted as Nataraja which is kind of awesome because he’s standing on a dwarf (which apparently represents ignorance). I can get behind that (smashing ignorance, not dwarfs, who have never done anything to me).
We walk to a building called the Barton Center to eat at a restaurant on the 13th floor called Ebony.
The food is good and I’m dealing better with the spice now. I see one dish with sambal oelek, which I’ve had before, so I use it as a spiciness baseline. It turns out to be the hottest thing on my plate, so that works out well.
Our original plan was to eat here at night to see the city lights from up high, but the daytime view works relatively well, although we’re facing away from downtown.
On the way to the elevator, I notice that the interior of the building looks like a rectangular version the tower from Dredd, which was filmed in Ponte City Appartments in Johannesburg. So it actually looks like Ponte City, I guess. I only know this because I listened to an episode of 99% Invisible about it.
The leader of our shopping expedition takes us to an area called Commerce Street for “street shopping”. She is committed to helping me get a saree for my wife and says that this is the place to do it. What I thought was going to be “buying a blanket” turns out to be more involved than I expected and I am grateful to have a guide.
I buy the saree, then a blouse, then a petticoat. Then we have to visit a tailor in a semi-collapsed alleyway who adds some structural support to the decorative edges of the saree. All of this is done while wading through the densest sea of people and vehicles I’ve ever seen.
My arms are constantly getting clipped by the mirrors of tuk-tuks and I have to step over the tires of multiple motorcycles which are edging up on me as I walk. I do my best not to get run over by the dozens of food carts we encounter, several of which are advertising “American Sweet Corn”.
I momentarily forget about the specter of Delhi belly and try some spiced pineapple from one of the stands. It is very good.
The smells and colors of the market are overwhelming. As is the variety of random items for sale. Socks seem to be one of the more popular items, but there are also quite a few vendors selling stuffed animals.
The crush of people is incredible but there are occasional gaps where we can catch our breath.
I notice lots of birds in the sky as we wander deeper into the market. They are accompanied by an increasingly strong smell of dead fish so I assume we are getting close to a seafood market. Our shopping guide confirms this.
We avoid the fish market but walk to a nearby church. They are holding service inside and I can see the altar lit up in flashing neon, which I’m not used to associating with Catholic mass.
This seems to be the edge of the market area so we begin making our way back to one of the main streets to call an Ola for the return trip the hotel. This takes quite a bit longer than expected. Our coworkers tell us that the area is about to flip to nighttime pricing so the several drivers we summon put us off to wait for the pricing to go up. Eventually, a driver arrives and we say goodbye to the kind people who have accompanied us all afternoon.
On the way back, we notice the driver has selected one of the longer routes that Google offered him. He also stops at one point to pee by the side of the road. So basically just like Uber. Our hour-long ride only costs 365 rupees (about $5.60 USD), so taking the longer route isn’t as annoying as it could have been.
A quick dinner, then a shower and I am ready to head to the airport. During the car ride, I think of how much I like this place. The people have been incredibly kind. The food has (mostly) been good, and the perspective has been invaluable.
I close my eyes and sink into the seat. As we hit potholes and soar over speed bumps, I let myself melt into the motion like I’m riding a wave. Soon, I’m asleep.
A few months ago, a recruiter sent me a LinkedIn message with a link to a recruitment video. I’ve been seeing more of these lately, but this one was particularly impressive at how quickly it turned me off from the company.
It described their idea of a tech utopia, a place where developers could use whatever technology they desired.
“Read a blog post about something new in the morning. Deploy it to prod in the afternoon.” said a developer vibrating from over-caffeination.
“Pick the tools and languages you want. Every team is using something different.”
“We’re on the cutting edge, innovating like crazy, using tech you won’t see anywhere else.”
Everything about the culture they described seemed terrible. Sure, the people looked happy and engaged, but what the video communicated to me was “This is a portal into a chaotic hellscape of pain and suffering. God help their Ops team.”
These were kids playing with toys: not developers. Look closely, and you’ll see one or two of these folks in most dev groups. They pick their tools based on the latest blog post they’ve read, chasing technologies like a brain-damaged squirrel.
They’re either not in the on-call rotation or are OK with not having a life outside of work. The things they build are always fragile or in some way broken.
Corralled by a good manager, these guys (and they are always guys) can be solid members of the team who do, in fact, drive innovation and get people thinking about new ways of doing things. Left to their own devices, though, as they often are, and pretty soon the building will be on fire. Granted, it may be a spectacular fire.
Sometimes, this shiny-penny chasing is actively encouraged (as at the business in the recruitment video). These are places where leadership and staff have confused tooling innovation with business innovation. They have a culture where no one ever asks “Will rewriting our entire app in Haskell give us a competitive advantage?” (The answer to this is always no.)
Nor do they ask:
Who else is using this?
Where would we get support?
Is it well documented?
Does this actually do what we think it will do?
And so on.
I’m not arguing that people shouldn’t get excited about new technologies, just that there needs to be some prudence when picking tools. It’s better to innovate in the business, in the product and processes, not the tooling.
For the business to succeed (and for you to have a job long-term), it’s better to use proven technologies that have had time to bake, develop community support, and for the crazy early adopters to figure out production problems. “Well-documented and supported” is way more important than shiny.
So is “right”. Established tools with a lot of hype that don’t fit the problem being solved are just as dangerous as the new-born ones.
If you’re on a team with or managing some tinkerers, ask questions and help guide them towards good decisions. Make them think through scenarios like “I just read about MongoDB and it’s awesome. We should use Mongo!” and arrive at “All of our data is relational. We should NOT use Mongo.”
Help them think like a carpenter. Using an old hammer doesn’t keep you from building an awesome house. In fact, having tools you know and can trust will let you take risks elsewhere and push boundaries you might not have otherwise been able to if all your tools are in 15 different variations of “on fire”.
Play with new tech. I’d even argue that there needs to be sprint time dedicated to testing and playing with new stuff. Keep pushing forward and learning, but for what you put in production, pick the boring stuff, the tech you know you can count on.
Resources like Thoughtworks’ Tech Radar are really useful for figuring out which technologies are in the sweet spot of new enough to be relevant but established enough that you won’t be out on an island. If you’re in the enterprise space, resources like Gartner are also handy.
This is the stuff that separates kids from adults, and the successful from the burnt-out husks. Choosing technologies responsibly isn’t sexy or exciting, but it’s wise, which is discounted far too often. It also shows kindness to your teammates and co-workers – the people who are ultimately going to have to deal with the downstream effects of the decisions you make.
I previously wrote a python lambda function that copies AWS RDS snapshots to a different region. This has been working for months but recently started throwing this obfuscated error:
An error occurred (DBSnapshotAlreadyExists) when calling the CopyDBSnapshot operation: Cannot create the snapshot because a snapshot with the identifier copy-snap-of-TIMESTAMP-DB_NAME already exists.
Thinking this might be due to some timestamp shenanigans, I looked at the Cloudwatch Events trigger for the lambda and saw that there were two triggers instead of the original one that I setup. Both were scheduled for the same time. I deleted the new one and waited until the next day to see if the error re-occurred, which it did.
Looking through the Cloudwatch errors, even though the second trigger was gone, the lambda was still trying to execute twice. I’ve filed a support ticket with AWS, but in the meantime, needed to silence the false positives to keep people from getting paged.
The error handling had been done as:
except botocore.exceptions.ClientError as e:
raise Exception("Could not issue copy command: %s" % e)
Initially, I tried:
except botocore.exceptions.ClientError as e:
if 'DBSnapshotAlreadyExists' in e:
raise Exception("Could not issue copy command: %s" % e)
Instead, I had to do:
except botocore.exceptions.ClientError as e:
if e.response['Error']['Code'] == 'DBSnapshotAlreadyExists':
raise Exception("Could not issue copy command: %s" % e)
I’ve been trying to flesh out my Python knowledge and learn more about machine learning(ML) in the process. Most of my day-to-day Python use is focused on text manipulation, API calls, and JSON parsing, so leveling up on ML is more math (specifically stat-related) than I’m used to.
Today I played around with the NumPy Python package a bit and figured out some simple things.
For example, if I wanted to multiply the numbers in two lists with vanilla Python, like this:
a = [1, 2, 3, 4]
b = [4, 3, 2, 1]
print(a * b)
I’d get TypeError: can’t multiply sequence by non-int of type ‘list’ . I’d have to write something to iterate through each list.
NumPy, on the other hand, can handle this like a champ. And this is probably the simplest thing you could use it for.
And then it’s just turtles all the way down. You can slice intersections, calculate standard deviations, and so on. It’s a handy Python package that I knew literally nothing about prior today and a nice tool to add to the toolbox.