What I Gained from the Past Month
The greatest reward, naturally, was my interview with old Joe — it was priceless. I published an article about it a while back: joe. Those who haven’t read it can click through. I hope that at sixty I’ll be as wise and as clear-eyed about the world as the old man, so that anyone chatting with me comes away with that same feeling — just as Frodo, gazing into the future, praised Sam: Tyr the thinker.
The Code Beam Talk
At this Code Beam I gave a thirty-some-minute talk: Release, Deploy, Upgrade, Monitor Elixir Services in Real World. The title is absurdly long. You can grab the slides from my GitHub: tyrchen/unchained. Video: https://www.youtube.com/watch?v=kwHtlmyxE6c (you can also click “Read the original” to get it, but you’ll need a ladder).
I revised the script five times, and ten minutes before going on stage I was still making final edits. The entire script was hand-drawn in Notability — the crowning achievement of my three months with the app. I showed my first draft to old Joe and even talked him into buying a copy himself. As for my Notability skills, I wouldn’t claim mastery of the sublime, but I’m certainly top-tier — I control layout better, my use of color has grown steadily more consistent (forming a system of my own), my composition keeps improving so I can express my intent more clearly, and I’ve built all sorts of templates for different occasions. Most importantly — hand-drawing is no longer a bottleneck; most of the time my hand-drawn work in Notability is considerably more efficient than using Grafio or Gliffy. If people are interested, I’ll write a piece one day on using Notability + iPencil to boost productivity.
I did two rehearsals. One was a BBL for the US Eng team; the other an elixir knowledge share for the China elixir team. Two rehearsals gave me a decent handle on the content, the key points, and the timing. The rehearsal differed considerably from my final talk, thanks to Joe’s pointers. Much as I can recite “Great leaders start with why” backwards and forwards, when it came to my own important talk I still didn’t give “why” enough weight. If Joe hadn’t pushed back, I might have delivered a thoroughly mediocre talk. “What problem have you solved? Why does it matter to me?” is a question every speaker should take seriously and answer first.
The talk itself still had plenty of flaws — fussy gestures, eyes glued too much to the script instead of the audience, confined to one spot on the stage the whole time without much movement, an outfit that could have been sharper, language that could be more concise and forceful, fewer ums and ahs, more pauses after emphatic sentences. But for a first public English talk, I think it was pretty good — I’d give myself 80. No thousand-mile journey without accumulating small steps; take it slow.
Erlang VM Training
At this Code Beam I also attended the Erlang VM training that Saturday. The instructor was Erik Stenman, author of the Beam Book (I only started following him because I read his book). Dr. Erik looks like a harmless, jolly engineer, but he’s astonishingly brilliant: during his PhD he worked on Erlang’s HiPE project (Erlang Native Compiler), improving Erlang’s execution efficiency by 10–50x; as a postdoc he helped develop the first version of the Scala compiler, delivering Scala’s first usable compiler. A titan like this charges only $300 a day for training — where else would you find such a deal? Last year I wrote an article, “What Am I Doing When I’m in Training.” I don’t know how many people read it, or how many actually applied it strategically to their work and life.
For training at a conference of this caliber, the topic itself hardly matters. What matters is which instructor you choose and what you hope to get from them. Don’t treat training as training — treat it as a cheap, peer-to-peer consulting opportunity. $300 a day to consult Erik Stenman — that’s a fire-sale price that’ll have you laughing till you cry. My friend Xiaoshan clearly took inspiration from that article: this time he spent three days in old Joe’s training, over a thousand dollars, learned a great deal of life wisdom from the old man (I saw his notebook was full of crib notes), and got thoroughly friendly with him. My own two-hour one-on-one with the old man this time came from his strong recommendation.
So: make good use of training. Make good use of the instructor’s time. Spend a little time learning about the instructor, do your homework, figure out what you want to get out of it, or what you can get out of it. Persuade your boss to reimburse the training fee (a boss who won’t even cover this — you can fire him), then write down every question you want to ask, and use the breaks in the training (over a day, tea breaks + lunch + Q&A will always yield two or three hours) to do some proper “consulting.” :)
This time I’d planned to walk through the Erlang emulator source code before the training and accumulate a pile of questions for Erik, but squeezed between the conference and a big company project, I was exhausted in body and mind, with no energy to read code. I could only skim the Beam Book, ask a few questions, and then chase Erik down to talk at length about aeternity.
Last year I took OTP training and got to know Francesco, the head of Erlang Solutions; we built a solid connection — before my Beijing business trip I even invited him to visit Tubi HQ and discuss Hive and the Overseer I’d built. Once this busy stretch is over, I’ll reconnect with Erik too and learn from him properly about aebytecode (the compiler inside ae that handles smart contracts).
The Data Center Migration
At Tubi, the past few months we’ve been consumed by one big project: migrating data centers. Moving all services and data from one data center to another — companies bigger than us either don’t need to do it or have a dedicated team for it; companies smaller than us either haven’t hit this headache yet or designed things well enough from the start to avoid it. We’re neither big nor small, and we carry years of technical debt, so our engineers got the chance to face an unprecedented challenge — one they may never encounter a second time in their lives. Truly a gift from heaven. A data center migration isn’t really an intellectual exercise; it’s more a test of stamina, resilience, and coordination. Several wildly different database services, dozens of microservices, hundreds of machines, and a long action list — all migrated by about ten backend engineers within a few hours during off-peak traffic, while minimizing impact on users and ad revenue. It was nothing short of changing the engine on an airplane in mid-flight. Months of effort poured into a single decisive battle; the pressure each person bore was extraordinary.
In the early hours of Thursday two weeks ago — two days before my business trip — our backend team burned its boats and pulled off the feat. I’ll write a separate post summarizing the details when I get the chance. Here I’ll note a few takeaways.
The Book of Rites, Doctrine of the Mean, says: “In all things, preparation brings success; lack of preparation brings failure. If words are settled beforehand, there is no stumbling; if affairs are settled beforehand, there is no distress; if conduct is settled beforehand, there is no regret; if the way is settled beforehand, there is no exhaustion.” If we plan and prepare thoroughly before acting, the odds of finding ourselves with regrets beyond remedy and nowhere to turn drop enormously. The data center migration was grinding, exhausting work; success hinged on thinking through every last detail. Which services do we have, which are external, which internal, which third-party, which in-house, who owns each, what state each is in, what its ETA is — such a responsibility matrix must be built. How many phases the migration has, what to do in the hours before, the order of DNS cutover during the migration, what to verify afterward, and what the contingency plan is if something goes wrong. Like chess — think each move through as far as possible beforehand, so you don’t end up frantic and error-prone when the moment comes.
Even if the new data center tested clean, we still needed to route some production traffic to verify — I’d been worried all along: the new data center used a new domain, abc.com, while all clients use tubitv.com. Even if every service under abc.com worked perfectly, we could hardly guarantee that nothing would break when tubitv.com pointed to the corresponding services on abc.com. The trouble with Tubi TV’s whole business is that the clients are legion — some legacy clients are black boxes even to us. So we had to come up with a very hacky way to test: we switched the company HQ’s DNS server over to one we built ourselves with bind9. This server hijacks all subdomains of tubitv.com, pointing them at the new data center, while forwarding all other DNS requests. This approach helped us uncover several severe bugs we had never anticipated.
The Blockchain Talk
Every business trip I try to organize an event and share something. This trip was a week long; I meant to keep it simple — do a Zhihu Live on bitcoin — because ever since I wrote about my internal Tubi talk, many readers wanted me to give a public presentation. Unfortunately, the Zhihu Live editor was obstinate, insisting I prove I’m an authority in the field or they wouldn’t approve it. I don’t think I’m an authority at all — it’s been less than three months since I seriously started studying blockchain technology, and there’s no comparing my experience to people who got into the field three or five years ago. But from my recent observations of the Chinese-language community, those who truly dig deep into blockchain technology and are willing to share are few and far between. This isn’t surprising: 1) the restless hype of the blockchain world makes status in the community measured in money — or rather, the ability to rake it in — so the voices of those genuinely deep in the technology get drowned out; 2) a good technical expert isn’t necessarily a good communicator — just as there are seas of people who code better than I do, but few who, like me, have written a technical WeChat account for four years and still have more to say. So even if I’m no authority, I believe what I have to say can still inspire most people.
Alas, things went against my wishes — I had to prove I was an authority, and in the end I couldn’t prove myself one in the eyes of the Zhihu editor. I submitted the repos for my blockchain code and talks on GitHub; I have no idea whether the editor even knows how to use GitHub — anyway, I updated twice and was rejected twice. In a fit of pique I deleted the Live. I have no time for this cat-and-mouse game. If it can’t be online, then offline it is — so on a friend’s recommendation I got in touch with the people at Yunxiangke and set up an in-person talk.
Allow me to invoke once more the words of the esteemed Prof. Randy: The brick walls are not there to keep us out; the brick walls are there to give us a chance to show how badly we want something.
Hitting a wall on Zhihu provoked my stubborn pride — if I’m not qualified for a one-hour talk, then how about I just do an entire afternoon? I nailed down the details with Helen at Yunxiangke: no guests, just me talking solo. From 1:00 to 5:00 — four hours, with two breaks. Four hours of stand-up comedy — something I’d never attempted.
Though I had a little something in my belly, anyone who’s given a lecture knows: understanding something yourself and explaining it so others understand are not on the same order of difficulty. Every hour on stage costs at least double that off stage. Once the data center migration was done and the new system stable, I started writing slides — on trains, before bed, on planes — pouring almost all my spare time during the trip into them (except for meals with people). On Friday night, after a little sharing session at Geek Time, I returned to the hotel, and my wife asked how preparations for tomorrow’s talk were going. I smiled bitterly: only a third of the content is ready. So Friday I worked until midnight again. Saturday I got up at 5:30, looked at the remaining content, and estimated five hours would do it. After an hour of writing, the hotel network + VPN proved too much of a bottleneck, so after breakfast around 7 I went to the office to keep working. Another five hours of writing, and by 12:10 I finally finished 120 pages of slides, then skimmed through the script in a friend’s car.
In the end I started at 1:15 Saturday afternoon. Because I’d prepared reasonably well, my worst fear didn’t materialize: my content running dry before filling four hours. I ended up talking for nearly five hours — and because time ran short, I even skipped content I’d planned to cover for half an hour.
The slides can be downloaded from my GitHub repo tyrchen/unchained. I’ll keep updating them to carry more content, truly reaching my ideal: one deck in hand and blockchain mastery is within reach.
At the end of the slides I put the story of Roger Bannister, using it to illustrate that belief is a self-fulfilling prophecy (a lesson I stole from Tal Ben-Shahar’s Positive Psychology course). This was the deepest thing I took away from the entire process of giving this talk. In many things, what others think doesn’t matter; what matters is how you see things yourself. Confidence never comes from outside; it comes from your own conviction that you can do it — as Zhuangzi says in “Free and Easy Wandering”: “When the whole world praises him, he is not spurred on; when the whole world condemns him, he is not cast down.” Only then can you hold to your goal, press relentlessly forward, cutting a path through mountains and building bridges over rivers.
