October 25, 2024 • 2 min read
Summary A classic deep dive into Recurrent Neural Networks (RNNs) by Andrej Karpathy. This article brilliantly demonstrates how RNNs can learn and generate text, code, and even LaTeX math with remarkable coherence.
Key Takeaways RNNs can learn long-range dependencies in sequences Character-level models can generate surprisingly good text The model learns grammar, structure, and even code syntax Practical examples include Shakespeare, Wikipedia, Linux source code, and algebraic geometry papers Why I’m Sharing This Despite being from 2015, this remains one of the best introductions to understanding how neural networks process sequential data.
Read more →October 22, 2024 • 3 min read
Summary Ken Thompson’s legendary 1984 Turing Award lecture demonstrating how a compiler could be backdoored to insert vulnerabilities into code it compiles - including into future versions of itself - leaving no trace in the source code.
The Attack Thompson describes a three-stage attack:
Modify the compiler to recognize when it’s compiling login and insert a backdoor Modify the compiler to recognize when it’s compiling itself and insert the backdoor insertion code Remove all evidence from the source code The result: A compiler whose source appears clean but produces compromised binaries forever.
Read more →October 18, 2024 • 3 min read
Summary Eric Raymond’s seminal essay on open source software development, comparing the “cathedral” model (centralized, controlled) with the “bazaar” model (decentralized, community-driven). This essay influenced the modern open source movement.
Core Thesis “Given enough eyeballs, all bugs are shallow” - Linus’s Law
The bazaar model (exemplified by Linux) succeeds through:
Releasing early and often Delegating everything possible Being open to the point of promiscuity Treating users as co-developers Key Lessons for Developers Good software starts by scratching a developer’s personal itch Plan to throw one away; you will, anyhow If you treat your beta-testers as your most valuable resource, they will respond by becoming your most valuable resource Release early, release often, and listen to your customers Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly Historical Impact This essay:
Read more →