danluu

<p>There've been regular viral stories about ML/AI bias with LLMs and generative AI for the past couple years. One thing I find interesting about discussions of bias is how different the reaction is in the LLM and generative AI case when compared to &quot;classical&quot; bugs in cases where there's a clear bug. In particular, if you look at forums or other discussions with lay people, people frequently deny that a model which produces output that's sort of the opposite of what the user asked for is even a bug. For example, a year ago, an Asian MIT grad student asked Playground AI (PAI) to &quot;Give the girl from the original photo a professional linkedin profile photo&quot; and PAI converted her face to a white face with blue eyes.</p> <p>The top &quot;there's no bias&quot; response on the front-page reddit story, and one of the top overall comments, was</p> <blockquote> <p>Sure, now go to <a href="https://civitai.com/">the most popular Stable Diffusion model website</a> and look at…

<p>There's a common narrative that Microsoft was moribund under Steve Ballmer and then later saved by the miraculous leadership of Satya Nadella. This is the dominant narrative in every online discussion about the topic I've seen and it's a commonly expressed belief &quot;in real life&quot; as well. While I don't have anything negative to say about Nadella's leadership in this post, this narrative underrates Ballmer's role in Microsoft's success. Not only did Microsoft's financials, revenue and profit, look great under Ballmer, Microsoft under Ballmer made deep, long-term bets that set up Microsoft for success in the decades after his reign. At the time, the bets were widely panned, indicating that they weren't necessarily obvious, but we can see in retrospect that the company made very strong bets despite the criticism at the time.</p> <p>In addition to overseeing deep investments in areas that people would later credit Nadella for, Ballmer set Nadella up for success by clearing out …

<p>About eight years ago, I was playing a game of <a href="https://amzn.to/4cgpzow">Codenames</a> where the game state was such that our team would almost certainly lose if we didn't correctly guess all of our remaining words on our turn. From the given clue, we were unable to do this. Although the game is meant to be a word guessing game based on word clues, a teammate suggested that, based on the physical layout of the words that had been selected, most of the possibilities we were considering would result in patterns that were &quot;too weird&quot; and that we should pick the final word based on the location. This worked and we won.</p> <p><details closed> <summary>[Click to expand explanation of Codenames if you're not familiar with the game]</summary> Codenames is played in two teams. The game has a 5x5 grid of words, where each word is secretly owned by one of {blue team, red team, neutral, assassin}. Each team has a &quot;spymaster&quot; who knows the secret word &lt;-&gt; owne…

<p>From 2011-2012, the FTC investigated the possibility of pursuing antitrust action against Google. The FTC decided to close the investigation and not much was publicly known about what happened until Politico released 312 pages of internal FTC memos that from the investigation a decade later. As someone who works in tech, on reading the memos, the most striking thing is how one side, the side that argued to close the investigation, repeatedly displays a lack of basic understanding of tech industry and <abbr title="that are available">the memos from directors and other higher-ups</abbr> don't acknowledge that this at all.</p> <p>If you don't generally follow what regulators and legislators are saying about tech, seeing the internal c(or any other industry) when these decisions are, apparently, being made with little to no understanding of the industries<sup class="footnote-ref" id="fnref:K"><a rel="footnote" href="#fn:K">1</a></sup>.</p> <p>Inside the FTC, the Bureau of Competition …

<p><meta property="og:image" content="/slow-device-performance.png"/></p> <p>In 2017, <a href="https://danluu.com/web-bloat/">we looked at how web bloat affects users with slow connections</a>. Even in the U.S., <a href="https://twitter.com/danluu/status/1116565029791260672">many users didn't have broadband speeds</a>, making much of the web difficult to use. It's still the case that many users don't have broadband speeds, both inside and outside of the U.S. and that much of the modern web isn't usable for people with slow internet, but the exponential increase in bandwidth (Nielsen suggests <abbr title="Unfortunately, I don't know of a public source for low-end data, say 10%-ile or 1%-ile; let me know if you have numbers on this">this is 50% per year for high-end connections</abbr>) has outpaced web bloat for typical sites, making this less of a problem than it was in 2017, although it's still a serious problem for people with poor connections.</p> <p>CPU performance for web apps ha…

<p>If I ask myself a question like &quot;I'd like to buy an SD card; who do I trust to sell me a real SD card and not some fake, Amazon or my local Best Buy?&quot;, of course the answer is that I trust my local Best Buy<sup class="footnote-ref" id="fnref:C"><a rel="footnote" href="#fn:C">1</a></sup> more than Amazon, which is notorious for selling counterfeit SD cards. And if I ask who do I trust more, my local reputable electronics shop (Memory Express, B&amp;H Photo, etc.), I trust my local reputable electronics shop more. Not only are they <a href="https://www.pentaxforums.com/forums/22-pentax-camera-field-accessories/53884-counterfeit-sandisk-extreme-iii-sdhc-best-buy.html">less likely to</a> <a href="https://www.dpreview.com/forums/thread/2308411">sell me a counterfeit than Best Buy</a>, in the event that they do sell me a counterfeit, the service is likely to be better.</p> <p>Similarly, let's say I ask myself a question like, &quot;on which platform do I get a higher rate of sc…

<p>On large platforms, it's impossible to have policies on things like moderation, spam, fraud, and sexual content that people agree on. David Turner made a simple game to illustrate how difficult this is even in a trivial case, <a href="https://novehiclesinthepark.com/">No Vehicles in the Park</a>. If you haven't played it yet, I recommend playing it now before continuing to read this document.</p> <p>The idea behind the site is that it's very difficult to get people to agree on what moderation rules should apply to a platform. Even if you take a much simpler example, what vehicles should be allowed in a park given a rule and some instructions for how to interpret the rule, and then ask a small set of questions, people won't be able to agree. On doing the survey myself, one of the first reactions I had was that the questions aren't chosen to be particularly nettlesome and there are many edge cases Dave could've asked about if he wanted to make it a challenge. And yet, despite not mak…

<p>This is a set of notes on the Quinn Emanuel report on Cruise's handling of the 2023-10-02 accident where a Cruise autonomous vehicle (AV) hit a pedestrian, stopped, and then started moving again with the pedestrian stuck under the bottom of the AV, dragging the pedestrian 20 feet. After seeing some comments about this report, I read five stories on this report and then skimmed the report and my feeling is that the authors of four of the stories probably didn't read the report, and that people who were commenting had generally read stories by journalists who did not appear to read the source material, so the comments were generally way off base. <a href="https://danluu.com/dunning-kruger/">As we previously discussed, it's common for summaries to be wildly wrong, even when they're summarizing a short paper that's easily read by laypeople</a>, so of course summaries of a 200-page report are likely to be misleading at best.</p> <p>On reading the entire report, I'd say that Cruise both …

<p>There's a class of comment you often see when someone makes a popular thread on Mastodon/Twitter/Threads/etc., that you also see on videos that's basically &quot;Why make a Twitter thread? This would be better as a blog post&quot; or &quot;Why make a video? This would be better as a blog post&quot;. But, these comments are often stronger in form, such as:</p> <blockquote> <p><a href="https://news.ycombinator.com/item?id=31311904">I can't read those tweets that span pages because the users puts 5 words in each reply. I find common internet completely stupid: Twitter, tiktok, Instagram, etc. What a huge waste of energy</a>.</p> </blockquote> <p>or</p> <blockquote> <p><a href="https://news.ycombinator.com/item?id=31312967">When someone chooses to blog on twitter you know it's facile at best, and more likely simply stupid (as in this case)</a></p> </blockquote> <p>These kinds of comments are fairly common, e.g., I pulled up Foone's last 10 Twitter threads that scored 200 points or m…

<p>In <a href="https://xeiaso.net/blog/birth-death-seo/">The birth &amp; death of search engine optimization</a>, Xe suggests</p> <blockquote> <p>Here's a fun experiment to try. Take an open source project such as <code>yt-dlp</code> and try to find it from a very generic term like &quot;youtube downloader&quot;. You won't be able to find it because of all of the content farms that try to rank at the top for that term. Even though <code>yt-dlp</code> is probably actually what you want for a tool to download video from YouTube.</p> </blockquote> <p>More generally, most tech folks I'm connected to seem to think that Google search results are significantly worse than they were ten years ago (<a href="https://mastodon.social/@danluu/111506788692079608">Mastodon poll</a>, <a href="https://twitter.com/danluu/status/1730705885037801686">Twitter poll</a>, <a href="https://www.threads.net/@danluu.danluu/post/C0UnBr9vEeY">Threads poll</a>). However, there's a sizable group of vocal folks who c…

<p>This is a transcription of videos Elon Musk's appearance on stage with Dave Chapelle using OpenAI's Whisper model with some manual error corrections and annotations for crowd noise.</p> <p>As with the <a href="https://danluu.com/elon-twitter-texts/">Exhibit H Twitter text message release</a>, there are a lot of articles that quote bits of this, but the articles generally missing a lot of what happened and often paint a misleading picture of happened and the entire thing is short enough that you might as well watch or read it instead of reading someone's misleading summary. In general, the media seems to want to paint a highly unflattering picture of Elon, resulting in articles and virtual tweets that are factually incorrect. For example, it's been widely incorrectly reported that, during the &quot;I'm rich, bitch&quot; part, horns were played to drown out the crowd's booing of Elon, but the horn sounds were played when the previous person said the same thing, which was the most che…

<p>This is a scan/OCR of Exhibits H and J from the Twitter v. Musk case, with some of the conversations de-interleaved and of course converted from a fuzzy scan to text to make for easier reading.</p> <p>I did this so that I could easily read this and, after reading it, I've found that most accountings of what was said are, in one way or another, fairly misleading. Since the texts aren't all that long, if you're interested in what they said, <a href="https://danluu.com/dunning-kruger/">I would recommended that you just read the texts in their entirety</a> (to the extent they're available — the texts make it clear that some parts of conversations are simply not included) instead of reading what various journalists excerpted, which seems to sometimes be deliberately misleading because selectively quoting allows them to write a story that matches their agenda and sometimes accidentally misleading because they don't know what's interesting about the texts.</p> <p>If you want to compare t…

<p>I've been reading a lot of predictions from people who are looking to understand what problems humanity will face 10-50 years out (and sometimes longer) in order to work in areas that will be instrumental for the future and wondering how accurate these predictions of the future are. The timeframe of predictions that are so far out means that only a tiny fraction of people making those kinds of predictions today have a track record so, if we want to evaluate which predictions are plausible, we need to look at something other than track record.</p> <p>The idea behind the approach of this post was to look at predictions from an independently chosen set of predictors (Wikipedia's list of well-known futurists<sup class="footnote-ref" id="fnref:W"><a rel="footnote" href="#fn:W">1</a></sup>) whose predictions are old enough to evaluate in order to understand which prediction techniques worked and which ones didn't work, allowing us to then (mostly in a future post) evaluate the plausibili…

<p>Wave is a $1.7B company with 70 engineers<sup class="footnote-ref" id="fnref:R"><a rel="footnote" href="#fn:R">1</a></sup> whose product is a CRUD app that adds and subtracts numbers. In keeping with this, our architecture is a standard CRUD app architecture, a Python monolith on top of Postgres. <a href="https://twitter.com/danluu/status/1462607028585525249">Starting with a simple architecture and solving problems in simple ways</a> where possible has allowed us to scale to this size while engineers mostly focus on work that delivers value to users.</p> <p>Stackoverflow scaled up a monolith to good effect (<a href="https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-overflow/">2013 architecture</a> / <a href="https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/">2016 architecture</a>), eventually getting acquired for $1.8B. If we look at traffic instead of market cap, Stackoverflow is among the top 100 highest traffic sites on the inte…

<p>There's a <a href="https://danluu.com/cocktail-ideas/">cocktail party version</a> of the efficient markets hypothesis I frequently hear that's basically, &quot;markets enforce efficiency, so it's not possible that a company can have some major inefficiency and survive&quot;. <a href="https://danluu.com/tech-discrimination/">We've previously discussed Marc Andreessen's quote that tech hiring can't be inefficient here</a> and <a href="https://danluu.com/talent/">here</a>:</p> <blockquote> <p>Let's launch right into it. I think the critique that Silicon Valley companies are deliberately, systematically discriminatory is incorrect, and there are two reasons to believe that that's the case. ... No. 2, our companies are desperate for talent. Desperate. Our companies are dying for talent. They're like lying on the beach gasping because they can't get enough talented people in for these jobs. The motivation to go find talent wherever it is unbelievably high.</p> </blockquote> <p>Variants …

<p><details open> <summary>[Click to collapse / expand section on sports]</summary> Here are some notes from talent scouts:</p> <ul> <li>Recruit A: <ul> <li>... will be a real specimen with chance to have a Dave Parker body. Facially looks like Leon Wagner. Good body flexibility. Very large hands.</li> </ul></li> <li>Recruit B: <ul> <li>Outstanding physical specimen – big athletic frame with broad shoulders and long, solid arms and leg. Good bounce in his step and above avg body control. Good strong face.</li> </ul></li> <li>Recruit C: <ul> <li>Hi butt, longish arms &amp; legs, leanish torso, young colt</li> <li>[different scout]: Wiry loose good agility with good face</li> <li>[another scout]: Athletic looking body, loose, rangy, slightly bow legged.</li> </ul></li> </ul> <p>Out of context, you might think they were scouting actors or models, but these are baseball players (&quot;A&quot; is Lloyd Moseby, &quot;B&quot; is Jim Abbott, and &quot;C&quot; is Derek Jeter), ones that we…

<p><i>This was co-authored with Yao Yue</i></p> <p>This is a collection of information on severe (<code>SEV-0</code> or <code>SEV-1</code>, the most severe incident classifications) incidents at Twitter that were at least partially attributed to cache from the time Twitter started using its current incident tracking JIRA (2012) to date (2022), with one bonus incident from before 2012. Not including the bonus incident, there were 6 <code>SEV-0</code>s and 6 <code>SEV-1</code>s that were at least partially attributed to cache in the incident tracker, along with 38 less severe incidents that aren't discussed in this post.</p> <p>There are a couple reasons we want to write this down. First, historical knowledge about what happens at tech companies is lost at a fairly high rate and we think it's nice to preserve some of it. Second, we think it can be useful to look at incidents and reliability from a specific angle, putting all of the information into one place, because that can sometimes…

<p>You don't have to be at a party to see this phenomenon in action, but there's a curious thing I regularly see at parties in social circles where people value intelligence and cleverness without similarly valuing on-the-ground knowledge or intellectual rigor. People often discuss the standard trendy topics (some recent ones I've observed at multiple parties are how to build a competitor to Google search and how to solve the problem of high transit construction costs) and explain why people working in the field today are doing it wrong and then explain how they would do it instead. I occasionally have good conversations that fit that pattern (with people with very deep expertise in the field who've been working on changing the field for years), but the more common pattern is that someone with cocktail-party level knowledge of a field will give their ideas on how the field can be fixed.</p> <p>Asking people why they think their solutions would solve valuable problems in the field has …

<p><i>This is an excerpt from an internal document <b>David Mackey</b> and I co-authored in April 2019. The document is excerpted since much of the original doc was about comparing possible approaches to increasing efficency at Twitter, which is mostly information that's meaningless outside of Twitter without a large amount of additional explanation/context.</i></p> <p>At Twitter, most CPU bound services start falling over at around 50% reserved container CPU utilization and almost all services start falling over at not much more CPU utilization even though CPU bound services should, theoretically, be able to get higher CPU utilizations. Because load isn't, in general, evenly balanced across shards and the shard-level degradation in performance is so severe when we exceed 50% CPU utilization, this makes the practical limit much lower than 50% even during peak load events.</p> <p>This document will describe potential solutions to this problem. We'll start with describing why we shoul…

<p>I see a lot of essays framed as writing advice which are actually thinly veiled descriptions of how someone writes that basically say &quot;you should write how I write&quot;, e.g., <a href="https://twitter.com/danluu/status/1437539076324790274">people who write short posts say that you should write short posts</a>. As with technical topics, <a href="https://danluu.com/learn-what/">I think a lot of different things can work and what's really important is that you find a style that's suitable to you and the context you operate in</a>. <a href="https://twitter.com/danluu/status/1467235582199812097">Copying what's worked for someone else is unlikely to work for you</a>, making &quot;write how I write&quot; bad advice.</p> <p>We'll start by looking at how much variety there's been in what's worked<sup class="footnote-ref" id="fnref:P"><a rel="footnote" href="#fn:P">1</a></sup> for people, come back to what makes it so hard to copy someone else's style, and then discuss what I try to do…

<p><small><i>This is a pseudo-transcript (actual words modified to be more readable than a 100% faithful transcription) of a short lightning talk I did at Twitter a year or two ago, on pitfalls of how we use latency metrics (with the actual service names anonymized per a comms request). Since this presentation, significant progress has been made on this on the infra side, so the situation is much improved over what was presented, but I think this is still relevant since, from talking to folks at peer companies, many folks are facing similar issues.</i></small></p> <p>We frequently use <a href="https://research.google/pubs/pub40801/">tail latency</a> metrics here at Twitter. Most frequently, service owners want to get cluster-wide or Twitter-wide latency numbers for their services. Unfortunately, the numbers that service owners tend to use differ from what we'd like to measure due some historical quirks in our latency measurement setup:</p> <ul> <li>Opaque, uninstrumented, latency</li…

<p>Here's a list of errors on this blog that I think were fairly serious. While what I think of as serious is, of course, subjective, I don't think there's any reasonable way to avoid that because, e.g., I make a huge number of typos, so many that the majority of acknowledgements on many posts are for people who e-mailed or DM'ed me typo fixes.</p> <p>A list that included everything, including typos would both be uninteresting for other people to read as well as high overhead for me, which is why I've drawn the line somewhere. An example of an error I don't think of as serious is, <a href="https://danluu.com/learning-to-program/">in this post on how I learned to program</a>, I originally had the dates wrong on when the competition programmers from my high school made money (it was a couple years after I thought it was). In that case, and many others, I don't think that the date being wrong changes anything significant about the post.</p> <p>Although I'm publishing the original versio…

<p>One of the most common mistakes I see people make when looking at data is incorrectly using an overly simplified model. A specific variant of this that has derailed the majority of work roadmaps I've looked at is treating people as interchangeable, as if it doesn't matter who is doing what, as if individuals don't matter.</p> <p>Individuals matter.</p> <p>A pattern I've repeatedly seen during the roadmap creation and review process is that people will plan out the next few quarters of work and then assign some number of people to it, one person for one quarter to a project, two people for three quarters to another, etc. Nominally, this process enables teams to understand what other teams are doing and plan appropriately. I've never worked in an organization where this actually worked, where this actually enabled teams to effectively execute with dependencies on other teams.</p> <p>What I've seen happen instead is, when work starts on the projects, people will ask who's working th…

<p>Three major tools that companies have to influence behavior are incentives, process, and culture. People often mean different things when talking about these, so I'll provide an example of each so we're on the same page (if you think that I should be using a different word for the concept, feel free to mentally substitute that word).</p> <ul> <li><p>Getting people to show up to meetings on time</p> <ul> <li>Incentive: dock pay for people who are late</li> <li>Process: don't allow anyone who's late into the meeting</li> <li>Culture: people feel strongly about showing up on time</li> </ul></li> <li><p>Getting people to build complex systems</p> <ul> <li>Incentive: require complexity in promo criteria</li> <li>Process: make process for creating or executing on a work item so heavyweight that people stop doing simple work</li> <li>Culture: people enjoy building complex systems and/or building complex systems results in respect from peers and/or prestige</li> </ul></li> <li><p>Avoid…

<p>People frequently<sup class="footnote-ref" id="fnref:F"><a rel="footnote" href="#fn:F">1</a></sup> think that I'm very stupid. I don't find this surprising, since I don't mind if other people think I'm stupid, which means that I don't adjust my behavior to avoid seeming stupid, which results in people thinking that I'm stupid. Although there are some downsides to people thinking that I'm stupid, e.g., failing interviews where the interviewer very clearly thought I was stupid, I think that, overall, the upsides of being willing to look stupid have greatly outweighed the downsides.</p> <p>I don't know why this one example sticks in my head but, for me, the most memorable example of other people thinking that I'm stupid was from college. I've had numerous instances where more people thought I was stupid and also where people thought the depths of my stupidity was greater, but this one was really memorable for me.</p> <p>Back in college, there was one group of folks that, for whatever…

<p>It's common to see people advocate for learning skills that they have or using processes that they use. For example, Steve Yegge has a set of blog posts where he recommends reading compiler books and learning about compilers. His reasoning is basically that, if you understand compilers, you'll see compiler problems everywhere and will recognize all of the cases where people are solving a compiler problem without using compiler knowledge. Instead of hacking together some half-baked solution that will never work, you can apply a bit of computer science knowledge to solve the problem in a better way with less effort. That's not untrue, but it's also not a reason to study compilers in particular because you can say that about many different areas of computer science and math. Queuing theory, computer architecture, mathematical optimization, operations research, etc.</p> <p>One response to that kind of objection is to say that <a href="https://twitter.com/danluu/status/89914188276011008…

<p>A common topic of discussion among my close friends is where the bottlenecks are in our productivity and how we can execute more quickly. This is very different from what I see in my extended social circles, where people commonly say that <a href="https://twitter.com/danluu/status/1440106603093495810">velocity doesn't matter</a>. In online discussions about this, I frequently see people go a step further and assign moral valence to this, saying that it is actually bad to try to increase velocity or be more productive or work hard (see appendix for more examples).</p> <p>The top reasons I see people say that productivity doesn't matter (or is actually bad) fall into one of three buckets:</p> <ul> <li>Working on the right thing is more important than working quickly</li> <li>Speed at X doesn't matter because you don't spend much time doing X</li> <li>Thinking about productivity is bad and you should &quot;live life&quot;</li> </ul> <p>I certainly agree that working on the right thi…

<p>An alternate title for this post might be, &quot;Twitter has a kernel team!?&quot;. At this point, I've heard that surprised exclamation enough that I've lost count of the number times that's been said to me (I'd guess that it's more than ten but less than a hundred). If we look at trendy companies that are within a couple factors of two in size of Twitter (in terms of either market cap or number of engineers), they mostly don't have similar expertise, often as a result of path dependence — because they &quot;grew up&quot; in the cloud, they didn't need kernel expertise to keep the lights on the way an on prem company does. While that makes it socially understandable that people who've spent their career at younger, trendier, companies, are surprised by Twitter having a kernel team, I don't think there's a technical reason for the surprise.</p> <p>Whether or not it has kernel expertise, a company Twitter's size is going to regularly run into kernel issues, from major production inc…

<p>A question I get asked with some frequency is: why bother measuring X, why not build something instead? More bluntly, in a recent conversation with a newsletter author, his comment on some future measurement projects I wanted to do (in the same vein as other projects like <a href="https://danluu.com/keyboard-v-mouse/">keyboard vs. mouse</a>, <a href="https://danluu.com/keyboard-latency/">keyboard</a>, <a href="https://danluu.com/term-latency/">terminal</a> and <a href="https://danluu.com/input-lag/">end-to-end</a> latency measurements), delivered with a smug look and a bit contempt in the tone, was &quot;so you just want to get to the top of Hacker News?&quot;</p> <p>The implication for the former is that measuring is less valuable than building and for the latter that measuring isn't valuable at all (perhaps other than for fame), but I don't see measuring as lesser let alone worthless. If anything, because measurement is, <a href="https://twitter.com/danluu/status/1082321431109795…

<p>In the classic 1986 essay, <a href="http://worrydream.com/refs/Brooks-NoSilverBullet.pdf">No Silver Bullet</a>, Fred Brooks argued that there is, in some sense, not that much that can be done to improve programmer productivity. His line of reasoning is that programming tasks contain a core of essential/conceptual<sup class="footnote-ref" id="fnref:C"><a rel="footnote" href="#fn:C">1</a></sup> complexity that's fundamentally not amenable to attack by any potential advances in technology (such as languages or tooling). He then uses an <a href="https://en.wikipedia.org/wiki/Amdahl%27s_law">Ahmdahl's law</a> argument, saying that because 1/X of complexity is essential, it's impossible to ever get more than a factor of X improvement via technological improvements.</p> <p>Towards the end of the essay, Brooks claims that at least 1/2 (most) of complexity in programming is essential, bounding the potential improvement remaining for all technological programming innovations combined to, at …

<p>Any time you have a benchmark that gets taken seriously, some people will start gaming the benchmark. Some famous examples in computing are the CPU benchmark <a href="https://spec.org/benchmarks.html">specfp</a> and video game benchmarks. With specfp, Sun managed to increase its score on <a href="https://www.spec.org/osg/cpu2000/CFP2000/179.art/docs/179.art.html">179.art</a> (a sub-benchmark of specfp) by 12x with a compiler tweak that essentially re-wrote the benchmark kernel, which increased the Sun <a href="https://en.wikipedia.org/wiki/UltraSPARC_III">UltraSPARC</a>’s overall specfp score by 20%. At times, GPU vendors have added specialized benchmark-detecting code to their drivers that lowers image quality during benchmarking to produce higher benchmark scores. Of course, gaming the benchmark isn't unique to computing and we see people do this <a href="https://danluu.com/discontinuities/">in other fields</a>. It’s not surprising that we see this kind of behavior since improving…

<p><em>This is an archive of an old pseudonymously written post from the 90s from someone whose former pseudonym seems to have disappeared from the internet.</em></p> <p>I see that <cite>Star Trek: Voyager</cite> has added a new character, a Borg. (From the photos, I also see that they're still breeding women for breast size in the 24th century.) What ticked me off was the producer's comment (I'm paraphrasing), &quot;The addition of Seven of Nine will give us limitless story possibilities.&quot;</p> <p>Uh-huh. Riiiiiight.</p> <p>Look, they did't recognize the stories they <i>had</i>. I watched the first few episodes of <cite>Voyager</cite> and quit when my bullshit meter when off the scale. (Maybe that's not fair, to judge them by only a few episodes. But it's not fair to subject me to crap like the holographic lungs, either.)</p> <p>For those of you who don't watch <cite>Star Trek: Voyager</cite>, the premise is that the <i>Voyager</i>, sort of a space corvette, gets transported u…

<p>A lot of people seem to think that distributed tracing isn't useful, or at least not without extreme effort that isn't worth it for companies smaller than FB. For example, <a href="https://twitter.com/theatrus/status/1220205677672452097">here are</a> <a href="https://twitter.com/mattklein123/status/1049813546077323264">a couple of</a> public conversations that sound like a number of private conversations I've had. Sure, there's value somewhere, <a href="https://twitter.com/mattklein123/status/1220150248741326855">but it costs too much to unlock</a>.</p> <p>I think this overestimates how much work it is to get a lot of value from tracing. At Twitter, Rebecca Isaacs was able to lay out a vision for how to get value from tracing and executed on it (with help from a number other folks, including Jonathan Simms, Yuri Vishnevsky, Ruben Oanta, Dave Rusek, Hamdi Allam, and many others<sup class="footnote-ref" id="fnref:O"><a rel="footnote" href="#fn:O">1</a></sup>) such that the work easil…

<p>We spent one day<sup class="footnote-ref" id="fnref:D"><a rel="footnote" href="#fn:D">1</a></sup> building a system that immediately found a mid 7 figure optimization (which ended up shipping). In the first year, we shipped mid 8 figures per year worth of cost savings as a result. The key feature this system introduces is the ability to query metrics data across all hosts and all services and over any period of time (since inception), so we've called it LongTermMetrics (LTM) internally since I like boring, descriptive, names.</p> <p>This got started when I was looking for a starter project that would both help me understand the Twitter infra stack and also have some easily quantifiable value. Andy Wilcox suggested looking at <a href="https://stackoverflow.com/a/37102630/334816">JVM survivor space</a> utilization for some large services. If you're not familiar with what survivor space is, you can think of it as a configurable, fixed-size buffer, in the JVM (at least if you use the G…

<p>I've been comparing notes with people who run corporate engineering blogs and one thing that I think is curious is that it's pretty common for my personal blog to get more traffic than the entire corp eng blog for a company with a nine to ten figure valuation and it's not uncommon for my blog to get an order of magnitude more traffic.</p> <p>I think this is odd because tech companies in that class often have hundreds to thousands of employees. They're overwhelmingly likely to be better equipped to write a compelling blog than I am and companies get a lot more value from having a compelling blog than I do.</p> <p>With respect to the former, employees of the company will have done more interesting engineering work, have more fun stories, and have more in-depth knowledge than any one person who has a personal blog. On the latter, my blog helps me with job searching and it helps companies hire. But I only need one job, so more exposure, at best, gets me a slightly better job, whereas …

<p><a href="https://www.xkcd.com/1795/">My hobby</a>: opening up <a href="https://danluu.com/mcilroy-unix/">McIlroy’s UNIX philosophy</a> on one monitor while reading manpages on the other.</p> <p>The first of McIlroy's dicta is often paraphrased as &quot;do one thing and do it well&quot;, which is <a href="https://danluu.com/mcilroy-unix/">shortened from</a> &quot;Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new 'features.'&quot;</p> <p>McIlroy's example of this dictum is:</p> <blockquote> <p>Surprising to outsiders is the fact that UNIX compilers produce no listings: printing can be done better and more flexibly by a separate program.</p> </blockquote> <p>If you open up a manpage for <code>ls</code> on mac, you’ll see that it starts with</p> <pre><code>ls [-ABCFGHLOPRSTUW@abcdefghiklmnopqrstuwx1] [file ...] </code></pre> <p>That is, the one-letter flags to <code>ls</code> include every lowercase letter except f…

<p>If you read any personal finance forums late last year, there's a decent chance you ran across a question from someone who was desperately trying to lose money before the end of the year. There are a number of ways someone could do this; one commonly suggested scheme was to buy <a href="https://en.wikipedia.org/wiki/Put_option">put options that were expected to expire worthless</a>, allowing the buyer to (probably) take a loss.</p> <p>One reason people were looking for ways to lose money was that, in the U.S., there's <a href="https://en.wikipedia.org/wiki/Patient_Protection_and_Affordable_Care_Act#Subsidy_Cliff_at_400%_FPL">a hard income cutoff for a health insurance subsidy</a> at $48,560 for individuals (higher for larger households; $100,400 for a family of four). There are a number of factors that can cause the details to vary (age, location, household size, type of plan), but across all circumstances, it wouldn't have been uncommon for an individual going from one side of the…

<p>Reaching 95%-ile isn't very impressive because it's not that hard to do. I think this is one of my most ridiculable ideas. It doesn't help that, when stated nakedly, that sounds elitist. But I think it's just the opposite: most people can become (relatively) good at most things.</p> <p>Note that when I say 95%-ile, I mean 95%-ile among people who participate, not all people (for many activities, just doing it at all makes you 99%-ile or above across all people). I'm also not referring to 95%-ile among people who practice regularly. The &quot;<a href="https://en.wikipedia.org/wiki/One_weird_trick_advertisements">one weird trick</a>&quot; is that, for a lot of activities, being something like 10%-ile among people who practice can make you something like 90%-ile or 99%-ile among people who participate.</p> <p>This post is going to refer to specifics since <a href="https://twitter.com/danluu/status/926493613097439232">the discussions I've seen about this are all in the abstract</a>, w…

<p>When I ask people at trendy big tech companies why algorithms quizzes are mandatory, the most common answer I get is something like &quot;we have so much scale, we can't afford to have someone accidentally write an <code>O(n^2)</code> algorithm and bring the site down&quot;<sup class="footnote-ref" id="fnref:A"><a rel="footnote" href="#fn:A">1</a></sup>. One thing I find funny about this is, even though a decent fraction of the value I've provided for companies has been solving phone-screen level algorithms problems on the job, I can't pass algorithms interviews! When I say that, people often think I mean that I fail half my interviews or something. It's more than half.</p> <p>When I wrote a draft blog post of my interview experiences, draft readers panned it as too boring and repetitive because I'd failed too many interviews. I should summarize my failures as a table because no one's going to want to read a 10k word blog post that's just a series of failures, they said (which is …

<p><small><em>This is a psuedo-transcript for a talk given at Deconstruct 2019. <a href="//danluu.com/web-bloat/">To make this accessible for people on slow connections</a> as well as people using screen readers, the slides have been replaced by in-line text (the talk has ~120 slides; at an average of 20 kB per slide, that's 2.4 MB. If you think that's trivial, consider that <a href="https://blogs.microsoft.com/on-the-issues/2019/04/08/its-time-for-a-new-approach-for-mapping-broadband-data-to-better-serve-americans/">half of Americans still aren't on broadband</a> and the situation is much worse in developing countries.</em></small></p> <p>Let's talk about files! Most developers seem to think that files are easy. Just for example, let's take a look at the top reddit r/programming comments from when Dropbox announced that they were only going to support ext4 on Linux (the most widely used Linux filesystem). For people not familiar with reddit r/programming, I suspect r/programming is t…

<p>A recurring discussion in Overwatch (as well as other online games) is whether or not women are treated differently from men. If you do a quick search, you can find hundreds of discussions about this, some of which have well over a thousand comments. These discussions tend to go the same way and involve the same debate every time, with the same points being made on both sides. Just for example, <a href="https://www.reddit.com/r/Overwatch/comments/8hvmih/the_girl_problem_an_open_letter_to_the_overwatch/">these</a> <a href="https://www.reddit.com/r/Overwatch/comments/8i4wvs/a_response_to_the_girl_problem_post_moral/">three</a> <a href="https://www.reddit.com/r/Overwatch/comments/8i7cbx/when_we_call_talking_about_sexism_in_overwatch/">threads</a> on reddit that spun out of a single post that have a total of 10.4k comments. On one side, you have people saying &quot;sure, women get trash talked, but I'm a dude and I get trash talked, everyone gets trash talked there's no difference&quot;…

<p><small><em>This is an archive of the original &quot;fsyncgate&quot; email thread. This is posted here because I wanted to have a link that would fit on a slide for <a href="//danluu.com/deconstruct-files/">a talk on file safety</a> with <a href="//danluu.com/web-bloat/">a mobile-friendly non-bloated format</a>.</em></small></p> <pre><code>From:Craig Ringer &lt;craig(at)2ndquadrant(dot)com&gt; Subject:Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS Date:2018-03-28 02:23:46 </code></pre> <p>Hi all</p> <p>Some time ago I ran into an issue where a user encountered data corruption after a storage error. PostgreSQL played a part in that corruption by allowing checkpoint what should've been a fatal error.</p> <p>TL;DR: Pg should PANIC on fsync() EIO return. Retrying fsync() is not OK at least on Linux. When fsync() returns success it means &quot;all writes since the last fsync have hit disk&quot; but we assume it means &quot;all writes since th…

<p>I've had this nagging feeling that the computers I use today feel slower than the computers I used as a kid. As a rule, I don’t trust this kind of feeling because human perception has been shown to be unreliable in empirical studies, so I carried around a high-speed camera and measured the response latency of devices I’ve run into in the past few months. Here are the results:</p> <p><style>table {border-collapse:collapse;margin:0px auto;}table,th,td {border: 1px solid black;}td {text-align:center;}td.l {text-align:left;}</style> <table> <tr> <th>computer</th><th>latency<br>(ms)</th><th>year</th><th>clock</th><th># T</th></tr> <tr> <td class="l"><a href="https://en.wikipedia.org/wiki/Apple_IIe">apple 2e</a></td><td bgcolor=#ffffcc><font color=black>30</td><td bgcolor=#54278f><font color=white>1983</font></td><td bgcolor=#08306b><font color=white>1 MHz</font></td><td bgcolor=#08306b><font color=white>3.5k</font></td></tr> <tr> <td class="l"><a href="https://www.amazon.com/gp/product/…

<p>A statement I commonly hear in tech-utopian circles is that some seeming inefficiency can’t actually be inefficient because the market is efficient and inefficiencies will quickly be eliminated. A contentious example of this is <a href="//danluu.com/tech-discrimination/">the claim that companies can’t be discriminating because the market is too competitive to tolerate discrimination</a>. A less contentious example is that when you see a big company doing something that seems bizarrely inefficient, maybe it’s not inefficient and you just lack the information necessary to understand why the decision was efficient. These kinds of statements are often accompanied by statements about how &quot;incentives matter&quot; or the CEO has &quot;skin in the game&quot; whereas the commentator does not.</p> <p>Unfortunately, arguments like this are difficult to settle because, even in retrospect, it’s usually not possible to get enough information to determine the precise “value” of a decision. E…

<p>It's common knowledge that Android device tend to be more out of date than iOS devices, but what does this actually mean? Let’s look at android marketshare data to see how old devices in the wild are. The x axis of the plot below is date, and the y axis is Android marketshare. The share of all devices sums to 100% (with some artifacts because the public data Google provides is low precision).</p> <p>Color indicates age:</p> <ul> <li>blue: current (API major version)</li> <li>yellow: 6 months</li> <li>orange: 1 year</li> <li>dark red: 2 years</li> <li>bright red/white: 3 years</li> <li>light grey: 4 years</li> <li>grey: 5 years</li> <li>black: 6 years or more</li> </ul> <p>If we look at the graph, we see a number of reverse-S shaped contours; between each pair of contours, devices get older as we go from left to right. Each contour corresponds to the release of a new android version and the associated devices running that android version. As time passes, devices on that version ge…

<p>About once a month, an app that I regularly use will change its UI in a way that breaks muscle memory, basically tricking the user into doing things they don’t want.</p> <h3 id="zulip">Zulip</h3> <p>In recent memory, Zulip (a slack competitor) changed its newline behavior so that <code>ctrl + enter</code> sends a message instead of inserting a new line. After this change, I sent a number of half-baked messages and it seemed like some other people did too.</p> <p>Around the time they made that change, they made another change such that a series of clicks that would cause you to send a private message to someone would instead cause you to send a private message to the alphabetically first person who was online. Most people didn’t notice that this was a change, but when I mentioned that this had happened to me a few times in the past couple weeks, multiple people immediately said that the exact same thing happened to them. Some people also mentioned that the behavior of navigation s…

<p>We’re going to reproduce some <a href="//danluu.com/file-consistency/">results from papers on filesystem robustness that were written up roughly a decade ago</a>: <a href="http://research.cs.wisc.edu/wind/Publications/iron-sosp05.pdf">Prabhakaran et al. SOSP 05 paper</a>, which injected errors below the filesystem and <a href="https://www.usenix.org/legacy/event/fast08/tech/full_papers/gunawi/gunawi_html/index.html">Gunawi et al. FAST 08</a>, which looked at how often filesystems failed to check return codes of functions that can return errors.</p> <p><a href="http://research.cs.wisc.edu/wind/Publications/iron-sosp05.pdf">Prabhakaran et al.</a> injected errors at the block device level (just underneath the filesystem) and found that <code>ext3</code>, <code>resierfs</code>, <code>ntfs</code>, and <code>jfs</code> mostly handled read errors reasonbly but <code>ext3</code>, <code>ntfs</code>, and <code>jfs</code> mostly ignored write errors. While the paper is interesting, someone i…

<p>If you look at “gaming&quot; keyboards, a lot of them sell for $100 or more on the promise that they’re fast. Ad copy that you’ll see includes:</p> <ul> <li>a custom designed keycap that has been made shorter to reduce the time it takes for your actions to register</li> <li>8x FASTER - Polling Rate of 1000Hz: Response time 0.1 milliseconds</li> <li>Wield the ultimate performance advantage over your opponents with light operation 45g key switches and an actuation 40% faster than standard Cherry MX Red switches</li> <li>World's Fastest Ultra Polling 1000Hz</li> <li>World's Fastest Gaming Keyboard, 1000Hz Polling Rate, 0.001 Second Response Time</li> </ul> <p>Despite all of these claims, I can only find <a href="http://forums.blurbusters.com/viewtopic.php?f=10&amp;t=1836">one person who’s publicly benchmarked keyboard latency</a> and they only tested two keyboards. In general, my belief is that if someone makes performance claims without benchmarks, the claims probably aren’t true, …

<p><em>This is a pseudo-transcript for a talk on branch prediction given at Two Sigma on 8/22/2017 to kick off &quot;localhost&quot;, a talk series organized by <a href="https://www.recurse.com/scout/click?t=b504af89e87b77920c9b60b2a1f6d5e8">RC</a>.</em></p> <p>How many of you use branches in your code? Could you please raise your hand if you use if statements or pattern matching?</p> <p><code>Most of the audience raises their hands</code></p> <p>I won’t ask you to raise your hands for this next part, but my guess is that if I asked, how many of you feel like you have a good understanding of what your CPU does when it executes a branch and what the performance implications are, and how many of you feel like you could understand a modern paper on branch prediction, fewer people would raise their hands.</p> <p>The purpose of this talk is to explain how and why CPUs do “branch prediction” and then explain enough about classic branch prediction algorithms that you could read a modern p…

<p>I recently had a problem where part of the solution was to do a series of pointer accesses that would walk around a chunk of memory in pseudo-random order. Sattolo's algorithm provides a solution to this because it produces a permutation of a list with exactly one cycle, which guarantees that we will reach every element of the list even though we're traversing it in random order.</p> <p>However, the explanations of why the algorithm worked that I could find online either used some kind of mathematical machinery (Stirling numbers, assuming familiarity with cycle notation, etc.), or used logic that was hard for me to follow. I find that this is common for explanations of concepts that could, but don't have to, use a lot of mathematical machinery. I don't think there's anything wrong with using existing mathematical methods per se -- it's a nice mental shortcut if you're familiar with the concepts. If you're taking a combinatorics class, it makes sense to cover Stirling numbers and th…

<p>There’s <a href="https://www.youtube.com/watch?v=vOvQCPLkPt4">a great MSR demo from 2012 that shows the effect of latency on the experience of using a tablet</a>. If you don’t want to watch the three minute video, they basically created a device which could simulate arbitrary latencies down to a fraction of a millisecond. At 100ms (1/10th of a second), which is typical of consumer tablets, the experience is terrible. At 10ms (1/100th of a second), the latency is noticeable, but the experience is ok, and at &lt; 1ms the experience is great, as good as pen and paper. If you want to see a mini version of this for yourself, you can try a random Android tablet with a stylus vs. the current generation iPad Pro with the Apple stylus. The Apple device has well above 10ms end-to-end latency, but the difference is still quite dramatic -- it’s enough that I’ll actually use the new iPad Pro to take notes or draw diagrams, whereas I find Android tablets unbearable as a pen-and-paper replacement.…

<p>Which is faster, keyboard or mouse? A large number of programmers believe that the keyboard is faster for all (programming-related) tasks. However, there are a few widely cited webpages on AskTog which claim that Apple studies show that using the mouse is faster than using the keyboard for everything and that people who think that using the keyboard is faster are just deluding themselves. This might sound extreme, but, just for example, one page says that the author has “never seen [the keyboard] outperform the mouse”.</p> <p>But it can’t be the case that the mouse is faster for everything — almost no one is faster at clicking on an on-screen keyboard with a mouse than typing at a physical keyboard. Conversely, there are tasks for which mice are much better suited than keyboards (e.g., aiming in FPS games). For someone without an agenda, the question shouldn’t be, which is faster at all tasks, but which tasks are faster with a keyboard, which are faster with a mouse, and which are …

<p>I often talk to startups that claim that their compensation package has a higher expected value than the equivalent package at a place like Facebook, Google, Twitter, or Snapchat. One thing I don’t understand about this claim is, if the claim is true, why shouldn’t the startup go to an investor, sell their options for what they claim their options to be worth, and then pay me in cash? The non-obvious value of options combined with their volatility is a barrier for recruiting.</p> <p>Additionally, given my risk function and the risk function of VCs, this appears to be a better deal for everyone. Like most people, extra income gives me diminishing utility, but VCs have an arguably nearly linear utility in income. Moreover, even if VCs shared my risk function, because VCs hold a diversified portfolio of investments, the same options would be worth more to them than they are to me because they can diversify away downside risk much more effectively than I can. If these startups are maki…

<p>A couple years ago, I took a road trip from Wisconsin to Washington and mostly stayed in rural hotels on the way. I expected the internet in rural areas too sparse to have cable internet to be slow, but I was still surprised that a large fraction of the web was inaccessible. Some blogs with lightweight styling were readable, as were pages by academics who hadn’t updated the styling on their website since 1995. But very few commercial websites were usable (other than Google). When I measured my connection, I found that the bandwidth was roughly comparable to what I got with a 56k modem in the 90s. The latency and packetloss were significantly worse than the average day on dialup: latency varied between 500ms and 1000ms and packetloss varied between 1% and 10%. Those numbers are comparable to what I’d see on dialup on a bad day.</p> <p>Despite my connection being only a bit worse than it was in the 90s, the vast majority of the web wouldn’t load. Why shouldn’t the web work with dialu…

<p><a href="https://twitter.com/danluu/status/1433297087362486272">HN</a> <a href="https://twitter.com/altluu/status/1449285906360266754">comments</a> <a href="https://twitter.com/altluu/status/1452707802921652224">are</a> <a href="https://twitter.com/danluu/status/1584260625920032768">terrible</a>. <a href="https://twitter.com/altluu/status/1586623057019699200">On</a> <a href="https://twitter.com/danluu/status/1346392037365436418">any</a> <a href="https://twitter.com/danluu/status/1280651255275110400">topic</a> <a href="https://twitter.com/danluu/status/1498343295545581568">I’m</a> <a href="https://twitter.com/danluu/status/1512835522002972680">informed</a> <a href="https://twitter.com/danluu/status/1298477505272147968">about</a>, <a href="https://twitter.com/danluu/status/1277222715456282625">the</a> <a href="https://twitter.com/danluu/status/872676217350172673">vast</a> <a href="https://twitter.com/danluu/status/1484268111687663620">majority</a> <a href="https://twitter.com/danluu/s…

<p>There are a lot of “12 CS books every programmer must read” lists floating around out there. That's nonsense. The field is too broad for almost any topic to be required reading for all programmers, and even if a topic is that important, people's learning preferences differ too much for any book on that topic to be the best book on the topic for all people.</p> <p>This is a list of topics and books <a href="https://twitter.com/danluu/status/1574819407058325505">where I've read the book</a>, am familiar enough with the topic to say what you might get out of learning more about the topic, and have read other books and can say why you'd want to read one book over another.</p> <h3 id="algorithms-data-structures-complexity">Algorithms / Data Structures / Complexity</h3> <p>Why should you care? Well, there's the pragmatic argument: even if you never use this stuff in your job, most of the best paying companies will quiz you on this stuff in interviews. On the non-bullshit side of things…

<p>Joel Spolsky has a classic blog post on &quot;Finding Great Developers&quot; where he popularized the meme that great developers are impossible to find, a corollary of which is that if you can find someone, they're not great. Joel writes,</p> <blockquote> <p>The great software developers, indeed, the best people in every field, are quite simply never on the market.</p> <p>The average great software developer will apply for, total, maybe, four jobs in their entire career.</p> <p>...</p> <p>If you're lucky, if you're really lucky, they show up on the open job market once, when, say, their spouse decides to accept a medical internship in Anchorage and they actually send their resume out to what they think are the few places they'd like to work at in Anchorage.</p> <p>But for the most part, great developers (and this is almost a tautology) are, uh, great, (ok, it is a tautology), and, usually, prospective employers recognize their greatness quickly, which means, basically, they get…

<p>I can't think of a single large software company that doesn't regularly draw internet comments of the form “What do all the employees do? I could build their product myself.” <a href="https://bitquabit.com/post/one-which-i-call-out-hacker-news/">Benjamin Pollack</a> and <a href="https://blog.codinghorror.com/code-its-trivial/">Jeff Atwood</a> called out people who do that with Stack Overflow. But <a href="http://nickcraver.com/blog/2016/03/29/stack-overflow-the-hardware-2016-edition/">Stack Overflow is relatively obviously lean</a>, so the general response is something like “oh, sure maybe Stack Overflow is lean, but FooCorp must really be bloated”. And since most people have relatively little visibility into FooCorp, for any given value of FooCorp, that sounds like a plausible statement. After all, what product could possible require hundreds, or even thousands of engineers?</p> <p>A few years ago, in the wake of the rapgenius SEO controversy, a number of folks called for someone …

<p>Developer compensation has skyrocketed since the demise of the <a href="https://danluu.com/google-wage-fixing/">Google et al. wage-suppressing no-hire agreement</a>, to the point where compensation rivals and maybe even exceeds compensation in traditionally remunerative fields like law, consulting, etc. In software, &quot;senior&quot; dev salary at a high-paying tech company is $350k/yr, where &quot;senior&quot; can mean &quot;someone three years of out school&quot; and it's not uncommon for someone who's considered a high performing engineer to make seven figures.</p> <p>The fields have sharply bimodal income distributions. Are programmers in for the same fate? Let's see what data we can find. First, let's look at <a href="http://www.nalp.org/uploads/0812Research.pdf">data from the National Association for Law Placement</a>, which shows when legal salaries become bimodal.</p> <h3 id="lawyers-in-1991">Lawyers in 1991</h3> <p><img src="https://danluu.com/images/bimodal-compensatio…

<p>Tavish Armstrong has a great document <a href="https://github.com/tarmstrong/longcv/blob/master/bio.md">where he describes how and when he learned the programming skills he has</a>. I like this idea because I've found that the paths that people take to get into programming are much more varied than stereotypes give credit for, and I think it's useful to see that there are many possible paths into programming.</p> <p>Personally, I spent a decade working as an electrical engineer before taking a programming job. When I talk to people about this, they often want to take away a smooth narrative of my history. Maybe it's that my math background gives me tools I can apply to a lot of problems, maybe it's that my hardware background gives me a good understanding of performance and testing, or maybe it's that the combination makes me a great fit for hardware/software co-design problems. <a href="https://www.youtube.com/watch?v=RoEEDKwzNBw">People like a good narrative</a>. One narrative pe…

<p>Do concurrency bugs matter? From the literature, we know that <a href="http://danluu.com/postmortem-lessons/">most reported bugs in distributed systems</a> have really simple causes and can be caught by trivial tests, even when we only look at bugs that cause really bad failures, like loss of a cluster or data corruption. The filesystem literature echos this result -- <a href="http://danluu.com/file-consistency/">a simple checker that looks for totally unimplemented error handling can find hundreds of serious data corruption bugs</a>. Most bugs are simple, at least if you measure by bug count. But if you measure by debugging time, the story is a bit different.</p> <p>Just from personal experience, I've spent more time debugging complex non-deterministic failures than all other types of bugs combined. In fact, I've spent more time debugging some individual non-deterministic bugs (weeks or months) than on all other bug types combined. Non-deterministic bugs are rare, but they can be …

<p>This is one of those “N technical things every programmer must read” lists, except that “programmer” is way too broad a term and the styles of writing people find helpful for them are too different for any such list to contain a non-zero number of items (if you want the entire list to be helpful to everyone). So here's a list of some things you might want to read, and why you might (or might not) want to read them.</p> <h4 id="aleksey-shipilev-https-shipilev-net"><a href="https://shipilev.net/">Aleksey Shipilev</a></h4> <p>If you want to understand how the JVM really works, this is one of the best resources on the internet.</p> <h4 id="bruce-dawson-https-randomascii-wordpress-com"><a href="https://randomascii.wordpress.com/">Bruce Dawson</a></h4> <p>Performance explorations of a Windows programmer. Often implicitly has nice demonstrations of tooling that has no publicly available peer on Linux.</p> <h4 id="chip-huyen-https-huyenchip-com-blog"><a href="https://huyenchip.com/blog…

<p><a href="https://www.amazon.com/gp/product/149192912X/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&amp;tag=abroaview-20&amp;camp=1789&amp;creative=9325&amp;linkCode=as2&amp;creativeASIN=149192912X&amp;linkId=2a578b357abc8b995368a039dd517601">The book</a> starts with a story about a time Margaret Hamilton brought her young daughter with her to NASA, back in the days of the Apollo program. During a simulation mission, her daughter caused the mission to crash by pressing some keys that caused a prelaunch program to run during the simulated mission. Hamilton submitted a change request to add error checking code to prevent the error from happening again, but the request was rejected because the error case should never happen.</p> <p>On the next mission, Apollo 8, that exact error condition occurred and a potentially fatal problem that could have been prevented with a trivial check took NASA’s engineers 9 hours to resolve.</p> <p><em>This sounds familiar -- I’ve lost track of the number of dev po…

<p>An acquaintance of mine, let’s call him Mike, is looking for work after getting laid off from a contract role at Microsoft, which has happened to a lot of people I know. Like me, Mike has 11 years in industry. Unlike me, he doesn't know a lot of folks at trendy companies, so I passed his resume around to some engineers I know at companies that are desperately hiring. My engineering friends thought Mike's resume was fine, but most recruiters rejected him in the resume screening phase.</p> <p>When I asked why he was getting rejected, the typical response I got was:</p> <ol> <li>Tech experience is in irrelevant tech</li> <li>&quot;Experience is too random, with payments, mobile, data analytics, and UX.&quot;</li> <li>Contractors are generally not the strongest technically</li> </ol> <p>This response is something from a recruiter that was relayed to me through an engineer; the engineer was incredulous at the response from the recruiter. Just so we have a name, let's call this company…

<p><em>These are archived from the now defunct <a href="http://su3su2u1.tumblr.com/">su3su2u1 tumblr</a>. Since there was some controversy over su3su2u1's identity, I'll note that I am not su3su2u1 and that hosting this material is neither an endorsement nor a sign of agreement.</em></p> <h3 id="harry-potter-and-the-methods-of-rationality-full-review">Harry Potter and the Methods of Rationality full review</h3> <p>I opened up a bottle of delicious older-than-me scotch when Terry Pratchett died, and I’ve been enjoying it for much of this afternoon, so this will probably be a mess and cleaned up later.</p> <p>Out of 5 stars, I’d give HPMOR a 1.5. Now, to the review (this is almost certainly going to be long)</p> <h4 id="the-good">The good</h4> <p>HPMOR contains some legitimately clever reworkings of the canon books to fit with Yudkowsky’s modified world:</p> <p>A few examples- In HPMOR, the “interdict of Merlin” prevents wizards from writing down powerful spells, so Slytherin put …

<p>These are archived from the now defunct <a href="http://su3su2u1.tumblr.com/">su3su2u1 tumblr</a>.</p> <h3 id="a-roundabout-approach-to-quantum-mechanics">A Roundabout Approach to Quantum Mechanics</h3> <p>This will be the first post in what I hope will be a series that outlines some ideas from quantum mechanics. I will try to keep it light, and not overly math filled- which means I’m not really teaching you physics. I’m teaching you some flavor of the physics. I originally wrote here “you can’t expect to make ice cream just having tasted it,” but I think a better description might be “you can’t expect to make ice cream just having heard someone describe what it tastes like.” AND PLEASE, PLEASE PLEASE ask questions. I’m used to instant feedback on my (attempts at) teaching, so if readers aren’t getting anything out of this, I want to stop or change or something.</p> <p>Now, unfortunately I can’t start with quantum mechanics without talking about classical physics first. Mos…

<p>Perf is probably the most widely used general purpose performance debugging tool on Linux. There are multiple contenders for the #2 spot, and, like perf, they're sampling profilers. Sampling profilers are great. They tend to be easy-to-use and low-overhead compared to most alternatives. However, there are large classes of performance problems sampling profilers can't debug effectively, and those problems are becoming more important.</p> <p>For example, consider a Google search query. Below, we have a diagram of how a query is carried out. Each of the black boxes is a rack of machines and each line shows a remote procedure call (RPC) from one machine to another.</p> <p></p> <p><img src="https://danluu.com/images/intel-cat/search_query.png" width="640" height="352"></p> <p>The diagram shows a single search query coming in, which issues RPCs to over a hundred machines (shown in green), each of which delivers another set of requests to the next, lower level (shown in blue). Each req…

<p>2015 was a pretty good year for Intel. Their quarterly earnings reports exceeded expectations every quarter. They continue to be the only game in town for the serious server market, which continues to grow exponentially; from the earnings reports of the two largest cloud vendors, we can see that AWS and Azure grew by 80% and 100%, respectively. That growth has effectively offset the damage Intel has seen from the continued decline of the desktop market. For a while, it looked like <a href="//danluu.com/new-cpu-features/#update">cloud vendors might be able to avoid the Intel tax by moving their computation onto FPGAs</a>, but Intel bought one of the two serious FPGA vendors and, combined with their fab advantage, they look well positioned to dominate the high-end FPGA market the same way they've been dominating the high-end server CPU market. Also, their fine for anti-competitive practices turned out to be $1.45B, much less than the benefit they gained from their anti-competitive pra…

<p>Have you ever mentioned something that seems totally normal to you only to be greeted by surprise? Happens to me all the time when I describe something everyone at work thinks is normal. For some reason, my conversation partner's face morphs from pleasant smile to rictus of horror. Here are a few representative examples.</p> <p>There's the company that is perhaps the nicest place I've ever worked, combining the best parts of Valve and Netflix. The people are amazing and you're given near total freedom to do whatever you want. But as a side effect of the culture, they lose perhaps half of new hires in the first year, some voluntarily and some involuntarily. Totally normal, right? Here are a few more anecdotes that were considered totally normal by people in places I've worked. And often not just normal, but laudable.</p> <p>There's the company that's incredibly secretive about infrastructure. For example, there's the team that was afraid that, if they reported bugs to their hardwar…

<p>There's a meme that's been going around for a while now: you should join a startup because the money is better and the work is more technically interesting. <a href="http://paulgraham.com/wealth.html">Paul Graham says</a> that the best way to make money is to &quot;start or join a startup&quot;, which has been &quot;a reliable way to get rich for hundreds of years&quot;, and that you can &quot;compress a career's worth of earnings into a few years&quot;. Michael Arrington says that you'll <a href="https://www.jwz.org/blog/2011/11/watch-a-vc-use-my-name-to-sell-a-con/">become a part of history</a>. Joel Spolsky says that by joining a big company, you'll end up <a href="http://www.joelonsoftware.com/items/2008/05/01.html">playing foosball and begging people to look at your code</a>. Sam Altman says that if you join Microsoft, <a href="http://blog.samaltman.com/advice-for-ambitious-19-year-olds">you won't build interesting things and may not work with smart people</a>. They all claim t…

<p>I haven't used a desktop email client in years. None of them could handle the volume of email I get without at least occasionally corrupting my mailbox. Pine, Eudora, and outlook have all corrupted my inbox, forcing me to restore from backup. How is it that desktop mail clients are less reliable than gmail, even though my gmail account not only handles more email than I ever had on desktop clients, but also allows simultaneous access from multiple locations across the globe? Distributed systems have an unfair advantage, in that they can be robust against total disk failure in a way that desktop clients can't, but none of the file corruption issues I've had have been from total disk failure. Why has my experience with desktop applications been so bad?</p> <p></p> <p>Well, what sort of failures can occur? Crash consistency (maintaining consistent state even if there's a crash) is probably the easiest property to consider, since we can assume that everything, from the filesystem to t…

<p>Jeff Atwood, perhaps the most widely read programming blogger, has a post that makes <a rel="nofollow" href="http://blog.codinghorror.com/to-ecc-or-not-to-ecc/">a case against using ECC memory</a>. My read is that his major points are:</p> <ol> <li>Google didn't use ECC when they built their servers in 1999</li> <li>Most RAM errors are hard errors and not soft errors</li> <li>RAM errors are rare because hardware has improved</li> <li>If ECC were actually important, it would be used everywhere and not just servers. Paying for optional stuff like this is &quot;awfully enterprisey&quot;</li> </ol> <p></p> <p>Let's take a look at these arguments one by one:</p> <h2 id="1-google-didn-t-use-ecc-in-1999">1. Google didn't use ECC in 1999</h2> <p>Not too long after Google put these non-ECC machines into production, they realized this was a serious error and not worth the cost savings. If you think cargo culting what Google does is a good idea because it's Google, here are some things yo…

<p>In 1999, Butler Lampson gave a talk about the <a href="http://research.microsoft.com/pubs/68591/computersystemsresearch.pdf">past and future of “computer systems research”</a>. Here are his opinions from 1999 on &quot;what worked&quot;.</p> <table> <thead> <tr> <th>Yes</th> <th>Maybe</th> <th>No</th> </tr> </thead> <tbody> <tr> <td>Virtual memory</td> <td>Parallelism</td> <td>Capabilities</td> </tr> <tr> <td>Address spaces</td> <td>RISC</td> <td>Fancy type systems</td> </tr> <tr> <td>Packet nets</td> <td>Garbage collection</td> <td>Functional programming</td> </tr> <tr> <td>Objects / subtypes</td> <td>Reuse</td> <td>Formal methods</td> </tr> <tr> <td>RDB and SQL</td> <td></td> <td>Software engineering</td> </tr> <tr> <td>Transactions</td> <td></td> <td>RPC</td> </tr> <tr> <td>Bitmaps and GUIs</td> <td></td> <td>Distributed computing</td> </tr> <tr> <td>Web</td> <td></td> <td>Security</td> </tr> <tr> <td>Algorithms</td> <td></td> <td></td> </tr> </tbody> </table> <p></p> …

<p>Hardware performance “obviously” affects software performance and affects how software is optimized. For example, the fact that caches are multiple orders of magnitude faster than RAM means that <a href="http://suif.stanford.edu/papers/lam-asplos91.pdf">blocked array accesses</a> give better performance than repeatedly striding through an array.</p> <p>Something that's occasionally overlooked is that hardware performance also has profound implications for system design and architecture. <a href="https://gist.github.com/jboner/2841832">Let's look at this table of latencies that's been passed around since 2012</a>:</p> <p></p> <pre><code>Operation Latency (ns) (ms) L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 25 ns Main memory reference 100 ns Compress 1K bytes with…

<p><a href="http://csl.stanford.edu/~christos/publications/2015.heracles.isca.pdf">Typical server utilization is between 10% and 50%. Google has demonstrated 90% utilization <em>without impacting latency SLAs</em></a>. <a href="https://what-if.xkcd.com/63/">Xkcd estimated that Google owns 2 million machines</a>. If you estimate an amortized total cost of <a href="http://www.morganclaypool.com/doi/pdf/10.2200/S00516ED2V01Y201306CAC024">$4k per machine per year</a>, that's $8 billion per year. With numbers like that, even small improvements have a large impact, and this isn't a small improvement.</p> <p></p> <p>How is it possible to get 2x to 9x better utilization on the same hardware? The low end of those typical utilization numbers comes from having a service with variable demand and fixed machine allocations. Say you have 100 machines dedicated to Jenkins. Those machines might be very busy when devs are active, but they might also have 2% utilization at 3am. Dynamic allocation (swit…

<p>Every once in awhile, you hear a story like “there was a case of a 1-Gbps NIC card on a machine that suddenly was transmitting only at 1 Kbps, which then caused a chain reaction upstream in such a way that the performance of the entire workload of a 100-node cluster was crawling at a snail's pace, effectively making the system unavailable for all practical purposes”. The stories are interesting and the postmortems are fun to read, but it's not really clear how vulnerable systems are to this kind of failure or how prevalent these failures are.</p> <p>The situation reminds me of distributed systems failures before <a href="https://aphyr.com/tags/jepsen">Jepsen</a>. There are lots of anecdotal horror stories, but a common response to those is “works for me”, even when talking about systems that are now known to be fantastically broken. A handful of companies that are really serious about correctness have good tests and metrics, but they mostly don't talk about them publicly, and the g…

<p></p> <p>I try to avoid making predictions<sup class="footnote-ref" id="fnref:1"><a rel="footnote" href="#fn:1">1</a></sup>. It's a no-win proposition: if you're right, <a href="https://en.wikipedia.org/wiki/Hindsight_bias">hindsight bias</a> makes it look like you're pointing out the obvious. And most predictions are wrong. Every once in a while when someone does a review of predictions from pundits, they're almost always wrong at least as much as you'd expect from random chance, and then hindsight bias makes each prediction look hilariously bad.</p> <p>But, occasionally, you run into someone who makes pretty solid non-obvious predictions. I was re-reading some of Steve Yegge's old stuff and it turns out that he's one of those people.</p> <p>His most famous prediction is probably <a href="http://steve-yegge.blogspot.com/2007/02/next-big-language.html">the rise of JavaScript</a>. This now seems incredibly obvious in hindsight, so much so that the future laid out in Gary Bernhardt'…

<p>I love reading postmortems. They're educational, but unlike most educational docs, they tell an entertaining story. I've spent a decent chunk of time reading postmortems at both Google and Microsoft. I haven't done any kind of formal analysis on the most common causes of bad failures (yet), but there are a handful of postmortem patterns that I keep seeing over and over again.</p> <p></p> <h3 id="error-handling">Error Handling</h3> <p>Proper error handling code is hard. Bugs in error handling code are a major cause of <em>bad</em> problems. This means that the probability of having sequential bugs, where an error causes buggy error handling code to run, isn't just the independent probabilities of the individual errors multiplied. It's common to have cascading failures cause a serious outage. There's a sense in which this is obvious -- error handling is generally regarded as being hard. If I mention this to people they'll tell me how obvious it is that a disproportionate number of …

<p></p> <p>If you've followed any tech news aggregator in the past week (the week of the 24th of May, 2015), you've probably seen the story about how <a href="https://plus.google.com/+gimp/posts/cxhB1PScFpe">SourceForge</a> is taking over admin accounts for existing projects and injecting adware in installers for packages like GIMP. For anyone not following the story, SourceForge has a long history of adware laden installers, but they used to be opt-in. It appears that the process is now mandatory for many projects.</p> <p>People have been wary of SourceForge ever since they added a feature to allow projects to opt-in to adware bundling, but you could at least claim that projects are doing it by choice. But now that SourceForge is clearly being malicious, they've wiped out all of the user trust that was built up over sixteen years of operating. No clueful person is going to ever download something from SourceForge again. If search engines start penalizing SourceForge for distributing…

<p>TIL that Bell Labs and a whole lot of other websites block archive.org, not to mention most search engines. Turns out I have <a href="https://github.com/danluu/debugging-stories/issues/3">a broken website link</a> in a GitHub repo, caused by the deletion of an old webpage. When I tried to pull the original from archive.org, I found that it's not available because Bell Labs blocks the archive.org crawler in their robots.txt:</p> <p></p> <pre><code>User-agent: Googlebot User-agent: msnbot User-agent: LSgsa-crawler Disallow: /RealAudio/ Disallow: /bl-traces/ Disallow: /fast-os/ Disallow: /hidden/ Disallow: /historic/ Disallow: /incoming/ Disallow: /inferno/ Disallow: /magic/ Disallow: /netlib.depend/ Disallow: /netlib/ Disallow: /p9trace/ Disallow: /plan9/sources/ Disallow: /sources/ Disallow: /tmp/ Disallow: /tripwire/ Visit-time: 0700-1200 Request-rate: 1/5 Crawl-delay: 5 User-agent: * Disallow: / </code></pre> <p>In fact, Bell Labs not only blocks the Internet Archiver bot, it b…

<p>Boring languages are underrated. Many appear to be rated quite highly, at least if you look at market share. But even so, they're underrated. Despite the popularity of Dan McKinley's <a href="https://mcfunley.com/choose-boring-technology">&quot;choose boring technology&quot;</a> essay, boring languages are widely panned. People who use them are too (e.g., they're a target of essays by Paul Graham and Joel Spolsky, <a href="https://news.ycombinator.com/item?id=2379259">and other people have picked up a similar attitude</a>).</p> <p>A commonly used pitch for interesting languages goes something like &quot;Sure, you can get by with writing blub for boring work, which almost all programmers do, but if you did interesting work, then you'd want to use an interesting language&quot;. My feeling is that this has it backwards. When I'm doing boring work that's basically bottlenecked on the speed at which I can write boilerplate, it feels much nicer to use an interesting language (like F#), w…

<p>Here's a conversation I keep having:</p> <blockquote> <p><strong>Someone</strong>: Did you hear that Facebook/Google uses a giant monorepo? WTF!<br> <strong>Me</strong>: Yeah! It's really convenient, don't you think?<br> <strong>Someone</strong>: That's THE MOST RIDICULOUS THING I've ever heard. Don't FB and Google know what a terrible idea it is to put all your code in a single repo?<br> <strong>Me</strong>: I think engineers at FB and Google are probably familiar with using smaller repos (doesn't <a href="http://en.wikipedia.org/wiki/Junio_Hamano">Junio Hamano</a> work at Google?), and they still prefer a single huge repo for [reasons].<br> <strong>Someone</strong>: Oh that does sound pretty nice. I still think it's weird but I could see why someone would want that.<br></p> </blockquote> <p>“[reasons]” is pretty long, so I'm writing this down in order to avoid repeating the same conversation over and over again.</p> <p></p> <h3 id="simplified-organization">Simplified organizat…

<p>Why are people so concerned with hardware power consumption nowadays? Some common answers to this question are that <a href="http://www.vox.com/2015/3/9/8178213/apple-macbook-all-batteries">power is critically important for phones, tablets, and laptops</a> and that <a href="ftp://ftp.cs.utexas.edu/pub/dburger/papers/ISCA11.pdf">we can put more silicon on a modern chip than we can effectively use</a>. In 2001 <a href="http://ieeexplore.ieee.org/xpl/login.jsp?tp=&amp;arnumber=912412&amp;url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D912412">Patrick Gelsinger observed that if scaling continued at then-current rates</a>, chips would have the power density of a nuclear reactor by 2005, a rocket nozzle by 2010, and the surface of the sun by 2015, implying that power density couldn't continue on its then-current path. Although this was already fairly obvious at the time, now that it's 2015, we can be extra sure that power density didn't continue to grow at unbounded…

<p>It's really common to see claims that some meme is backed by “studies” or “science”. But when I look at the actual studies, it usually turns out that the data are opposed to the claim. Here are the last few instances of this that I've run across. </p> <h2 id="dunning-kruger">Dunning-Kruger</h2> <p>A pop-sci version of Dunning-Kruger, the most common one I see cited, is that, the less someone knows about a subject, the more they think they know. Another pop-sci version is that people who know little about something overestimate their expertise because their lack of knowledge fools them into thinking that they know more than they do. The actual claim Dunning and Kruger make is much weaker than the first pop-sci claim and, IMO, the evidence is weaker than the second claim. The original paper isn't much longer than most of the incorrect pop-sci treatments of the paper, and we can get pretty good idea of the claims by looking at the four figures included in the paper. In the graphs bel…

<p>I've been reading a lot about software testing, lately. Coming from a hardware background (CPUs and hardware accelerators), it's interesting how different software testing is. Bugs in software are much easier to fix, so it makes sense to spend a lot less effort spent on testing. Because less effort is spent on testing, methodologies differ; software testing is biased away from methods with high fixed costs, towards methods with high variable costs. But that doesn't explain all of the differences, or even most of the differences. Most of the differences come from a cultural <a href="http://en.wikipedia.org/wiki/Path_dependence">path dependence</a>, which shows how non-optimally test effort is allocated in both hardware and software.</p> <p></p> <p>I don't really know anything about software testing, but here are some notes from what I've seen at Google, on a few open source projects, and in a handful of papers and demos. Since I'm looking at software, I'm going to avoid talking abo…

<p>I've been hearing this question a lot lately, and when I do, it reminds me how much I don't know. Here are some questions this question brings to mind.</p> <p></p> <ol> <li>How does a keyboard work? Why can’t you press an arbitrary combination of three keys at once, except on fancy gaming keyboards? That implies something about how key presses are detected/encoded.</li> <li>How are keys debounced? Is there some analog logic, or is there a microcontroller in the keyboard that does this, or what? How do membrane switches work?</li> <li>How is the OS notified of the keypress? I could probably answer this for a 286, but nowadays it's somehow done through x2APIC, right? How does that work?</li> <li>Also, USB, PS/2, and AT keyboards are different, somehow? How does USB work? And what about laptop keyboards? Is that just a USB connection?</li> <li>How does a USB connector work? You have this connection that can handle 10Gb/s. That surely won't work if there's any gap at all between the p…

<p>Most real-world problems are big enough that you can't just head for the end goal, you have to break them down into smaller parts and set up intermediate goals. For that matter, most games are that way too. “Win” is too big a goal in chess, so you might have a subgoal like <a href="http://www.chesstactics.org/index.php?Type=page&amp;Action=none&amp;From=2,1,1,1">don't get forked</a>. While creating subgoals makes intractable problems tractable, it also creates the problem of determining the relative priority of different subgoals and whether or not a subgoal is relevant to the ultimate goal at all. In chess, there are libraries worth of books written on just that.</p> <p>And chess is really simple compared to a lot of real world problems. 64 squares. 32 pieces. Pretty much any analog problem you can think of contains more state than chess, and so do a lot of discrete problems. Chess is also relatively simple because you can directly measure whether or not you succeeded (won). Many …

<p>There's an ongoing debate over whether &quot;AI&quot; will ever be good enough to displace humans and, if so, when it will happen. In this debate, the optimists tend to focus on how much AI is improving and the pessimists point to all the ways AI isn't as good as an ideal human being. I think this misses two very important factors.</p> <p>One, is that jobs that are on the potential chopping block, such as first-line customer service, customer service for industries that are either low margin or don't care about the customer, etc., tend to be filled by apathetic humans in a poorly designed system, and <a href="https://danluu.com/p95-skill/">humans aren't even very good at simple tasks</a> they <a href="https://danluu.com/bad-decisions/">care a lot about</a>. When we're apathetic, we're absolutely terrible; it's not going to take a nearly-omniscient sci-fi level AI to perform at least somewhat comparably.</p> <p>Two, companies are going to replace humans with AI in many roles even i…

<p>It's generally accepted that any piece of software could be compromised with a backdoor. Prominent examples include the <a href="http://en.wikipedia.org/wiki/Sony_BMG_copy_protection_rootkit_scandal">Sony/BMG installer</a>, which had a backdoor built-in to allow Sony to keep users from copying the CD, which also allowed malicious third-parties to take over any machine with the software installed; the <a href="http://redmine.replicant.us/projects/replicant/wiki/SamsungGalaxyBackdoor">Samsung Galaxy</a>, which has a backdoor that allowed the modem to access the device's filesystem, which also allows anyone running <a href="https://www.youtube.com/watch?v=RXqQioV_bpo">a fake base station to access files on the device</a>; <a href="http://www.cypherspace.org/adam/hacks/lotus-nsa-key.html">Lotus Notes</a>, which had a backdoor which allowed encryption to be defeated; and <a href="https://forums.lenovo.com/t5/Lenovo-P-Y-and-Z-series/Lenovo-Pre-instaling-adware-spam-Superfish-powerd-by/td-…

<p>Does it make sense for me to run ads on my blog? I've been thinking about this lately, since Carbon Ads contacted me about putting an ad up. What are the pros and cons? This isn't a rhetorical question. I'm genuinely <a href="https://twitter.com/danluu">interested in what you think</a>.</p> <p></p> <h3 id="pros">Pros</h3> <h4 id="money">Money</h4> <p>Hey, who couldn't use more money? And it's basically free money. Well, except for the all of the downsides.</p> <h4 id="data">Data</h4> <p>There's lots of studies on the impact of ads on site usage and behavior. But as with any sort of benchmarking, it's not really clear how or if that generalizes to other sites if you don't have a deep understanding of the domain, and I have almost no understanding of the domain. If I run some ads and do some A/B testing I'll get to see what the effect is on my site, which would be neat.</p> <h3 id="cons">Cons</h3> <h4 id="money-1">Money</h4> <p>It's not enough money to make a living off of, a…

<p>This is a response to the following question from David Albert:</p> <blockquote> <p>My mental model of CPUs is stuck in the 1980s: basically boxes that do arithmetic, logic, bit twiddling and shifting, and loading and storing things in memory. I'm vaguely aware of various newer developments like vector instructions (SIMD) and the idea that newer CPUs have support for virtualization (though I have no idea what that means in practice).</p> <p>What cool developments have I been missing? What can today's CPU do that last year's CPU couldn't? How about a CPU from two years ago, five years ago, or ten years ago? The things I'm most interested in are things that programmers have to manually take advantage of (or programming environments have to be redesigned to take advantage of) in order to use and as a result might not be using yet. I think this excludes things like Hyper-threading/SMT, but I'm not honestly sure. I'm also interested in things that CPUs can't do yet but will be able to …

<p>Here's a language that gives near-C performance that feels like Python or Ruby with optional type annotations (that you can feed to <a href="https://github.com/tonyhffong/Lint.jl">one of two</a> <a href="https://github.com/astrieanna/TypeCheck.jl">static analysis tools</a>) that has good support for macros plus decent-ish support for FP, plus a lot more. What's not to like? I'm mostly not going to talk about how great Julia is, though, because you can find plenty of blog posts that do that all over the internet.</p> <p>The last time I used Julia (around Oct. 2014), I ran into two new (to me) bugs involving bogus exceptions when processing Unicode strings. To work around those, I used a try/catch, but of course that runs into a non-deterministic bug I've found with try/catch. I also hit a bug where a function returned a completely wrong result if you passed it an argument of the wrong type instead of throwing a &quot;no method&quot; error. I spent half an hour writing a throwaway sc…

<p>How much overhead should we expect from enabling integer overflow checks? Using a compiler flag or built-in intrinsics, we should be able to do the check with a conditional branch that branches based on the overflow flag that <code>add</code> and <code>sub</code> set. Code that looks like</p> <pre><code>add %esi, %edi </code></pre> <p>should turn into something like</p> <pre><code>add %esi, %edi jo &lt;handle_overflow&gt; </code></pre> <p>Assuming that branch is always correctly predicted (which should be the case for most code), the costs of the branch are the cost of executing that correctly predicted not-taken branch, the pollution the branch causes in the branch history table, and the cost of decoding the branch (on x86, <code>jo</code> and <code>jno</code> don't fuse with <code>add</code> or <code>sub</code>, which means that on the fast path, the branch will take up one of the 4 opcodes that can come from the decoded instruction cache per cycle). That's probab…

<p>Let's write a <a href="http://man7.org/linux/man-pages/man3/malloc.3.html">malloc</a> and see how it works with existing programs!</p> <p>This is basically an expanded explanation of what I did after reading <a href="http://www.inf.udec.cl/~leo/Malloc_tutorial.pdf">this tutorial</a> by Marwan Burelle and then sitting down and trying to write my own implementation, so the steps are going to be fairly similar. The main implementation differences are that my version is simpler and more vulnerable to memory fragmentation. In terms of exposition, my style is a lot more casual.</p> <p>This tutorial is going to assume that you know what pointers are, and that you know enough C to know that <code>*ptr</code> dereferences a pointer, <code>ptr-&gt;foo</code> means <code>(*ptr).foo</code>, that malloc is used to <a href="http://duartes.org/gustavo/blog/post/how-the-kernel-manages-your-memory/">dynamically allocate space</a>, and that you're familiar with the concept of a linked list. For a b…

<p>Public discussions of discrimination in tech often result in someone claiming that discrimination is impossible because of market forces. Here's a quote from Marc Andreessen that sums up a common view<sup class="footnote-ref" id="fnref:1"><a rel="footnote" href="#fn:1">1</a></sup>.</p> <blockquote> <p>Let's launch right into it. I think the critique that Silicon Valley companies are deliberately, systematically discriminatory is incorrect, and there are two reasons to believe that that's the case. ... No. 2, our companies are desperate for talent. Desperate. Our companies are dying for talent. They're like lying on the beach gasping because they can't get enough talented people in for these jobs. The motivation to go find talent wherever it is unbelievably high.</p> </blockquote> <p>Marc Andreessen's point is that the market is too competitive for discrimination to exist. But VC funded startups aren't the first companies in the world to face a competitive hiring market. Consider t…

<p>I was curious what different people worked on in Linux, so I tried grabbing data from the current git repository to see if I could pull that out of commit message data. This doesn't include history from before they switched to git, so it only goes back to 2005, but that's still a decent chunk of history.</p> <p>Here's a list of the most commonly used words (in commit messages), by the top four most frequent committers, with users ordered by number of commits.</p> <p></p> <table> <thead> <tr> <th>User</th> <th>1</th> <th>2</th> <th>3</th> <th>4</th> <th>5</th> </tr> </thead> <tbody> <tr> <td>viro</td> <td>to</td> <td>in</td> <td>of</td> <td>and</td> <td>the</td> </tr> <tr> <td>tiwai</td> <td>alsa</td> <td>the</td> <td>-</td> <td>for</td> <td>to</td> </tr> <tr> <td>broonie</td> <td>the</td> <td>to</td> <td>asoc</td> <td>for</td> <td>a</td> </tr> <tr> <td>davem</td> <td>the</td> <td>to</td> <td>in</td> <td>and</td> <td>sparc64</td> </tr> </tbody> </table> <p><br> Alright, so th…

<p>If I had to guess, I'd say I probably work around hundreds of bugs in an average week, and thousands in a bad week. It's not unusual for me to run into a hundred new bugs in a single week. But I often get skepticism when I mention that I run into multiple new (to me) bugs per day, and that this is inevitable if we don't change how we write tests. Well, here's a log of one week of bugs, limited to bugs that were new to me that week. After a brief description of the bugs, I'll talk about what we can do to improve the situation. The obvious answer to spend more effort on testing, but everyone already knows we should do that and no one does it. That doesn't mean it's hopeless, though.</p> <h3 id="one-week-of-bugs">One week of bugs</h3> <h4 id="ubuntu">Ubuntu</h4> <p>When logging into my machine, I got a screen saying that I entered my password incorrectly. After a five second delay, it logged me in anyway. This is probably at least two bugs, perhaps more.</p> <h4 id="github">GitHub<…

<p>I've seen all these studies that show how a 100ms improvement in page load time has a significant effect on page views, conversion rate, etc., but I'd never actually tried to optimize my site. This blog is a static Octopress site, hosted on GitHub Pages. Static sites are supposed to be fast, and GitHub Pages uses Fastly, which is supposed to be fast, so everything should be fast, right?</p> <p>Not having done this before, I didn't know what to do. But in a great talk on how the internet works, <a href="https://twitter.com/danielespeset">Dan Espeset</a> suggested trying <a href="http://www.webpagetest.org">webpagetest</a>; let's give it a shot.</p> <p></p> <p>Here's what it shows with my nearly stock Octopress setup<sup class="footnote-ref" id="fnref:S"><a rel="footnote" href="#fn:S">1</a></sup>. The only changes I'd made were enabling Google Analytics, the social media buttons at the bottom of posts, and adding CSS styling for tables (which are, by default, unstyled and unreadabl…

<p>I've noticed that builds are broken and tests fail a lot more often on open source projects than on “work” projects. I wasn't sure how much of that was my perception vs. reality, so I grabbed the Travis CI data for a few popular categories on GitHub<sup class="footnote-ref" id="fnref:C"><a rel="footnote" href="#fn:C">1</a></sup>.</p> <p></p> <p><img src="https://danluu.com/images/broken-builds/reliability.png" alt="Graph of build reliability. Props to nu, oryx, caffe, catalyst, and Scala." width="640" height="1280"></p> <p>For reference, at every place I've worked, two 9s of reliability (99% uptime) on the build would be considered bad. That would mean that the build is failing for over three and a half days a year, or seven hours per month. Even three 9s (99.9% uptime) is about forty-five minutes of downtime a month. That's kinda ok if there isn't a hard system in place to prevent people from checking in bad code, but it's quite bad for a place that's serious about having workin…

<p>There are some pretty strong statements about types floating around out there. The claims range from the oft-repeated phrase that when you get the types to line up, everything just works, to “not relying on type safety is unethical (if you have an SLA)”<sup class="footnote-ref" id="fnref:P"><a rel="footnote" href="#fn:P">1</a></sup>, &quot;<a href="https://news.ycombinator.com/item?id=4369359">It boils down to cost vs benefit, actual studies, and mathematical axioms, not aesthetics or feelings</a>&quot;, and <a href="https://twitter.com/posco/status/566130704157667328">I think programmers who doubt that type systems help are basically the tech equivalent of an anti-vaxxer</a>. The first and last of these statements are from &quot;types&quot; thought leaders who are widely quoted. There are probably plenty of strong claims about dynamic languages that I'd be skeptical of if I heard them, but I'm not in the right communities to hear the stronger claims about dynamically typed language…

<p>The latest version of the Intel manual has a couple of <a href="https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf">new instructions for non-volatile storage, like SSDs</a>. What's that about?</p> <p>Before we look at the instructions in detail, let's take a look at the issues that exist with super fast NVRAM. One problem is that next generation storage technologies (PCM, 3d XPoint, etc.), will be fast enough that syscall and other OS overhead can be more expensive than the actual cost of the disk access<sup class="footnote-ref" id="fnref:1"><a rel="footnote" href="#fn:1">1</a></sup>. Another is the impedance mismatch between the x86 memory hierarchy and persistent memory. In both cases, it's basically an <a href="https://en.wikipedia.org/wiki/Amdahl%27s_law">Amdahl's law</a> problem, where one component has improved so much that other components have to improve to keep up.</p> <p></p> <p>There's a <a href="http://research.microsoft.com/pubs/198366/Asplos…

<p>Once upon a time, my <a href="http://pages.cs.wisc.edu/~sohi/">computer architecture professor</a> mentioned that using a random eviction policy for caches really isn't so bad. That random eviction isn't bad can be surprising — if your cache fills up and you have to get rid of something, choosing the <a href="http://www.mathcs.emory.edu/~cheung/Courses/355/Syllabus/9-virtual-mem/LRU-replace.html">least recently used (LRU)</a> is an obvious choice, since you're more likely to use something if you've used it recently. If you have a tight loop, LRU is going to be perfect as long as the loop fits in cache, but it's going to cause a miss every time if the loop doesn't fit. A random eviction policy degrades gracefully as the loop gets too big.</p> <p>In practice, on real workloads, random tends to do worse than other algorithms. But what if we take two random choices (2-random) and just use LRU between those two choices?</p> <p>Here are the relative miss rates we get for SPEC CPU<sup cl…

<p>This is an off-the-cuff comment for Hacker School's Paper of the Week Read Along series for <a href="https://github.com/papers-we-love/papers-we-love/raw/master/design/out-of-the-tar-pit.pdf">Out of the Tar Pit</a>.</p> <p></p> <p>I find the idea itself, which is presented in sections 7-10, at the end of the paper, pretty interesting. However, I have some objections to the motivation for the idea, which makes up the first 60% of the paper.</p> <p>Rather than do one of those blow-by-blow rebuttals that's so common on blogs, I'll limit my comments to one widely circulated idea that I believe is not only mistaken but actively harmful.</p> <p>There's a claim that “informal reasoning” is more important than “testing”<sup class="footnote-ref" id="fnref:1"><a rel="footnote" href="#fn:1">1</a></sup>, based mostly on the strength of this quote from Dijkstra:</p> <blockquote> <p>testing is hopelessly inadequate....(it) can be used very effectively to show the presence of bugs but never t…

<p>Every once in a while, I hear how intrinsics have improved enough that it's safe to use them for high performance code. That would be nice. The promise of intrinsics is that you can write optimized code by calling out to functions (intrinsics) that correspond to particular assembly instructions. Since intrinsics act like normal functions, they can be cross platform. And since your compiler has access to more computational power than your brain, as well as a detailed model of every CPU, the compiler should be able to do a better job of micro-optimizations. Despite decade old claims that intrinsics can make your life easier, it never seems to work out.</p> <p>The last time I tried intrinsics was around 2007; for more on why they were hopeless then (<a href="http://virtualdub.org/blog/pivot/entry.php?id=162">see this exploration by the author of VirtualDub</a>). I gave them another shot recently, and while they've improved, they're still not worth the effort. The problem is that intri…

<pre><code>UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF CALIFORNIA SAN JOSE DIVISION IN RE: HIGH-TECH EMPLOYEE ANTITRUST LITIGATION THIS DOCUMENT RELATES TO: ALL ACTIONS Case No.: 11-CV-02509-LHK </code></pre> <h3 id="order-denying-plaintiffs-motion-for-preliminary-approval-of-settlements-with-adobe-apple-google-and-intel">ORDER DENYING PLAINTIFFS' MOTION FOR PRELIMINARY APPROVAL OF SETTLEMENTS WITH ADOBE, APPLE, GOOGLE, AND INTEL</h3> <p>Before the Court is a Motion for Preliminary Approval of Class Action Settlement with Defendants Adobe Systems Inc. (&quot;Adobe&quot;), Apple Inc. (&quot;Apple&quot;), Google Inc. (&quot;Google&quot;), and Intel Corp. (&quot;Intel&quot;) (hereafter, &quot;Remaining Defendants&quot;) brought by three class representatives, Mark Fichtner, Siddharth Hariharan, and Daniel Stover (hereafter, &quot;Plaintiffs&quot;). See ECF No. 920. The Settlement provides for $324.5 million in recovery for the class in exchange for release of antitrust claims. …

<p><i>This is an archived USENET post from John Cooley on a competitive comparison between VHDL and Verilog that was done in 1997.</i></p> <p>I knew I hit a nerve. Usually when I publish a candid review of a particular conference or EDA product I typically see around 85 replies in my e-mail &quot;in&quot; box. Buried in my review of the recent Synopsys Users Group meeting, I very tersely reported that 8 out of the 9 Verilog designers managed to complete the conference's design contest yet <em>none</em> of the 5 VHDL designers could. I apologized for the terseness and promised to do a detailed report on the design contest at a later date. Since publishing this, my e-mail &quot;in&quot; box has become a veritable Verilog/VHDL Beirut filling up with 169 replies! Once word leaked that the detailed contest write-up was going to be published in the DAC issue of &quot;Integrated System Design&quot; (formerly &quot;ASIC &amp; EDA&quot; magazine), I started getting phone calls from the chairma…

<p><a href="https://danluu.com/everything-is-broken/">I can't remember the last time I went a whole day without running into a software bug</a>. For weeks, I couldn't invite anyone to Facebook events due to a bug that caused the invite button to not display on the invite screen. Google Maps has been giving me illegal and sometimes impossible directions ever since I moved to a small city. And Google Docs regularly hangs when I paste an image in, giving me a busy icon until I delete the image.</p> <p>It's understandable that bugs escape testing. Testing is hard. Integration testing is harder. End to end testing is even harder. But there's an easier way. A third of bugs like this – bugs I run into daily – could be found automatically using analytics.</p> <p></p> <p>If you think finding bugs with analytics sounds odd, ask a hardware person about performance counters. Whether or not they're user accessible, every ASIC has analytics to allow designers to figure out what changes need to be…

<p>Editing binaries is a trick that comes in handy a few times a year. You don't often need to, but when you do, there's no alternative. When I mention patching binaries, I get one of two reactions: complete shock or no reaction at all. As far as I can tell, this is because most people have one of these two models of the world:</p> <blockquote> <ol> <li><p>There exists source code. Compilers do something to source code to make it runnable. If you change the source code, different things happen.</p></li> <li><p>There exists a processor. The processor takes some bits and decodes them to make things happen. If you change the bits, different things happen.</p></li> </ol> </blockquote> <p>If you have the first view, breaking out a hex editor to modify a program is the action of a deranged lunatic. If you have the second view, editing binaries is the most natural thing in the world. Why wouldn't you just edit the binary? It's often the easiest way to get what you need.</p> <p></p> <p>Fo…

<p>Last week, Quartz published an article titled <a href="https://news.ycombinator.com/item?id=7334659">“There is no gender gap in tech salaries”</a>. That resulted in linkbait copycat posts all over the internet, from obscure livejournals to <a href="http://www.smithsonianmag.com/smart-news/female-computer-scientists-make-same-salary-their-male-counterparts-180949965/?no-ist">Smithsonian.com</a>. The claims are awfully strong, considering that the main study cited only looked at people who graduated with a B.S. exactly one year ago, not to mention the fact that the study makes literally the opposite claim.</p> <p>Let's look at the evidence from the <a href="http://www.aauw.org/files/2013/02/graduating-to-a-pay-gap-the-earnings-of-women-and-men-one-year-after-college-graduation.pdf?_ga=1.7578036.722397424.1379578621">AAUW study</a> that all these posts cite.</p> <p><img src="https://danluu.com/images/gender-gap/actual_result.png" alt="Who are you going to believe, me or your lying ey…

<p>In 1983, at the University of Wisconsin, Dina Bitton, David DeWitt, and Carolyn Turbyfill created a <a href="http://pages.cs.wisc.edu/~dewitt/includes/benchmarking/vldb83.pdf">database benchmarking framework</a>. Some of their results included (lower is better):</p> <p>Join without indices</p> <style>table {border-collapse:collapse;margin:0px auto;}table,th,td {border: 1px solid black;}td {text-align:center;}</style> <table> <thead> <tr> <th>system</th> <th>joinAselB</th> <th>joinABprime</th> <th>joinCselAselB</th> </tr> </thead> <tbody> <tr> <td>U-INGRES</td> <td>10.2</td> <td>9.6</td> <td>9.4</td> </tr> <tr> <td>C-INGRES</td> <td>1.8</td> <td>2.6</td> <td>2.1</td> </tr> <tr> <td>ORACLE</td> <td>&gt; 300</td> <td>&gt; 300</td> <td>&gt; 300</td> </tr> <tr> <td>IDMnodac</td> <td>&gt; 300</td> <td>&gt; 300</td> <td>&gt; 300</td> </tr> <tr> <td>IDMdac</td> <td>&gt; 300</td> <td>&gt; 300</td> <td>&gt; 300</td> </tr> <tr> <td>DIRECT</td> <td>10.2</td> <td>9.5</td> <td>5.6</td> <…

<p>In the fall of 2000, I took my first engineering class: <a href="http://courses.engr.wisc.edu/ece/ece352.html">ECE 352</a>, an entry-level digital design class for first-year computer engineers. It was standing room only, filled with waitlisted students who would find seats later in the semester as people dropped out. We had been warned in orientation that half of us wouldn't survive the year. In class, We were warned again that half of us were doomed to fail, and that ECE 352 was the weed-out class that would be responsible for much of the damage.</p> <p>The class moved briskly. The first lecture wasted little time on matters of the syllabus, quickly diving into the real course material. Subsequent lectures built on previous lectures; anyone who couldn't grasp one had no chance at the next. Projects began after two weeks, and also built upon their predecessors; anyone who didn't finish one had no hope of doing the next.</p> <p>A friend of mine and I couldn't understand why some p…

<p>Dear <a href="http://dpb.bitbucket.org/">David</a>,</p> <p>I'm afraid my off the cuff response the other day wasn't too well thought out; when you talked about taking calc III and linear algebra, and getting resistance from one of your friends because &quot;wolfram alpha can do all of that now,&quot; my first reaction was horror-- which is why I replied that while I've often regretted not taking a class seriously because I've later found myself in a situation where I could have put the skills to good use, I've never said to myself &quot;<a href="http://www.dartmouth.edu/~matc/MathDrama/reading/Wigner.html">what a waste of time it was to learn that fundamental mathematical concept and use it enough to that I truly understand it</a>.&quot;</p> <p></p> <p>But could this be selection bias? It's easier to recall the math that I use than the math I don't. To check, let's look at the nine math classes I took as an undergrad. If I exclude the jobs I've had that are obviously math oriente…

<p>Here's the graph of <a href="https://github.com/danluu/setjmp-longjmp-ucontext-snippets/tree/master/benchmark/cache-conflict">a toy benchmark</a><sup class="footnote-ref" id="fnref:A"><a rel="footnote" href="#fn:A">1</a></sup> of page-aligned vs. mis-aligned accesses; it shows a ratio of performance between the two at different working set sizes. If this benchmark seems contrived, it actually comes from a real world example of the <a href="https://groups.google.com/forum/#!msg/comp.arch/_uecSnSEQc4/mvfRnOvIyzUJ">disastrous performance implications of using nice power of 2 alignment, or page alignment in an actual system</a><sup class="footnote-ref" id="fnref:M"><a rel="footnote" href="#fn:M">2</a></sup>.</p> <p><img src="https://danluu.com/images/3c-conflict/sandy.png" alt="Graph of Sandy Bridge Performance" width="504" height="216"> <img src="https://danluu.com/images/3c-conflict/westmere.png" alt="Graph of Westmere Performance" width="504" height="504"></p> <p>Except for very s…

<p>Earlier this year, I interviewed with a well-known tech startup, one of the hundreds of companies that claims to have harder interviews, more challenging work, and smarter employees than Google<sup class="footnote-ref" id="fnref:1"><a rel="footnote" href="#fn:1">1</a></sup>. My first interviewer, John, gave me the standard tour: micro-kitchen stocked with a combination of healthy snacks and candy; white male 20-somethings gathered around a foosball table; bright spaces with cutesy themes; a giant TV set up for video games; and the restroom. Finally, he showed me a closet-sized conference room and we got down to business.</p> <p>After the usual data structures and algorithms song and dance, we moved on to the main question: how would you design a classification system for foo<sup class="footnote-ref" id="fnref:2"><a rel="footnote" href="#fn:2">2</a></sup>? We had a discussion about design tradeoffs, but the key disagreement was about the algorithm. I said, if I had to code something…

<p>In CPU design, most successful teams have a fairly long lineage and rely heavily on experienced engineers. When we look at CPU startups, teams that have a successful exist often have a core team that's been together for decades. For example, PA Semi's acquisition by Apple was a moderately successful exit, but where did that team come from? They were the SiByte team, which left after SiByte was acquired by Broadcom, and SiByte was composed of many people from DEC who had been working together for over a decade. My old company was similar: an IBM fellow collected <a href="http://conservancy.umn.edu/handle/120173">the best people he worked with at IBM</a> who was a very early Dell employee and then exec (back when Dell still did interesting design work), then split off to create a chip startup. There have been quite a few CPU startups that have raised tens to hundreds of millions and leaned heavily on inexperienced labor; fresh PhDs and hardware engineers with only a few years of exper…

<p>What's the first thing you do when you find a bug or see a missing feature in an open source project? Check out the project page and submit a patch!</p> <p><img src="https://danluu.com/images/discourage-oss/send-us-116-pull-requests.png" alt="Send us a pull request! (116 open pull requests)" width="1012" height="152"></p> <p>Oh. Maybe their message is so encouraging that they get hundreds of pull requests a week, and the backlog isn't that bad.</p> <p><img src="https://danluu.com/images/discourage-oss/why-is-this-ignored.png" alt="Multiple people ask why this bug fix is being ignored. No response." width="745" height="472"></p> <p>Maybe not. Giant sucker than I am, I submitted a pull request even after seeing that. All things considered, I should consider myself lucky that it's possible to <a href="http://www.jwz.org/blog/2013/02/wow-remember-when-people-tracked-bugs-welcome-to-the-next-level/">submit pull requests at all</a>. If I'm really lucky, maybe they'll get around to <a …

<p>You ever notice that there's this funny threshold for getting to the front page on sites like HN? The exact threshold varies depending on how much traffic there is, but, for articles that aren't wildly popular, there's this moment when the article is at N-1 votes. There is, perhaps, a 60% chance that the vote will come and the article will get pushed to the front page, where it will receive a slew of votes. There is, maybe, a 40% chance it will never get the vote that pushes it to the front page, causing it to languish in obscurity forever.</p> <p>It's non-optimal that an article that will receive 50 votes <a href="https://www.khanacademy.org/math/probability/random-variables-topic/random_variables_prob_dist/v/expected-value--e-x">in expectation</a> has a 60% chance of getting 100+ votes, and a 40% chance of getting 2 votes. Ideally, each article would always get its expected number of votes and stay on the front page for the expected number of time, giving readers exposure to the …

<p><img src="https://danluu.com/images/pl-troll/davidalbert-pl-troll.png" alt="PL troll: a statically typed language with no type declarations. Types are determined entirely using Hungarian notation" width="1160" height="448"></p> <p>Troll? That's how people write Verilog<sup class="footnote-ref" id="fnref:1"><a rel="footnote" href="#fn:1">1</a></sup>. At my old company, we had a team of formal methods PhD's who wrote a linter that typechecked our code, based on our naming convention. For our chip (which was small for a CPU), building a model (compiling) took about five minutes, running a single short test took ten to fifteen minutes, and long tests took CPU months. The value of a linter that can run in seconds should be obvious, not even considering the fact that it can take hours of tracing through waveforms to find out why a test failed<sup class="footnote-ref" id="fnref:2"><a rel="footnote" href="#fn:2">2</a></sup>.</p> <p>Lets look at some of the most commonly used naming conven…

<p>Verilog is the most commonly used language for hardware design in America (VHDL is more common in Europe). Too bad it's so baroque. If you ever browse the <a href="http://stackoverflow.com/questions/tagged/verilog">Verilog questions on Stack Overflow</a>, you'll find a large number of questions, usually downvoted, asking “why doesn't my code work?”, with code that's not just a little off, but completely wrong.</p> <p><img src="https://danluu.com/images/so_verilog.png" alt="6 questions, all but one with negative score"></p> <p></p> <p>Lets look at <a href="http://stackoverflow.com/questions/18678426/verilog-store-counter-value-when-reset-is-asserted">an example</a>: “Idea is to store value of counter at the time of reset . . . I get DRC violations and the memory, bufreadaddr, bufreadval are all optimized out.”</p> <pre><code>always @(negedge reset or posedge clk) begin if (reset == 0) begin d_out &lt;= 16'h0000; d_out_mem[resetcount] &lt;= d_out; laststoredvalue &lt…

<h3 id="about-the-blog">About The Blog</h3> <p>This started out as a way to jot down thoughts on areas that seem interesting but underappreciated. Since then, this site has grown to the point where it gets <a href="https://danluu.com/blog-ads/#update">millions of hits a month</a> and I see that it's commonly cited by professors in their courses and on stackoverflow.</p> <p>That's flattering, but more than anything else, I view that as a sign there's a desperate shortage of <a href="//danluu.com/teach-debugging/">understandable explanation of technical topics</a>. There's nothing here that most of my co-workers don't know (with the exception of maybe three or four posts where I propose novel ideas). It's just that they don't blog and I do. I'm not going to try to <a href="https://sites.google.com/site/steveyegge2/you-should-write-blogs">convince you to start writing a blog</a>, since that has to be something you want to do, but I will point out that there's a large gap that's waiting …

<p><i>this is an archive of an old article by John Carmack which seems to have disappeared off of the internet</i></p> <h4 id="abstract">Abstract</h4> <p>Virtual reality (VR) is one of the most demanding human-in-the-loop applications from a latency standpoint. The latency between the physical movement of a user’s head and updated photons from a head mounted display reaching their eyes is one of the most critical factors in providing a high quality experience.</p> <p>Human sensory systems can detect very small relative delays in parts of the visual or, especially, audio fields, but when absolute delays are below approximately 20 milliseconds they are generally imperceptible. Interactive 3D systems today typically have latencies that are several times that figure, but alternate configurations of the same hardware components can allow that target to be reached.</p> <p>A discussion of the sources of latency throughout a system follows, along with techniques for reducing the latency …

<p><i>This is a transcript of the Kara Swisher / Jack Dorsey interview from 2/12/2019, made by parsing the original Tweets because I wanted to be able to read this linearly. There's a &quot;moment&quot; that tries to track this, but since it doesn't distinguish between sub-threads in any way, you can't tell the difference between end of a thread and a normal reply. This linearization of the interview marks each thread break with a page break and provides some context from upthread where relevant (in <font color="#808080">grey text</font>).</i></p> <p><a href="https://twitter.com/karaswisher/status/1095440667373899776"><strong>Kara</strong></a>: Here in my sweatiest @soulcycle outfit for my Twitterview with @jack with @Laur_Katz at the ready @voxmediainc HQ. Also @cheezit acquired. #karajack</p> <p><a href="https://twitter.com/karaswisher/status/1095442716346011649"><strong>Kara</strong></a>: Oh hai @jack. Let’s set me set the table. First, I am uninterested in beard amulets or weird …

<p><em>This is an archive of the Jonathan Shapiro's &quot;Retrospective Thoughts on BitC&quot; that seems to have disappeared from the internet; at the time, BitC was aimed at the same niche as Rust</em></p> <pre><code>Jonathan S. Shapiro shap at eros-os.org Fri Mar 23 15:06:41 PDT 2012 </code></pre> <p>By now it will be obvious to everyone that I have stopped work on BitC. An explanation of <em>why</em> seems long overdue.</p> <p>One answer is that work on Coyotos stopped when I joined Microsoft, and the work that I am focused on <em>now</em> doesn't really require (or seem to benefit from) BitC. As we all know, there is only so much time to go around in our lives. But that alone wouldn't have stopped me entirely.</p> <p>A second answer is that BitC isn't going to work in its current form. I had hit a short list of issues that required a complete re-design of the language and type system followed by a ground-up new implementation. Experience with the first implementation suggested…

<p><em>This is an archive of an old Google Buzz conversation (circa 2010?) on a variety of topics, including whether or not it's inevetible that a closed platform will dominate social.</em></p> <p><strong>Piaw</strong>: Social networks will be dominated primarily by network effects. That means in the long run, Facebook will dominate all the others.</p> <p><strong>Rebecca</strong>: ... which is also why no one company should dominate it. &quot;The Social Graph&quot; and its associated apps should be like the internet, distributed and not confined to one company's servers. I wish the narrative surrounding this battle was centered around this idea, and not around the whole Silicon Valley &quot;who is the most genius innovator&quot; self-aggrandizing unreality field. Thank god Tim Berners Lee wasn't from Silicon Valley, or we wouldn't have the Internet the way we know it in the first place.</p> <p>I suppose I shouldn't be being so snarky, revealing how much I hate your narratives someti…

<p><em>This is an archive of an old Google Buzz conversation on MIT vs. Stanford and Silicon Valley vs. Boston</em></p> <p>There's no reason why the Boston area shouldn't be as much a hotbed of startups as Silicon Valley is. By contrast, there are lots of reasons why NYC is no good for startups. Nevertheless, Paul Graham gave up on the Boston area, so there must be something that hinders startup formation in the area.</p> <p><strong>Kevin</strong>: This has nothing to do with money, or talents, or what it. All it matters is &quot;entrepreneur density&quot;.</p> <p>Boston may have the money, the talent, the intelligence, but does it have an entrepreneurial spirit and enough of a density?</p> <p><strong>Marya</strong>: From <a href="http://www.xconomy.com/boston/2009/01/22/paul-graham-and-y-combinator-to-leave-cambridge-stay-in-silicon-valley-year-round/">http://www.xconomy.com/boston/2009/01/22/paul-graham-and-y-combinator-to-leave-cambridge-stay-in-silicon-valley-year-round/</a> &q…

<p><i>This is an archive of some posts in a forum thread titled &quot;Beware of Bioware&quot; in a now defunct forum, with comments from that forum as well as blog comments from a now defunct blog that archived that made the first attempt to archive this content. The original posts were deleted shortly after being posted, replaced with &quot;big scary company vs. li'l ol' me.&quot;</i></p> <p><i>Although the comments below seem to be about Bioware's main studio in Edmonton, I knew someone at Bioware Austin during the time period under discussion, which is how I learned about the term &quot;sympathy crunch&quot;, where you're required to be at the office because other teams are &quot;crunching&quot;. I'd never heard of this concept before, so I looked it up and found the following thread around 2008 or so.</i></p> <p><i>Searching for &quot;sympathy crunch&quot; today, in 2024, doesn't return many hits. One of the few hits is a 2011 blog post by a former director at BioWare titled &quo…

<p><em>This is an archive of Dan Weinreb's comments on Symbolics and Lisp machines.</em></p> <h3 id="rebuttal-to-stallman-s-story-about-the-formation-of-symbolics-and-lmi">Rebuttal to Stallman’s Story About The Formation of Symbolics and LMI</h3> <p>Richard Stallman has been telling a story about the origins of the Lisp machine companies, and the effects on the M.I.T. Artificial Intelligence Lab, for many years. He has published it in a book, and in a widely-referenced paper, which you can find at <a href="http://www.gnu.org/gnu/rms-lisp.html">http://www.gnu.org/gnu/rms-lisp.html</a>.</p> <p>His account is highly biased, and in many places just plain wrong. Here’s my own perspective on what really happened.</p> <p>Richard Greenblatt’s proposal for a Lisp machine company had two premises. First, there should be no outside investment. This would have been totally unrealistic: a company manufacturing computer hardware needs capital. Second, Greenblatt himself would be the CEO. The oth…

<p>Archived from an unknown source. Possibly Gravitron?</p> <p>In regards for history:</p> <h3 id="chapter-1">Chapter #1</h3> <p>(Around) December 1995 is when it all started. Rod Humble wished to create something like Air Warrior but online, he approached Virgin Interactive Entertainment with the idea and they replied with something along the lines of &quot;here's the cash, good luck&quot;. Rod called Jeff Petersen and asked if he's interested in helping him creating an online only game, Jeff agreed. Inorder to overcome lag Jeff decided it would be best to test an engine which simulates neutonian physic principles - an object in motion tends to keep the same vector, and on that he also built prediction forumlas. They also enlisted Juan Sanchez, with whom Rod had worked before to design them some graphics for it. Either way, after some initial advancement they've decided to put it on the public gamers block to test it and code named it Sniper. They got a few people who played it aro…