August 29, 2008
Eureka Moments and InterviewsI've been meaning to blog about this (or anything) for a while. In the July 28 New Yorker, there was an article titled "The Eureka Hunt", about where sudden insights come from. Some key quotes:
- "There is something inherently mysterious about moments of insight...Such tales all share a few essential features, which psychologists and neuroscientists use to define 'the insight experience.' The first of these is impasse: before there can be a breakthrough, there has to be a mental block...another key feature of insight [is] the feeling of certainty that accompanies an idea."
- "Schooler had demonstrated that it was possible to interfere with insight by making people explain their thought process while trying to solve a puzzle--a phenomenon he called 'verbal overshadowing.' This made sense to Jung-Beeman, since the act of verbal explanation would naturally shift activity over to the left hemisphere, causing people to ignore the more subtle associations coming from the right side of the brain."
- "Jung-Beeman began searching in the right hemisphere for the source of insight in the brain. He decided to compare puzzles solved in moments of insight with those solved by methodical testing of potential solutions, in which people can accurately trace their thought process and had no sense of surprise when the answer came. Unfortunately, all the classic puzzles developed by scientists to study insight required insight; if subjects didn't solve them in a sudden 'Aha!' moment, they didn't solve them at all."
- "Jung-Beeman needed to develop a set of puzzles that could be solved either by insight or by analysis....he eventually settled on a series of verbal puzzles...In a C.R.A. word puzzle, a subject is given three words, such as 'pine,' 'crab,' and 'sauce,' and asked to think of a word that can be combined with all three--in this case, 'apple.'"
- "The resulting studies...found that people who solved puzzles with insight activated a specific subset of cortical areas...The first activities activated during the problem-solving process were those involved with executive control...What happens next is the 'search phase' as the brain starts looking for answers in all the relevant places...The search can quickly get frustrating...'Almost all of the possibilities that your brain comes up with are going to be wrong,' Jung-Beeman says, 'And it's up to the executive-control areas to keep on searching, or, if necessary, change strategies and start searching something else.'"
- "The insight process, as sketched by Jung-Beeman and Kounios, is a delicate mental balancing act. At first, the brain lavishes the scarce resource of attention on a single problem. But once the brain is sufficiently focused, the cortex needs to relax in order to seek out the more remote associated in the right hemisphere, which will provide the insight. 'The relaxation phase is crucial,' Jung-Beeman said. 'That's why so many insights happen during warm showers.' Another ideal moments for insights...is the early morning, right after we wake up. The drowsy brain is unwound and disorganized, open to all sorts of unconventional ideas."
- "One of the surprising lessons of this research is that trying to force an insight can actually prevent the insight. While it's commonly assumes that the best way to solve a difficult problem is to focus, minimize distractions, and pay attention only to the relevant details, this clenched state of mind many inhibit the sort of creative connections that lead to sudden breakthroughs."
- "'There's a good reason Google [figures he would use THEM as the example - ed.] puts Ping-Pong tables in their headquarters,' Kounios said, 'If you want to encourage insights, then you've got to also encourage people to relax.'"
So, how does this relate to one of my favorite topics for rumination, namely how Microsoft does its interviews?
I've complained in the past about brainteaser interviews and how unfair they are, and the article seems to support that: a stressful situation like a job interview seems about the worst time to ask somebody to come up with an "Aha!" moment, being a prime example of the "clenched state of mind." I think a lot of people (the ones who got past the "Well, *I* know the answer" self-justification) rationalized asking such questions by figuring that people who couldn't get them just needed to focus harder, which the article implies is incorrect.
BUT I have been struck by two trends, which may be related. Microsoft has (largely) stopped asking brainteaser questions since around the year 2000; and Microsoft's stock hasn't gone up much since around the year 2000. Is it possible that these two are related--that brainteaser questions are actually a better way to interview developers than whatever replaced them?
I'll point out that some people did get "Aha!" brainteaser questions right, even during the stress of an interview. I was one of those people; for whatever reason, I was able to relax my brain enough for the right neurons to fire in the proper hemisphere. The question then becomes, is this the same kind of brain activity that you need to be a good developer? You could argue either side of this; certainly debugging a stress failure two days before you are supposed to ship is a stressful situation, the solution is often an "Aha!" realization of how the bug occured, and there may not be time to go take a warm shower while you are trying to figure it out. But at the same time, a lot of the skill in programming is in doing good design upfront, and given the current state of software design, this is also a series of "Aha!" moments (The Soul of a New Machine has a chapter titled "Wallach's Golden Moment" which describes such an event during the design of the machine; the situation involves somebody thinking about a problem all day, then coming back the next morning and "Suddenly, without thinking about it" having a moment of insight; at the end, as Kidder writes, "As for Wallach, after he had drawn the diagram, he started at it, wondering for a moment, 'Where did that come from?'"). Perhaps we want people who can come up with "Aha!" moments, but it's OK to give them time to shower (probably not a bad thing to encourage for general programmer hygiene). Thus, a good approach to brainteasers might be to pose one early in the morning, and then, after the candidate has spent some time on recruiting shuttles and waiting in building lobbies, come back at the end of the day and see if they have thought of anything.
In his book How Would You Move Mount Fuji?, William Poundstone writes about the approach to solving tricky problems: "To deal effectively with puzzles (and with the bigger problems for which they may be a model), you must operate on two or more levels simultaneously. One thread of consciousness tackles the problem while another, higher-level thread monitors the progress. You need to keep asking yourself 'Is this approach working? How much time have I spent on this approach, and how likely is it to produce an answer soon? Is there something else I should be trying?'" There is a nice parallel with the fairly conscious process that Poundstone describes and the subconscious one that the brain researchers describe in seeking eurekas. I pulled that quote from a review I wrote of the book, which summarizes my thinking on interviews as of mid-2003. I generally still agree with that (except for some waffling on the brainteaser question), and in the context of the discussion here, I like my point that brainteasers might be useful so that you can observe how the candidate approaches a difficult and ill-specified problem with an unbounded solution space (which, of course, is NOT how people used them; they wanted the right answer).
Here's a more radical thought. In the Slashdot article I quote James Fixx, in More Games for the Superintelligent: "While the less intelligent person, unsure of ever being able to solve a problem at all, is easily discouraged, the intelligent person is fairly sure of succeeding and therefore presses on, discouragements be damned." Perhaps what is really happening is that confident people assume they will succeed, which relaxes them (and their cortex) enough during the interview to actually allow them to succeed. Thus confidence becomes a self-fulfilling prophecy--not only during the job interview, but during the job itself. It would seem anathema to modern interviewing techniques to worry not about the answer the candidate gets OR about how they approach the problem, but simply whether they THINK they will be able to solve it--effectively boiling the interview down to the single question "Do you think you are an awesome developer." Maybe asking brainteasers is effectively the same thing, because confident people will solve the brainteasers, and later they will find the tricky bugs and come up with the tricky design breakthroughs. Maybe the awesomeness question IS the only one you need to ask during an interview. And since you CAN judge confidence in the first 15 seconds, maybe those quick judgements are correct--it IS all about the firm handshake and the level stare.
I'm also intrigued by the notion of questions that could be solved either by brute force or by insight. We could ask candidates such questions, figuring that most people would at least get an answer and therefore not hate us for asking impossible questions; then we could ask the candidate HOW they solved it (or just hook them up to a brain scanner first), and based on the answer to THAT seemingly innocuous question, we could do something clever (precisely what, I haven't figured out yet).
By coincidence (I think), that issue of The New Yorker also had an all-time classic article called "Dr. Kush", about California's medical marijuana laws and the effect on the pot industry in the state. The author's approach was to travel the length of the state, find people involved in various aspects of the business, and then spend some time getting baX0Red with them. From the haze of the article emerges the fact that there are two strains of the cheeb, indica and sativa: "Indica is a heavier, numbing drug; sativa is better for doing creative work or listening to music." OK, I can't be the only programmer who reads the magazine; there must be somebody at Google right now who is also putting two and two together. Surely the Big G can find some doctor who would prescribe headies as a suitable cure-all for the niggling physical ailments that befall programmers who spend too much time chained to a desk. A blanket prescription for the whole Googleplex would seem in order: dump those Ping-Pong tables, bring in some papasan chairs, and let a thousand cortexes relax. And the best part is, they already have all the free food they could want.
Posted by AdamBa at August 29, 2008 07:36 PM
A company's stock performance only indicates what the stock market thinks of the company. As far as the market's concerned, Microsoft's performance has been consistent, and so, boring. There have been many more interesting companies to speculate in. There's also been the constant legal threat to contend with (big target = lots of lawsuits) and that Microsoft has achieved a dominant position in most markets - the thinking is that it can only go down from here.
Don't consider stock market activity as rational. It's utterly irrational, driven only by where speculators think they're going to see price movement - and their actions invariably cause that price movement. Witness crude oil prices. There has been slightly increased demand from growing economies, but far less than would be necessary to shift prices this far this fast. What's happened is that speculators have observed a tiny shift and piled in to experience a hoped-for gain, which (as a result of their actions) has occurred. Then they've piled more in hoping for a repeat performance which (guess what) occurred. It takes something happening to dent their confidence in increased gains to reverse the trend - running out of credit (house prices) or the announcement of a fall in demand (oil) and suddenly the bubble bursts.
In actual real financial performance Microsoft have been stellar, increasing Windows' share even further and massive revenue increases in servers and tools. That's safe and dull for pension schemes to invest for gaining dividends.
The theory of markets is that they work perfectly if everyone has perfect information. You'd think in the Internet Age that traders would have all the information they need. Unfortunately they work far more on recirculated rumour than on actual fact.
Posted by: Mike Dimmick at August 31, 2008 04:22 PM
Identifying the lack of brainteaser questions as the cause of Microsoft's lack luster stock performance is something even Malcolm Gladwell wouldn't try to claim :). As Malcolm is slowly starting to figure out correlation and causation are not the same thing.
It takes 10 years of practice to get really good at something and the only way to improve is to continually attempt tasks that are just outside your competence (see "The Expert Mind" in Aug 2006 Scientific American). You mentioned that you were able to answer the brainteaser questions. I'm curious to know whether you have had prior practice at answering those questions? For instance are you a crossword fan or did your parents quiz you over the dinner table as a child? Having confidence that you can perform a particular act often comes from the knowledge that you done something similar in the past.
Posted by: Andrew at September 2, 2008 02:01 PM
Mike, I agree the stock market can be a bit zany, but I don't think that lasts for 8 years while your profits go up and up. The market is penalizing us for uncertainty created by mistakes we have made, which arguably are due to not having employees who are as good as they used to be.
Andrew, I did do crossword puzzles (although my parents did not quiz me at the table). But I don't see why your point contradicts mine; it doesn't matter WHY you are confident, it could be because of experience or it could be because you just decide to be that way.
Posted by: Adam Barr at September 3, 2008 11:13 PM
Anyway, as one of the big brains with lots of moments of insight once hired by MSFT and now frustrated to the point that I just started interviewing outside: I think a review system that rewards political astuteness and penalizes insightful endeavours (you wasted your time on designing a generic dataflowengine instead of helping out your teammates with their bugs? Bad 60! See, your lead just checked in an eerily similar dataflow engine out of nowhere, and he was busy attending meetings all year! Patent and promotion!) is more to blame for lackluster stock performance than anything else.
Posted by: merk at September 6, 2008 02:18 AM
You're ignoring the subset of the population that is supremely confident of its abilities but without any real basis (e.g. the current inhabitant of the Oval Office). Admittedly one would hope most of those people don't end up becoming programmers - they get sucked off into the financial industry (obviously) - but still, one or two slipping through the interview process could badly muck up the works.
Posted by: Becky at September 20, 2008 08:46 AM
So are you saying that you should just hire confident people? It's been established that incompetent people are usually very confident in their abilities and the reverse is often true. Check out the Dunning-Kruger effect.
The next time you interview somebody with 10+ years of practical experience on projects of ever increasing complexity in the areas you care about but they tell you that they consider themselves only mediocre and still have a lot to learn then hire that person and watch the sparks fly. As for people like merk above who profess to have big brains, let them move on.
Posted by: Andrew at September 24, 2008 11:05 AM
Just for the avoidance of doubt:-
s/very confident in their abilities/very confident about their abilities/
Posted by: Andrew at September 24, 2008 11:09 AM
Remember, I am talking about brainteaser-like jobs such as programming, not being a politician. Since being a politician doesn't particularly involves brainteaser-type activities, the effect doesn't matter there.
I don't know if there are a lot of competent people who feel they are mediocre; I do think there are a lot of competent people who are also modest and would STATE that they weren't so great, which is a problem you would need to handle during an interview. But my point is that a competent person who genuinely thought they were mediocre would turn out, almost by definition, to not actually be so competent (in a field like programming that required a relaxed cortex).
Posted by: Adam Barr at September 25, 2008 10:07 PM
I don't really understand your last sentence. Clearly a competent person who thinks they are mediocre is still competent and cannot "turn out" to be incompetent.
Did you read up about the Dunning-Kruger effect? Part of their study included logic tests so it seems reasonable to apply it to coders as well as politicians.
As I said above, peer-reviewed studies have proven that it takes 10 years of purposeful practice to get good at anything and competent people tend to be less sure of their abilities than incompetent people because they know what they don't know. In this context modesty looks a lot like self-knowledge.
Posted by: Andrew at September 29, 2008 01:12 PM