In this interview, Scott Swigart, interviewed Shawn Burke of Microsoft. Shawn Burke regarding the way software is developed within Microsoft. Shawn is a Director in Microsoft’s .NET Developer Platform group. Currently, Shawn is focused on building shared-source projects focused on new developer technologies from Developer Division. Since he started working at Microsoft in 1997, he’s worked on Visual J++, Windows Forms, and Visual Studio.
This interview covered a wide range of topics some of which follow:
- Eating your own Dogfood at Microsoft
- Product Idea Triage at Microsoft
- The difference between Product Milestones, CTP’s, and RC’s at Microsoft
- Closed Source Cores with Open Source Add-On’s
- Extensibility and Software Development
- Security, Reliability, Scalability and other “ities”…
Scott Swigart: Shawn, thanks for taking the time to chat with us. If you wouldn’t mind, could you introduce yourself a little bit?
Shawn Burke: My name is Shawn Burke. I’ve been at Microsoft going on 10 years now. I’ve spent most of my career at Microsoft on the Windows Forms team. Originally it was the WFC team, which was part of the Visual J++ product. On Windows Forms (which became a big chunk of the .NET framework) I went from just a software engineer to lead engineer up to development manager on that product. I ran that team for three or four years. When VS 2005 shipped in 2005, I was ready for a change. I saw what I thought was a bunch of opportunities for Microsoft to improve the way that we develop our software, specifically around driving good contextual feedback into our products at an earlier point in the development cycle.
I went to Scott Guthrie and said, “Hey, why don’t we create a team that’s fully focused on consuming early technologies here at Microsoft.” We’re going to expose our learning (and what we’re doing) really transparently out to the community. We’re trying to drive a tight feedback loop from the community into the development team who is actually building the product. We’re trying to improve quality all along the axis.
Shawn: Kind of, yeah. The dog food thing is usually more at a consumer-level. We’re really focused on development technologies. We’re focused on, “Hey, some technologies are just getting off the ground. What can we go out and try to build on top of that technology and really drive scenarios?” Can we take actual Microsoft plugged-in engineers and do things with technology and influence how it develops? It’s an extension of the dog-food-eating thing, but it’s a little deeper than that.
Scott: That makes sense. My background is also with developer technologies, and I’ve noticed regardless of whose technology you’re using, that you sit back and look at it and wonder if the company that built it ever tried to do anything “real” with it. There are always a lot of things you stub your toe on when you start.
Shawn: There’s one problem with developer technologies though, and this is something people don’t really often realize. When I build Minesweeper or Notepad or an application like that, it’s very easy for me to enumerate the scenarios that people are going to use on it. If I build a compiler or a framework, it’s very difficult to enumerate the scenarios. There are essentially an infinite number of them. The stubbing-the-toe phenomenon is one we definitely focus on, and one of my goals is to try to avoid that. At the end of the day, it’s really hard to know exactly what people are going to use the development tools and frameworks for.
Scott: Talk to me a little at a really high level about how Microsoft builds software? Where do ideas come from, and how do these get slated for coding? How do they get built? What are some of the things along the process to release and then even to servicing?
Shawn: So traditionally what’s happened is that’s been driven simultaneously bottom-up and top-down. What’s usually happened is that you have teams that have released a prior version of a product. They’ve learned about the space. They looked at the industry. They realize a need by talking to customers in a variety of ways. They start coming up with things they think are going to address the needs of their customer set. They do that via their engineers working with customers across their prior project. They do that via their engineers just having brainstorms about, “Oh my gosh. We could solve this problem this way,” et cetera.
So what ends up happening is that you wind up with a bunch of product ideas early in the product cycle, and people start slating out things that they want to do. We’ve gotten more regimented about this. In the past, it was more like we’d just write everything down that we wanted to do, and we’d start going and doing it. Then, farther down the road, we’d have to start cutting things.
We’ve gotten better about that. We have quite a bit more top-down influence now. Not over specific features, but what ends up happening is that the feature teams have a good read on what they think they need to do. What happens top-down is that we have what are called “pillars” for a given product.
For Visual Studio 2005 one pillar was “Please the VB [Visual Basic] customer.” The VB customer (from the old Visual Basic days) was not terribly thrilled with our VS 2003 and 2002 releases. It didn’t really address their needs in a way that we thought it was going to. For 2005, one of the pillars was to address the needs of the VB customer.
From the top-down we’ll have a few pillars, and the individual teams will need to justify how each one of their features fits into one of those pillars. At that point, there tends to be some agreement on what people go off and build.
The process varies from team-to-team and from division-to-division within Microsoft. In the developer division, the developers and the program managers will sit down and spec out a feature. PMs will write the specs, and the developers will give them technical input. The developers may write prototype, since the best way for most people to learn about a feature is to try and write it.
Then it becomes an iterative process where you refine the specs as you start to develop the product. As time goes on, resource constraints usually cause you to trim down what you’re going to deliver. You just iterate from there.
Scott: So software a lot of times goes through different milestones, you’ll hear about a product being an M1 or an M2.
Scott: There are CTPs and there are RCs. What is all that? What’s a layman’s glossary?
Shawn: Usually what happens is similar to Boyle’s Law: work expands to fit the space you give it, much like a gas. So you need to put constraints in there to get things to ship.
One of the ideas of milestones is to force teams to think about their products and their features in manageable chunks. The risk of having somebody go off and crank on a feature unsupervised for months or years at a time means you’re going to have issues. That just doesn’t work. So what you do with milestones is force people to break things into manageable chunks. You make them tightly figure out exactly what the costs are going to be, figure out exactly what the testing recommendations and requirements are going to be, etc.
Usually what happens is that teams take their features for an entire product and break them up into milestones: milestone one (M1) we are going to do this feature, milestone two (M2) we are going to do this feature, etc. At the end of each milestone, you have a mini-release. A lot of times you do Community Tech Preview (CTP) at the end of a milestone. Usually at the end of a milestone your at a know point where, which is to say the product is at zero active bugs at a certain priority level. You figure out some filter for your bugs, and you say we’re going to drive to zero at the end of the milestone. Then you start over again with the next milestone.
As milestones go on, you are more and more careful about adding features. Milestones are also used between teams. If team A needs a feature from team B, team B will do that as an earlier milestone so that team A can pick it up sooner.
Another thing we started to do is add something that we call milestone quality, MQ. Some people also call it M-crit, which is essentially a special milestone where we don’t actually want teams writing code. What we want them to do is make sure their processes work properly. We want them to make sure they’ve cleaned up any bugs that are left over from the prior product cycle, or bugs that have come in after the release of the product, time to improve check-in systems, things like that.
We’ve added this so teams get to focus on their infrastructure, which is a good thing because historically it hasn’t been on the schedule. It’s the old adage where a guy is walking down the path and sees two other guys trying to cut down a tree, and he says to them, “You know guys, that tree would get cut down a lot faster if you would just sharpen that saw you’ve got.” and the guys reply, “We don’t have time to sharpen this saw, we’ve got to get this tree cut down.”
So giving teams a time to really sharpen the saw and improve their efficiency is a good thing. That’s what usually happens in MQ. After that you usually have M0 which is the formal product scheduling and planning milestone and then you roll into the coding milestones – M1, M2 – from there. After that you usually have the Beta and RTM milestones.
Scott: We’re looking at open source and closed source, and Microsoft has obviously decided that a closed source way of doing business makes sense. What’s your impression of why that is? What’s your impression of why Microsoft doesn’t just open source everything?
Shawn: Well there’s a whole set of reasons. This project that I’m running now is an open source project. We’re building on top of the new technologies, and the things we build are actually open source. We are actually doing both. We take customer contributions, the whole bit.
What’s ending up happening, what is starting to evolve, specifically with my team (and we are looking at duplicating the model) is making a hybrid model where you have closed source cores, and open source projects focusing on added customer value, extended components, widgets, and stuff like that. As far as why Microsoft doesn’t go open source, there’s a whole set of reasons for that. I think the number one reason is just around IP pedigree. The issue with open source is that it’s very, very difficult–well it’s impossible–to verify with a hundred percent certainty that every piece of idea in there is licensed properly to be in there.
It’s very difficult for me to know if some developer that contributed to my project saw some code in some other project that he liked, and he picked that code up and dropped it into the project and maybe just tweaked it a little bit.
. “When your organization picks up software, you are taking on some amount of risk that there could be an IP issue with that code. That’s the number one issue that we see with open source. You don’t know how much risk there is, and there is limited, if any, IP protection provided. When people adopt Windows they are provided with significant IP protection.”
That’s a core reason. If you look at most open source projects, and actually just to be clear I’m a fan of open source, I think open source drives a lot of value and is great for a lot of types of projects. But if you look at most open source projects, there’s a couple of things I think that are important about them, and one of them is that they are not that different than closed source or proprietary projects.
Open source projects have a few key people that do ninety-five percent of the work, and then they have a bunch of other people that kind of help out. If you look at Microsoft at any given project, they have a few key people doing ninety-five percent of the work and you’ve got a bunch of other people that do other stuff. So the models end up looking pretty similar the more you squint at them. The big difference is just that a model in which you pay people and whose job is to show up and write software every day, it’s a little easier for us to drive accountability and drive productivity over longer term.
Scott: It seems to me too that one of the advantages of open source is that it’s easy to get a daily build, it’s easy to get the latest drop of something. Around open source projects that have achieved critical mass, even if a lot of people aren’t necessarily contributing to the code, the project is getting a lot of people using the code and submitting bugs, feedback, and feature requests. It seems like Microsoft over the last few years has addressed that same issue a different way with things like CTPs and their bug tracking system known as Connect .
Shawn: I think it’s important to point out that those aspects of taking customer feedback, issuing bugs, having rapid turn around, those things are separate from how the source is licensed.
Scott: Well, I agree. So, I wonder if you can take a minute and talk about CTPs and Connect specifically, and the rationale behind those. What does Microsoft get out of those? What do customers get out of those?
Shawn: One of the issues that we’ve had in the past, and this is one of the reasons that I went off and I started the team that I did, is that historically when the product got into beta and customers were actually able to use the product, it was basically too late for them to give any feedback that we could act on.
If we sent out a beta, and we’re three months before shipping, and somebody says, “Well, I don’t like the way this works,” all of a sudden we’re in a situation where we have to ask “Should we slip the product for a year because of this?” It’s just often too late to make those changes.
One of the benefits of having a milestone system (and then using CTPs where we drive the product to a higher quality bar at an intermediate point) is that people can pick up those CTPs and give us feedback at an earlier point in time.
That leads into where Connect becomes valuable, because Connect and the MSDN Product Feedback Center feed directly into our internal databases. We see those bugs just like any other bugs. They have no different appearance in the database.
That gives us a chance to see customer issues alongside testing-team issues. We do our typical triage process and address them earlier. I think the value of CTPs has definitely been to get people involved sooner.
Scott: So, you said you’re a fan of open source and you spend a certain amount of time around open source. How do you feel that Microsoft is influenced by things that are happening in the open source world? And how do you see the open source world being influenced by things that are happening in closed source proprietary shops like Microsoft?
Shawn: That’s a good question. I think that open source really, at the end of the day, has forced us to take a closer look at how we relate to customers. I think the expectations of customers have changed because of open source, because it gives them freedom to choose. It lets them modify things for themselves, even though I think, a very small percentage of customers actually do that.
It’s forced us to think deeply about how transparent we are. I think if you go back ten years or even five years, the way we interacted with customers bears very little resemblance to the way that it looks today. And I think all the changes that we’ve made around being more open with our product plans and source are nothing but good. We’re licensing all kinds of stuff under Ms-PL (Microsoft Permissive License) these days.
It’s definitely driven us to think about our value as being not purely based on our source code, but based on our brand, and our product, and our developer community. Open source work has really driven that direction not only as a competitive response to the open source world, but really as a “Wow that’s a better way of doing things.” response.
Scott: Have you seen anything flow the other direction? Looking for different things happening in open source, have you thought, “Hmm… they are kind of picking up on something we’ve been doing for awhile?
Shawn: Well, a little bit. Like I said, I think at the end of the day the software development practices are probably more similar than different. I think most developers work in similar ways.
I think that you’re starting to see some interesting kind of hybrids out there in the world. And the one that jumps to my mind is Eclipse. Eclipse started as funded by IBM. They had IBM engineers working on this project that happened to be open-source. It’s kind of a hybrid model as well.
I can’t say that Microsoft has necessarily driven that change, but there are definitely things that are more common to the proprietary source model that have crept into open source as well.
Scott: You know, one of the things that came out in a conversation we had awhile back was if you look at the successful open source projects, one thing that they tend to have in common is that they’re very modular and pluggable.
Eclipse is a good example, right? It’s very common for people to build all sorts of extensions to Eclipse. Apache, since forever, has had modules that you can load in.
I get the impression that closed source proprietary is usually focused on analyzing the customer, analyzing the need, building features that address that, and then later there’s recognition that certain things need to be extensible?
For closed source shops it’s always about building a feature. Extensibility has to be justified, and it’s viewed as making the feature more expensive.
In the open source world, a project doesn’t really seem to survive unless it’s extensible from day one.
Can you talk a little bit about how Microsoft views extensibility? And how Microsoft looks at making that tradeoff between just getting the feature out, versus getting it out in a really extensible way?
Shawn: Yeah, absolutely. I worked on frameworks for ten years, and we’re all about extensibility, but a lot of it has been based on what the developers think people are going to want to do. It’s a little bit of hit and miss there. We’ve gotten more regimented about that.
I think it goes with what I was mentioning earlier about having closed source teams focus on extensibility and the open source teams build out that extensibility. I think we’re picking up that method from the way the open source community works because it makes sense.
When we were developing the .NET Framework, for example, we thought really hard about every single member of a class that we made virtual.
The issue you mentioned is cost, and the real issue for extensibility at Microsoft is two things. One is supporting the extensibility point, and that’s actually not that big of a deal.
The real issue with extensibility is supporting it over versions. So, as soon as you open up a feature to be extensible it makes it a lot harder to write V2 because you have to maintain semantics, and calling orders, and event orders, and all sorts of stuff. You even have to look at whatever bugs you had in your earlier implementations. You often can’t fix things in your V2 implementation because people may have written code that depends on that behavior, and if you fix the bug, you break existing apps.
There’s a lot of stuff that goes into making a feature extensible. We certainly want to make things extensible, but we try to constrain it to the things that we believe are key features because of the long-term impact of extensibility.
Scott: One case in point is something like Visual Studio, right? You guys had come out with a shiny new framework. You had to have a development environment on top of it as fast as possible. I’m sure that in the initial version of Visual Studio, the focus was just on getting a really good IDE out there, and not so much getting the most extensible IDE out there, because the framework wasn’t really viable until you had a development environment on top of it.
Shawn: Well, Visual Studio predated the framework a little bit. Devenv, which is now Visual Studio, was originally the environment around Visual C++, because we used to have different IDEs for each language.
When managed code came along, we had an IDE that was written completely in native code, and we shoe-horned in a set of managed code functionality. But the realization we had was, we had an IDE that was extensible, but not with the technology that we were recommending people use the IDE to build.
Scott: Right, right. So you were inventing managed code, but you had an IDE built using unmanaged code – an earlier technology. So to extend, the IDE developers had to use an older technology.
Shawn: Exactly. Yeah, there’s a set of things that were very difficult. Something that we worked really hard on in VS 2005 was to get a bunch of those extensibility points more managed code-friendly.
Scott: So one last question. Software always has all these “ities,” right? Security, reliability, scalability. Since you’ve been involved in how Microsoft builds what it builds, what special things does Microsoft do to make sure the product is secure, reliable, scalable–those sorts of professional, enterprise-class features?
Shawn: Yeah, we do a lot. It’s actually amazing what we do there. Ever since the Trustworthy Computing memo we’ve done very rigorous security reviews of every single piece that’s gone into a product.
For any given feature, even if you know it’s not going to be a security issue, the PMs have people do threat models of the feature–which is to look at all the different kinds of typical threat vectors and think about, “Hey, could you use impersonation here? Could you use spoofing? Could you use denial of service?” Those kinds of things.
For each feature, we do that. And then we also do security reviews at a code level. We run a bunch of tools that look for things like buffer overruns, array access, uncaught exceptions, stuff like that. We have static tools that go and look at the code and look for different sorts of vulnerabilities as well.
Managed code adds a whole other layer of complex security issues that we’ve had to train up our developers on, and have had to build tools around. We have threat reviews done by people, we have code reviews done by our people in security, we have tools that run that try and find things that people aren’t very good at finding–so that’s just for security.
For stress, we have our testing teams that build these very elaborate tests that hit our products with all kinds of things, thousands of times, over days. There’s nothing better than at the end of the product cycle, when one of my developers has a bug that only repro’s after the stress harness has been running for two days.
Shawn: Yeah, seriously. Those are just a beast to track down, because you have to make a guess about the root cause of the problem. Usually, by the time it blows up, you don’t have a lot of good debugging information about what caused the problem.
Scott: Right. It’s kind of like communicating with one of the rovers on Mars.
Shawn: It’s exactly like that, yes.
Scott: You send your message, and four days later, you see if it got there.
Shawn: Exactly. We do that around stress, and we try and drive those bugs really hard. The tough thing about stress is, sometimes, it ends up being a bug in the OS or a bug in a driver–just really random stuff that we end up uncovering.
Security and reliability: We do a ton of work around globalization and localization, making sure things are working right-to-left, making sure things work accessibility-wise (we’re 508 compliant), making sure that things work on different operating systems–especially for the Windows Forms team, because we have to run on Windows 98, Win2K, WinXP, NT4… It was just incredible how many different operating systems we had to test.
Scott: Performance–what are some of the methodologies that you use to just wring more performance out of stuff?
Shawn: Right, performance. We have scenarios that try to mimic customer scenarios, like the amount of time it takes to go from what we call a cold start–which is a machine that’s just been rebooted that doesn’t have any of our code running on it yet–to begin with a cold start to, start the IDE, create a new project, run the project, and display the running form. We measure the amount of time it takes, and then we’ll try and optimize those types of scenarios. We take scenarios like that and then just really drill into them with a bunch of tools and a magnifying glass, and try and eke out exactly who’s taking the time and how we can optimize it.
Scott: I’ll hand you the microphone. What are some things that you see on the horizon that you’re pretty excited about? What are some messages that you’d like to take the opportunity to get out?
Shawn: I’m insanely excited about Silverlight. The technology rolled out over the last few weeks. Silverlight is really just one of these magical convergences of technologies that we have been developing piecemeal where some people said, “Oh my gosh! We can put these technologies together in a really compelling way.”
We have a full tool set. We have this great rendering engine. We’ve got managed code. We’ve got dynamic languages. We’ve got a bunch of other stuff. So I’m really excited about that.
And one of the reasons that I’m excited about Silverlight is that it’s going to enable a client programming model. ASP.NET is an extremely extensible and awesome product, but the kind of server stateless programming model is hard many developers to understand.
A client-side model is a little easier, so maybe we’re going to see a resurgence of people being really excited about writing client side code, because they can crank this out a lost faster.
I’m excited about how we’re looking at how we can be more transparent. I think that’s super-important for us. I think getting our customers more deeply involved and invested in our processes is not only good for us, but good for our customers. And I think that I’m really lucky to be in the Developer Division, honestly, because I think that’s where all the action’s at.
Scott: Thanks for taking the time to chat.