Because they're given the freedom to work on whatever they want to. This leads to doing all the fun stuff and not committing to bland tasks or work that usually lacks an immediate payoff in terms of reward. Working on a engine cannot be fulfilling or rewarding until it's about 2/3s the way done. When you can actually get into the environment you're making and look around without staring at code constantly.
You can see this trend based on how they simply tack things onto the engine as needed. No one wants to spend months and even years writing a new engine from the ground up when someone is working on something fun like Portal 2 or DotA2, not to mention those people parading around and telling others how fun it is in order to get their help. People don't need to commit to projects too. People can just getup and leave a project whenever they want to, so the latest and greatest replaces whatever they were working on.
Essentially their company works in the same way social networking sites work. While the company may have been originally formed around the goals and ideals of creative freedom, they originally had a desire to make one of the best engines out there and show the world. That was completed and since then that drives has completely died out in favor of maintaining their nirvana. People who worked on the source engine either have changed or aren't represented by the company as a whole anymore.
HL3, particularly the next Source engine wont be worked on as that's a major undertaking with very few short term rewards. That's why structure is sometimes needed when you're invisioning something huge. Creative freedom and lack of direction can be great for inspiring little ideas, but not for completing a large one. Arenanet gave their team a lot of creative freedom for GW2, but they are all there for the same reason... To make a game, the game, Guild Wars 2 the best game ever.
People at Valve are just there to make the company better... Which is too vague for a lot of people, especially if you want to start an immense project. This amounts to like maybe 2-3 of their staff working on their next engine because they know it has to get done and the rest of their staff running around willy nilly to the next shiny project because they can. It'll probably remain that way till everyone else can see the engine in a state that is closer to what is being envisioned and testable, which takes a very long time if only a couple people are working on it. Not to mention from the sound of it there is very little rewards internally from doing something that needs to get done, but no one wants to do it.
Sometimes leadership is needed, other times it just gets in the way, can be pointless, and sometimes even detrimental. This is an example of when leadership is needed.
The only thing Valve has done since the Orange Box is Portal 2. Alien Swarm was a mod they bought, L4D was being made before the company bought up the mod team (L4D2 is just L4D with some additional content), DotA 2 is just a rip off of the general DotA franchise and doesn't seem to represent anything new or innovative. That just adds TF2 which definitely shows that the company has some love for. I'm sure it's the same people working on it that were working on it when they first bought the TF team.
See the trend here? They seem to just buy up people with new or innovative ideas, but once they're indoctrinated into the company and produce that work that seems to stop. Valve sounds like a great company to work for, but not one that gets things done. Maybe if they had votes for projects and they were allowed structured choices that would be different, but when they can just run around willy nilly that doesn't bode well for the company overall.
Left_SHifted wrote:...by which I mean, most of the people want to work on the next "billion dollar" idea rather than work on something that might be near shipping date.
Left I completely disagree. I don't think the majority of the company are working on the next billion-dollar idea, rather they're all working on short term ideas that are close to blooming in terms of work that's been done. It's easy to do and the most attractive thing to work on. Since the company is mainly fueled by gratification for completing pieces of work... In other words seeing your work shipped and played. That's also why after L4D1 L4D2 almost immediately came out and Portal 2, for what the game is compared to Portal 1, which was a side project, was shot out the door lickidy split. You can totally see trends in which the company actually gets interested and involved in projects, verse ones that just stagnate.
Episode 3 isn't a huge project and that's pretty much been shelved.
None of this is to say an organic environment is bad. I think it's one of the best ways to treat your employees, but they give waaaaay too much freedom to people who may not have a killer idea even if they're amazing at programming physics or flow calculations. Some people just suck at managing themselves and others. They could easily add structure in a democratic nature too. Vote on ideas people come up with and then work on them together as a group... It's not hard to be structured without being intrusive.