WALL•E and Enterprise Data Landfills

"Life is nothing but imperfection and the computer likes perfection, so we spent probably 90% of our time putting in all of the imperfections, whether it's in the design of something or just the unconscious stuff. "
-Andrew Stanton, director of Disney/Pixar's WALL-E, in an interview on the topic of graphic detailing.

I'm enough of a sci-fi geek that I had to take my kids to see WALL*E the day it opened. I found it so entertaining that, while on vacation, I started browsing around the internet... digging for addititonal tidbits about the backstory.

I came across the quote, above, initially on Wikipedia's Wall-E page.

The simple truth carries across all applications of contemporary computer technology. Technology tools are designed for the "general" cases, and yet, more and more often, we're running into the imperfect, inconsistent, outlying, and exceptional cases.

To follow the thought along, perhaps 90% of what we do as software developers is about trying to get a grip on the complexities of... everything we get to touch on. I guess the remaining 10% would be akin to the root classes... the "Object" class, and the first few subordinates, for example.

Andrew Stanton's quote reminds me of the 90-10 rule of software engineering... 90% of the code is implemented in 10% of the time. (conversely, the remaining 10% of the code is implemented in the remaining 90% of time). I tend to think of this one as a myth, but it's fun thought.

It's dealing with the rough fringes of our data that's among the industry's current challenges, but it's not just corporate data landfills.

I recently heard a report that suggested that technology will get to the point that commercially available vehicles with an auto-pilot will be available within the next 20 or so years. What's really the difference, to a computer, between financial data, and, say, navigational sensor data?

So to flip that idea on its head, again, and you could have more intelligent artificial agents spelunking through data warehouses... WALL-DW ? (Data Warehouse edition)

Then again, I wonder if the 80-20% rule isn't what gets us into our binds to begin with.

Functional Expression

So one more thing crossed my mind about implementing code with respect to art & science, and I had to express it...

I looked up the term "Art" in the dictionary. The first definition is:

• the quality, production, expression, or realm, according to aesthetic principles, of what is beautiful, appealing, or of more than ordinary significance.

For me, regarding coding, it's a matter of remembering a few points:

1. implementation is expression
2. significance is subjective
3. beauty is in the eye of the beholder

So code can be expressed, fundamentally, in a bunch of ways:

• Electronically,
• Numerically,
• mnemonically,
• symbolically,
• graphically,
• gesturally,
• audibly,
• visually,
• etc... ?

Simple, clever, elegant, seemingly natural expressions of all kinds are typically beautiful to a programmer, when they function correctly.

Of course, to me, the most beautiful implementations are implementations that elegantly express its business in a way that's very clear to anyone familiar with the problem domain at that abstraction level, and to the target platform(s).

See also:
politechnosis: Art & Science

Hands on Surface

There's a developing UI paradigm growing, and some of it has been released in the wild.

Multi-point touch screen systems are starting to take shape out of the ether, and it really feels like it's going to usher in a new era of computing. We've talked about a few of them here in the Tech Mill. It's "Minority Report" without the goofy VR glove.

Microsoft's offering in this arena is Surface (formerly "Milan").( http://www.microsoft.com/surface )

From available marketing materials, Surface is much like the other offerings that are under development, with a few interesting differences. Rather than being an interactive "wall", it's a "table". In addition to interacting to a broad range of touch-based gestures, Surface also interacts with objects. Some of it's marketed use-cases involve direct interaction with smartphone, media, and storage devices.

This week, I'm on a training assignment in New Jersey, but within a bus ride to one of very few instances of Surface "in the wild".

I made it a secondary objective to hit one of the AT&T stores in mid-town Manhattan.

I had a lot of high expectations for it, so actually getting to play a bit with it struck me as a touch anti-climactic. The UI was great, but it was clear they cut costs on hardware a bit: responsiveness wasn't quite as smooth as the web demos. It did impress me with the physics modeling of the touch gestures... dragging "cards" around the table with one finger mimicked the behavior of a physical card, pivoting around the un-centered touch point as a real one would.


I was also a bit concerned that the security devices attached to the cell phones they had around the table were some sort of transponder to hide "vapor-ware" special effects. My own phone (an HTC Mogul by Sprint) was ignored when I placed it on the table.


All in all, I was happy to finally get to play with it. Between technology advances and price drops, this UI paradigm will start to make it into the power business user's desk.

I mean, can you imagine, for example, cube analysis.... data mining... report drilling... and then with a few gestures, you transform the results into charts and graphs... then throw those into a folder on your mobile storage / pda device...

I'm still loving the idea of interactivity between physical and virtual (and/or remote) logical constructs...

Imagine bringing up the file server and your laptop on a "Surface" UI, and litterally loading it with data and installing software with the wave of your hand....

or...

Having a portable "PDA" device with "big" storage... in fact, big enough to contain a virtual PC image... In stand-alone mode, the PDA runs the VPC in a "smart display" UI. When you set it on a Surface, the whole VPC sinks into it. You get access to all the Surface functional resources including horsepower, connectivity, additional storage, and the multi-touch UI while the PDA is in contact. When you're done, the VPC transfers back to the PDA, and you can take it to the next Surface UI in your room at the hotel, or the board room (which has one giant "Surface" as the board room table.)

The preview is over at AT&T today. According to Wikipedia, Microsoft expects they can get these down to consumer price ranges by 2010 (two years!).

Multiprocessing: How 'bout that Free Lunch?

I remember reading an article, a few years back...

The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software

Its tagline: "The biggest sea change in software development since the OO revolution is knocking at the door, and its name is Concurrency."

Mr. Sutter's article suggests that because CPUs are now forced to improve performance through multi-core architectures, applications will need to typically employ multi-threading to gain performance improvements on newer hardware. He made a great argument. I remember getting excited enough to bring up the idea to my team at the time.

There are a number of reasons why the tag line and most of its supporting arguments appeared to fail, and in retrospect, could have been predicted.

So in today's age of multi-core processing, where application performance gains necessarily come from improved hardware throughput, why does it still feel like we're getting a free lunch?

To some extent, Herb was right. I mean, really, a lot of applications, by themselves, are not getting as much out of their host hardware as they could.

Before and since this article, I've written multi-threaded application code for several purposes. Each time, the threading was in UI code. The most common reason for it: to monitor extra-process activities without blocking the UI message pump. Yes, that's right... In my experience, the most common reason for multi-threading is, essentially, to allow the UI message pump to keep pumping while waiting for… something else.

But many applications really have experienced significant performance improvements in multi-processor / multi-core systems, and no additional application code was written, changed, or even re-compiled to make that happen.

How?

• Reduced inter-process contention for processor time
• Client-server architectures (even when co-hosted, due to the above)
• Multi-threaded software frameworks
• Improved supporting hardware frameworks

Today's computers are typically doing more, all the time. The OS itself has a lot of overhead, especially Windows-based systems. New Vista systems rely heavily on multi-processing to get performance for the glitzy new GUI features.

The key is multi-processing, though, rather than multi-threading. Given that CPU time is a resource that must be shared, having more CPUs means less scheduling collision, less single-CPU context switching.

Many architectures are already inherent multi-processors. A client-server or n-tier system is generally already running on a minimum of two separate processes. In a typical web architecture, with an enterprise-grade DBMS, not only do you have built-in “free” multi-processing, but you also have at least some built-in, “free” multi-threading.

Something else that developers don’t seem to have noticed much is that some frameworks are inherently multi-threaded. For example the Microsoft Windows Presentation Foundation, a general GUI framework, does a lot of its rendering on separate threads. By simply building a GUI in WPF, your client application can start to take advantage of the additional CPUs, and the program author might not even be aware of it. Learning a framework like WPF isn’t exactly free, but typically, you’re not using that framework for the multi-threading features. Multi-threading, in that case, is a nice “cheap” benefit.

When it comes down to it, though, the biggest bottlenecks in hardware are not the processor, anyway. The front-side bus is the front-line to the CPU, and it typically can’t keep a single CPU’s working set fresh. Give it a team of CPUs to feed, and things get pretty hopeless pretty quick. (HyperTransport and QuickPath will change this, but only to the extent of pushing the bottle necks a little further away from the processors.)

So to re-cap, to date, the reason we haven’t seen a sea change in application software development is because we’re already leveraging multiple processors in many ways other than multi-threading. Further, multi-threading options have been largely abstracted away from application developers via mechanisms like application hosting, database management, and frameworks.

With things like HyperTransport (AMD’s baby) and QuickPath (Intel’s), will application developers really have to start worrying about intra-process concurrency?

I throw this one back to the Great Commandment… risk management. The best way to manage the risk of intra-process concurrency (threading) is to simply avoid it as much as possible. Continuing to use the above mentioned techniques, we let the 800-lb gorillas do the heavy lifting. We avoid struggling with race conditions and deadlocks.

When concurrent processing must be done, interestingly, the best way to branch off a thread is to treat it as if it were a separate process. Even the .NET Framework 2.0 has some nice threading mechanisms that make this easy. If there are low communications needs, consider actually forking a new process, rather than multi-threading.

In conclusion, the lunch may not always be free, but a good engineer should look for it, anyway. Concurrency is, and will always be an issue, but multi-core processors were not the event that sparked that evolution.

The Mature Software Industry, Corporate Specialization, p2

In driving down I-293 around the city of Manchester one night this weekend, I noticed some of the old factories across the river were lit up so you could see the machinery they contained. Those machines have probably been producing goods reliably for decades.

In my last post, ("Corporate Specialization") I used an analogy of power plants to describe how software engineering groups might someday fit into the corporate landscape.

I found myself thinking that a more precise analogy would be to liken software application development to... hardware application development.

When it comes down to it, hardware, software... the end user doesn't care. They're all just tools to get their real jobs done.

I remember seeing this when I was a kid. I recall observing that when the Atari 2600 has a cartridge inserted, and it's powered on, the hardware and software components were functionally indistinguishable. The complete system might as well be a dedicated-purpose hardware machine. It became an appliance.

Modern platforms don't really change that appliance effect to the end-user.

So, aside from operators, I'm sure these classic B&M manufacturers have technical people to maintain and manage their equipment. I'd be surprised to find out that they keep a full team of mechanical engineers on the staff, though. It seems to make sense that a mature software development industry will start to look much more like that over time.

Further, take a look at computer hardware. It's undergone some maturing over the past few decades too. There really aren't many companies that actually bake their own. I remember tinkering a bit with chips & solder. Then, I started building PC's from components. While my current desktop at home is still one of my own "Franken PC's", I think it's been over a year since I even cracked the case on it. I suspect that when I finally break down & replace the thing, it will be 100% Dell (or maybe Sony) [and Microsoft].

With respect to software engineering, never mind all that FUD we're hearing about IT getting sucked into business units. That's good for "operators" and "maintenance" types, who become analytics and process management in the new world order. I suspect the heavy lifting software engineering will eventually be done mostly by companies that specialize in it.

With that, it might be more educational to look at the histories-to-date of the industrial-mechanical and electrical engineering groups to see the future of the software engineering group.

I think this might be where MS & Google are really battling it out, long term. As the software industry continues to mature, the companies with the most proven, stable software components will end up with the market advantage when it comes to building "business factories" out of them for their clients. ...or maybe it will just be about the most matured development and engineering processes... or maybe both... ?

Corporate Specialization

There's an old adage: "Anything you can buy, you can produce yourself for less."


In our business, we're well aware that there are a few fundamentally flawed assumptions with that sentiment. Despite the barriers to entry and many other recurring costs, somehow the idea seems pervasive in the business world.


I started my career in a consulting shop that insisted it was a product company. Then I moved, through market forces, into products based companies. I stayed on board with some of them long enough to help shut out the lights and unplug the servers when the sales didn’t hit the marks. The other big problem I’ve seen with product shops was that it’s engineering group typically went through its own “release cycles”… once the product was released, my team was cut to the bone.


I've never been in a classic IT shop, per se, but I’ve definitely been on tangent IT-support projects within “product” oriented companies. In IT groups, I’ve often thought that companies might see IT projects as frills and overhead. At some level, I think the pervasive IT-alignment is a counter measure to that idea. Still, it seems IT projects are typically viewed as liability, rather than asset. When it comes time for the company to refocus on its core competencies, (which is inevitable in the ups & downs of business) IT projects are prime targets for cutbacks.


Since the “.COM bust”, in these types of companies, an engineer on the staff for just three years is often seen as a “long-timer”… but no one feels bad about that, since a lot of these companies fold in that time frame, as well.


After experiencing the product based companys' ups & downs, and seeing many colleagues who had already been through the IT shops, I’m convinced… the outsourced project / consulting route is really the wave of the future, even more than it has been in the past. It’s only a matter of time before business admits this necessity as well.


It makes sense... I wouldn’t figure on every company to own & operate their own power plant.



Why should they typically want their own software engineering group if they don't have to?

[Edit: See also Part 2 Here]

The Wetware Bus

A colleague pointed me to this post about display size improving productivity.

It brings up a favorite topic of mine... With my current laptop (@1024x768), it's a bit of a pet-peeve.

It totally makes sense that tuning bandwidth to wetware improves productivity. Display real-estate is the front-side bus to wetware!


That's why I also like the Dvorak keyboard layout... Personally, I never achieved the level of proficiency with twenty +/- years of QWERTY (with formal training) that I have with the Dvorak layout over the past three years. Dvorak's a hard curve to get past, but I committed to it in the interest of tuning bandwidth (and maybe hoping to reduce RSI).