poliTechnosis

Choen the counterBot

2011-11-15T20:26:00.008-05:00

I'm pretty excited... both my kids seem interested in computers and what you can do with them. My daughter, especially, is very interested in learning bits and pieces about programming. My son is interested, but not so much in the nuts & bolts of programming. He is interested in numbers and performance, though. They're both young, though... he's in 1st grade, and she's in 2nd.

She's expressed some interest in programming before. Last time we talked about application design... she got extremely ambitious and drew about twenty pages of "designs" (hand-drawn mock-ups of screens) for games that she wanted to create. (Essentially knock-offs of games she's played on favorite websites like PBS Kids and Pixie Hollow.)

More recently, she was using my computer, and discovered that when she's logged into my computer on her account, she has access to Visual Studio Express. When she asked what it was, I told her it was tools to write programs, and she just lit up. Immediately, she wanted to start working on a simple UI that mimics a math program she uses at school.

I started to realize, however, that she needs to learn more of the basics before we dive into a full fledged user interface. I started to think about some of the earliest programs I learned to write.

Back when I was in 4th grade, I got my hands on an Apple II, and one of the first programs I learned to write on it was a program that accepted two numbers (start and finish) and simply scrolled out a count between them.

Nowadays, we have object oriented programming, and C#, but I wanted to try to keep it simple and understandable for her. I decided that we would create a "counter-bot" class, and we would create an instance of it named whatever she wanted it to be named (She decided on Choen (pronounced more like Cohen)).

Today, we got the basics of the counterBot class written, and wrote a short program to create an instance of it, initialize it, and start counting, sending the numbers out to the screen. A countBot has beginAt, finishAt, countBy properties, and Start, Count, GetCurrentNumber and isFinished commands.

Here's the code:
counterBot.cs:
--------------------------------------------------------


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace Choen_the_counterBot
{
    class counterBot
    {
        public int beginAt;
        public int finishAt;
        public int countBy = 1;
        private int currentNumber;
    
        public void Start()
        {
            currentNumber = beginAt;
        }

        public int GetCurrentNumber()
        {
            return currentNumber;
        }

        public void Count()
        {
            currentNumber = currentNumber + countBy;
        }

        public bool isFinished()
        {
            bool result=false;
            if (currentNumber > finishAt)
            {
                result = true;
            }
            return result;
        }

    }
}

--------------------------------------------------------

and here's the console Main method in program.cs:
--------------------------------------------------------


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
namespace Choen_the_counterBot
{
    class Program
    {
        static void Main(string[] args)
        {
            counterBot Choen = new counterBot();
            Choen.beginAt = 0;
            Choen.finishAt = 2000000;

            Choen.Start();
            DateTime startTime = DateTime.Now;
            
            while (!Choen.isFinished())
            {
                Console.WriteLine(Choen.GetCurrentNumber());
                Choen.Count();
            }
            
            Console.WriteLine("Finished counting from " 
                + Choen.beginAt.ToString() 
                + " to " 
                + Choen.finishAt.ToString()
                + " by " 
                + Choen.countBy 
                + "'s in " 
                + (DateTime.Now - startTime)
                       .Seconds.ToString() 
                + " seconds!");
      
            Console.ReadKey();
        }
    }
}

--------------------------------------------------------

I got a bit fancy and added the timer code to see how long it took. Together, my son, daughter, and I watched Choen the counterBot count out various scenarios, and got as ambitious as watching it count from zero to two million. In order to get my son's interest, I added the timer to see how long it took. We discovered that my computer can count from zero to two million, "out loud" (showing the numbers on the screen) in 52 seconds. If we hid the console writes, it could do it in less than a second.

Next time we play with it, we'll write a graphic UI (a windows form) and hook Choen the counter bot up to it.

Code Generation

2011-03-11T15:22:00.005-05:00

I've been evangelizing code generation since the work I did at Providus / FRS Global...

One of my arguements on the topic got published by Edgewater...

I love the picture on it... :)

SharePoint Saturday NH

2011-03-04T11:43:00.004-05:00

I'm psyched to be a part of the founding and building of SharePoint Saturday for New Hampshire. It's a part of the Granite State SharePoint Users Group (follow twitter @NHSharePointUG ) or check us out on Linked-IN.

Stay tuned for more info.... lots of exciting stuff happening for the SharePoint community in NH.

Edit: May have jumped the gun on this a bit... the site may not be publicly accessible... yet. I'll update this post when it's made public.

Infrastructure Agility via Cloud Technology

2009-04-24T16:52:00.002-04:00

I'm honored to have just been published on Edgewater's public blog...

It's bit about managing infrastructure agility. The basic idea is architecting your infrastructure so that you can push off parts to different clouds when you need to, for any of a multitude of reasons. The idea goes a bit beyond virtualization.

Check it out:

http://edgewatertech.wordpress.com/2009/04/24/best-practice-cloud-computing/

Workstation Virtualization

2009-04-24T08:46:00.004-04:00

I've been having fun (ya, really... fun!) with MS VPC 2007 SP1 lately.

I've put some VM's on a portable USB disk. USB2 isn't the best connection, but it's workable. It does a few things... adds a spindle to the system config, offloading that overhead from the main disk.

An external disk also means the VM is "portable". I can launch the VM on my laptop or any other host system (like my home system) without any significant difficulty. Even better, by having the VPC config files on the host, rather than on the external disk, you can tune VM settings (like memory and network connectivity) for optimal conditions on the host system.

One other nice time-saver is differencing disks. You can create a "base" virtual hard disk, and create VHD's that are deltas of the base... by doing this, you can create a primary configuration, and then create several machines that inherit that basic config. It came in very handy for a recent product evaluation... I just created a base VM roughly according to what the client expects to host the system on, and then created VMs based on that for each product I wanted to evaluate.

Another nice feature is virtual assist hardware. At first, I didn't know my ThinkPad had it, but it turns out to be a BIOS setting. Flip that on, do a cold boot, and VM performance is visibly better. I knew of some of the other features from past experience with MS VPC 2005, but the hardware acceleration is new to me. (Ironically, my newer home desktop, a 64-bit monster with huge RAM doesn't support the hardware assist... performance isn't a problem, there, tho. )

One more trick: enable the Undo disk option... It put another layer of protection on your VM, allowing you snap a line on your VM at a point in time that you can back-out to. The cool part about this is that the Undo disk is created as a temporary file on the host system, (typically on the primary system drive). This distributes load across the spindles even more, which further improves run-time performance. The downside: when it comes time to commit the undo, it can take a while.

I still love the idea, going forward, of putting client dev environments on a config like this... Not only does it create a nice level of separation between client system configurations, but when you get your hands on better hardware, migration is not an issue.

Reading SharePoint Lists into an ADO.Net DataTable, Updated

2009-02-18T13:23:00.004-05:00

I've been integrating processes with SharePoint lists for some time now. I made a post back in September '08 about how to pull a SharePoint list into an ADO.NET DataTable.

A very kind reader contributed a much appreciated simplified solution. It actually worked for general cases, but didn't solve a fundamental problem (one of the reasons I posted my version of the code to begin with): What if you have a column with odd characters in the name, like "EntityA.Property1"?

Unfortunately, ADO.NET imports columns with certain character encodings as DBNull. I suspect this might be a bug, but the example below shows a very simple WPF form that uses the basic technique with a slight variation to work around the issue:

using System;
using System.Collections.Generic;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Shapes;
using data = System.Data;
using xml = System.Xml;
namespace WindowsApplication1
{
///

/// Interaction logic for Window1.xaml
///

public partial class Window1 : System.Windows.Window
{

public Window1()
{
InitializeComponent();
}

public void OnLoaded(object sender, RoutedEventArgs args)
{
SharePointLists.Lists listsService = new WindowsApplication1.SharePointLists.Lists();
listsService.Url = "http://spsite/_vti_bin/lists.asmx";
listsService.Proxy = null;
listsService.UseDefaultCredentials = true;
xml.XmlNode listData =
listsService.GetListItems(
"Test List",
default(string),
null,
null,
default(string),
null,
default(string));

//Takes care of columns with names like "EntityA.Property1", gets rid of "ows_" prefix:
listData.InnerXml = listData.InnerXml.Replace("_x002e_", "_").Replace("ows_", "");

data.DataSet listDataSet = new data.DataSet();
xml.XmlNodeReader listDataReader = new xml.XmlNodeReader(listData);
listDataSet.ReadXml(listDataReader);
CollectionViewSource dataObj = (CollectionViewSource)Resources["Data"];
dataObj.Source = new data.DataView(listDataSet.Tables[1]);

}
}
}

Here's the XAML component:

<Window x:Class="WindowsApplication1.Window1"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
Title="WindowsApplication1" Height="300" Width="300"
Loaded="OnLoaded"
>

<Window.Resources>
<CollectionViewSource x:Key="Data" />
</Window.Resources>

<Grid DataContext="{StaticResource Data}">
<ListBox ItemsSource="{Binding}" DisplayMemberPath="EntityA_Property1"/>
</Grid>

</Window>

WORKAROUND: Misconfigured Windows-Integrated Authentication for Web Services

2009-02-04T11:37:00.009-05:00

In trying to drive a process from a SharePoint list, I ran across a problem...

I couldn't create a web reference in my C# project due to some really weird problem... In the "Add web reference" wizard, I entered my URL, and was surprised by a pop-up titled "Discovery Credential", asking me for credentials for the site.

Since I was on the local domain and had "owner" permissions to the site, I thought I would just waltz in and get the WSDL.

Ok, so it wants creds... I gave it my own.

Negative...!?!?

After a few attempts and access denied errors, I hit Cancel, and was rewarded by, of all things, the WSDL display... but I still couldn't add the reference.

After quite a bit of wrestling, it turns out there was an authentication provider configuration problem. The site was configured to use Kerberos authentication, but the active directory configuration was not set up correctly. (I believe it needed someone to use SetSPN to update the Service Principal Name (SPN) for the service.)

One way to resolve the problem was to set the authentication provider to NTLM, but in my case, I didn't have, (and wasn't likely to get) that configuration changed in the site (a SharePoint Web Application) I really needed access to.

In order to make it work, I had to initially create my reference to a similar, accessible site.

(e.g. http://host/sites/myaccessiblesite/_vti_bin/lists.asmx )

Then, I had to initialize the service as such, in code:


private void InitWebService()
{
  System.Net.AuthenticationManager.Unregister("Basic");

  System.Net.AuthenticationManager.Unregister("Kerberos");

  //System.Net.AuthenticationManager.Unregister("Ntlm");

  System.Net.AuthenticationManager.Unregister("Negotiate");

  System.Net.AuthenticationManager.Unregister("Digest");


  SmokeTestSite.Lists workingLists = new SmokeTest.SmokeTestSite.Lists();

  workingLists.Url = "http://host/sites/mybrokensite/_vti_bin/lists.asmx";

  workingLists.UseDefaultCredentials = true;

  workingLists.Proxy = null;

  lists = workingLists;
}

What this accomplishes is it unregisters all authentication managers in your application domain. (This can only be done once in the same app domain. Attempts to unregister the same manager more than once while the program's running will throw an exception.)

So by having all the other authentication managers disabled in the client, the server would negotiate and agree on Ntlm authentication, which succeeds.

Anonymous Form Submission to Form Library with InfoPath in MOSS

2009-01-28T10:34:00.003-05:00

Here's a bit of a trick I ran across while helping to develop some MOSS2007 solutions.

I needed to configure InfoPath so that it could submit documents to a site that the submitter would not be able to access. In SharePoint, this is not directly possible.

A common work-around is to set up incoming email for the target list, and submit by email to that. Unfortunately, my client is part-way through a Notes to Exchange migration, so this wasn't practical in the given time frame.

The solution... create two sites, one that is accessible to the submitter, and the other that is not. On the accessible site, create a new, "hidden" list that the user can submit to. Add an event receiver to that list, such that whenever a new item is added, the item is moved to the real intended target using elevated privileges.

Using VSeWSS extensions, create a List Definition project that has something like this in the ItemEventReciever.cs file:


using System;
using System.Security.Permissions;
using System.Runtime.InteropServices;
using Microsoft.SharePoint;
using Microsoft.SharePoint.Security;
using Microsoft.SharePoint.Utilities;
using VSeWSS;

namespace AutoForwardingFormLibrary
{
    [CLSCompliant(false)]
    [TargetList("38aea319-af78-4489-9059-d124c68bf9fe")]
    [Guid("9d0f139b-d9ed-4b6c-b0ba-2353cb3bad85")]
    public class AutoForwardingFormLibraryItemEventReceiver : SPItemEventReceiver
    {
        private SPListItem addedItem = null;

        /// 
        /// Initializes a new instance of the Microsoft.SharePoint.SPItemEventReceiver class.
        /// 

        public AutoForwardingFormLibraryItemEventReceiver()
        {
        }

        /// 
        /// Asynchronous after event that occurs after a new item has been added to its containing object.
        /// 

        /// 
        /// A Microsoft.SharePoint.SPItemEventProperties object that represents properties of the event handler.
        /// 
        public override void ItemAdded(SPItemEventProperties properties)
        {
            addedItem = properties.ListItem;
            if (addedItem == null)
            {
                throw new ArgumentNullException("properties.ListItem");
            }
            SPSecurity.CodeToRunElevated moveItem = moveItem_Elevated;
            SPSecurity.RunWithElevatedPrivileges(moveItem_Elevated);
        }

        private void moveItem_Elevated()
        {
            addedItem.CopyTo(SPUrlUtility.CombineUrl("http://targetserver/sites/jimw/docs/formlibrary", addedItem.Name));
            addedItem.Delete();
        }
    }
}

"Rapture for the Geeks": Singularitarianism 101

2009-01-11T00:09:00.007-05:00

My wife and I were at Barnes & Noble last week. While we were there, she walked over to me with a book, with a grin on her face, called "Rapture for the Geeks". She was making a personal joke with it, but I was too immediately intrigued... which made it all the more hilarious to her.

Being a spiritualist, optimist, scientist, technologist, and a sci-fi fan-boy (I believe the way Albert Einstein did: Everything is a miracle), all my life, I've felt like humanity's on the brink (in geological terms) of something fascinating... Needless to say, I bought it and read it.

The author, Richard Dooling, has done some fiction, but this was (mostly) non-fiction. It turns out to be an introduction to a concept called the "Technological Singularity" for para-technologists. Dooling doesn't offer any insights of his own, but brings together a lot of interesting view points from a lot of notable "Singularitarians", especially Ray Kurzweil, and Bill Joy.

The thing that I found most interesting about this book was not the ideas Dooling was relaying, but the fact that there appears to be a build-up of buzz around it. Even as the crescendo that is the "Moore's Law" prediction persists, so does the interest crescendo around The Singularity.

I remember having conversations with a wise, elder grand-uncle when I was a kid that bordered on philosophical regarding the progression of technology... back in the early 80's. I remember wondering what would happen when humanity's technology over-reached its own capacity to manage it. What would we really do, for example, if we eventually automated ourselves out of all of our "work"?

It's true... technology sure has come a long way in the past 35 years that I've been observing it (my whole life). When I was a pre-schooler, my folks had this awesome TI calculator... it had a red LED display, and did all four arithmetic functions. Almost a decade later, my first programmable computer was a Timex-Sinclair 1000 with 2K of RAM (expanded to 16K) and a cassette player for storage. Today, my preschool-age children play with "old" 1-gigahertz Pentium III machines while my own machine (a Christmas gift from my wife) is a commercial-off-the-shelf quad-core 2.3 gigahertz monster with 8GB RAM and a full Terabyte of disk space in a RAID 0 array.

With humanity's ability to abstract and build on its own technology, it's not a question of "if" we will hit some sort of existentially disruptive technology... the only important questions are "when" and "will life on Earth survive it?"

The really fun part is imagining how The Singularity might manifest itself... artificial intelligence gaining sentience? Discovering the truly united nature of time/space/matter/energy? Teleportation? Limitless energy? Immortality (via perfected nutrition, nano-technology, replicable parts, or even transference into "robotic" bodies) ? Inter-galactic travel? Perhaps God, the uber geek creator of this simulation we call life, will don His Holy "VR gear" to be present to witness the birth of His grandchild(ren). (I wonder which Cloud He might be riding in on... EC2? Blue Cloud? Surely not Azure... :) )

There's some good humor in the book, but the last few chapters are really hard trudging through. It suffers from more than a couple bouts with verbal diarrhea... one, for example, is a ten page rant about making sure you save your work as text. He also comes across as a programmatic poseur to someone who really is a programmer.

If you are already familiar with these ideas, you'll probably be insulted by this book. What kept me reading was the false hope that the author had some synthesis of his own on the subject. That said, if you're not a programmer, and are new to the topic, the first nine chapters are good, and the remaining chapters might be forgivable.

Of course, Dooling does suggest that heightened interest in The Singularity may also just be a symptom of a mid-life crisis. :)

"ETL"ing Source Code

2008-10-14T16:22:00.009-04:00

The past couple weeks, I've been between projects, which has gotten me involved in a number of "odd jobs". An interesting pattern that I'm seeing in them is querying and joining, and updating data from very traditionally "unlikely" sources... especially code.

SQL databases are very involved, but I find myself querying system views of the schema itself, rather than its contents. In fact, I'm doing so much of this, that I'm finding myself building skeleton databases... no data, just schema, stored procs, and supporting structures.

I'm also pulling and updating metadata from the likes of SharePoint sites, SSRS RDL files, SSIS packages... and most recently, CLR objects that were serialized and persisted to a file. Rather than outputting in the form of reports, in some cases, I'm outputting in the form of more source code.

I've already blogged a bit about pulling SharePoint lists into ADO.NET DataSet's. I'll post about some of the other fun stuff I've been hacking at soon.

I think the interesting part is how relatively easy it's becoming to write code to "ETL" source code.

Reading SharePoint Lists into an ADO.Net DataTable

2008-09-30T13:28:00.006-04:00

[Feb 18, 2009: I've posted an update to show the newer technique suggested below by Kirk Evans, also compensating for some column naming issues.]

The other day, I needed to write some code that processed data from a SharePoint list. The list was hosted on a remote MOSS 2007 server.

Given more time, I'd have gone digging for an ADO.NET adapter, but I found some code that helped. Unfortunately, the code I found didn't quite seem to work for my needs. Out of the box, the code missed several columns for no apparent reason.

Here's my tweak to the solution:

(The ListWebService points to a web service like http://SiteHost/SiteParent/Site/_vti_bin/lists.asmx?WSDL )

private data.DataTable GetDataTableFromWSS(string listName)

{

ListWebService.Lists lists = new ListWebService.Lists();

lists.UseDefaultCredentials = true;

lists.Proxy = null;

//you have to pass the List Name here

XmlNode ListCollectionNode = lists.GetListCollection();

XmlElement List = (XmlElement)ListCollectionNode.SelectSingleNode(String.Format("wss:List[@Title='{0}']", listName), NameSpaceMgr);

if (List == null)

{

throw new ArgumentException(String.Format("The list '{0}' could not be found in the site '{1}'", listName, lists.Url));

}

string TechListName = List.GetAttribute("Name");

data.DataTable result = new data.DataTable("list");

XmlNode ListInfoNode = lists.GetList(TechListName);

System.Text.StringBuilder fieldRefs = new System.Text.StringBuilder();

System.Collections.Hashtable DisplayNames = new System.Collections.Hashtable();

foreach (XmlElement Field in ListInfoNode.SelectNodes("wss:Fields/wss:Field", NameSpaceMgr))

{

string FieldName = Field.GetAttribute("Name");

string FieldDisplayName = Field.GetAttribute("DisplayName");

if (result.Columns.Contains(FieldDisplayName))

{

FieldDisplayName = FieldDisplayName + " (" + FieldName + ")";

}

result.Columns.Add(FieldDisplayName, TypeFromField(Field));

fieldRefs.AppendFormat("", FieldName);

DisplayNames.Add(FieldDisplayName, FieldName);

}

result.Columns.Add("XmlElement", typeof(XmlElement));

XmlElement fields = ListInfoNode.OwnerDocument.CreateElement("ViewFields");

fields.InnerXml = fieldRefs.ToString();

XmlNode ItemsNode = lists.GetListItems(TechListName, null, null, fields, "10000", null, null);

// Lookup fields always start with the numeric ID, then ;# and then the string representation.

// We are normally only interested in the name, so we strip the ID.

System.Text.RegularExpressions.Regex CheckLookup = new System.Text.RegularExpressions.Regex("^\\d+;#");

foreach (XmlElement Item in ItemsNode.SelectNodes("rs:data/z:row", NameSpaceMgr))

{

data.DataRow newRow = result.NewRow();

foreach (data.DataColumn col in result.Columns)

{

if (Item.HasAttribute("ows_" + (string)DisplayNames[col.ColumnName]))

{

string val = Item.GetAttribute("ows_" + (string)DisplayNames[col.ColumnName]);

if (CheckLookup.IsMatch((string)val))

{

string valString = val as String;

val = valString.Substring(valString.IndexOf("#") + 1);

}

// Assigning a string to a field that expects numbers or

// datetime values will implicitly convert them

newRow[col] = val;

}

newRow["XmlElement"] = Item;

result.Rows.Add(newRow);

}

return result;

}

// The following Function is used to Get Namespaces

private static XmlNamespaceManager _nsmgr;

private static XmlNamespaceManager NameSpaceMgr

{

get

{

if (_nsmgr == null)

{

_nsmgr = new XmlNamespaceManager(new NameTable());

_nsmgr.AddNamespace("wss", "http://schemas.microsoft.com/sharepoint/soap/");

_nsmgr.AddNamespace("s", "uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882");

_nsmgr.AddNamespace("dt", "uuid:C2F41010-65B3-11d1-A29F-00AA00C14882");

_nsmgr.AddNamespace("rs", "urn:schemas-microsoft-com:rowset");

_nsmgr.AddNamespace("z", "#RowsetSchema");

}

return _nsmgr;

}

private Type TypeFromField(XmlElement field)

{

switch (field.GetAttribute("Type"))

{

case "DateTime":

return typeof(DateTime);

case "Integer":

return typeof(int);

case "Number":

return typeof(float);

default:

return typeof(string);

}

The Great Commandment

2008-09-25T18:42:00.002-04:00

While I was writing a post the other day, I noticed that I had neglected a topic that I find very important in software development. Risk management.

There are only a few guarantees in life. One of them is risk. Companies profit by seizing the opportunities that risks afford. Of course, they suffer loss by incidents of unmitigated risks. All our government and social systems are devices of risk management. In business, risk management is (now, and ever shall be) the great commandment.

Many software engineers forget that risk management is not just for PM’s. In fact, software and its development is fundamentally a tool of business, and, by extension, risk management. The practice of risk management in software really extends in to every expression in every line of source code.

Don’t believe me? Think of it this way… If it wasn’t a risk, it would be implemented as hardware. I've often heard hardware engineers say that anything that can be done in software can be done in hardware, and it will run faster. Usually, if a solution is some of the following…
· mature,
· ubiquitous,
· standard,
· well-known,
· fundamentally integral to its working environment

…it is probably low risk, particularly for change. It can likely be cost-effectively cast in stone (or silicone). (And there are plenty of examples of that… It’s what ASIC’s are all about.)

Software, on the other hand, is not usually so much of any of those things. Typically, it involves solutions which are…
· proprietary,
· highly customized,
· integration points,
· inconsistently deployed,
· relatively complex / error-prone
· immature or still evolving

These are all risk indicators for change. I don’t care what IT guys say… software is much easier to change than logic gates on silicone.

I’ve dug in to this in the past, and will dig in more on this in future posts, but when I refer to the “great commandment”, this is what I mean.

Application Platform Infrastructure Optimization

2008-09-25T18:41:00.000-04:00

In doing some research for a client on workflow in SharePoint, I came across this interesting article about the differences between BizTalk 2006 and the .NET Workflow Foundation (WF).

The article itself was worth the read for its main point, but I was also interested in Microsoft's Application Platform Infrastructure Optimization ("APIO") model.

The "dynamic" level of the APIO model describes the kind of system that I believe the .NET platform has been aiming at since 3.0.

I've been eyeing the tools... between MS's initiatives, my co-workers' project abstracts, and the types of work that's coming down the pike in consulting. From the timing of MS's releases, and the feature sets thereof, I should have known that the webinars they've released on the topic have been around for just over a year.

This also plays into Microsoft Oslo. I have suspected that Windows Workflow Foundation, or some derivative thereof, is at the heart of the modeling paradigm that Oslo is based on.

All this stuff feeds into a hypothesis I've mentioned before that I call "metaware", a metadata layer on top of software. I think it's a different shade of good old CASE... because, as we all know... "CASE is dead... Long live CASE!"

facebook

2008-09-15T21:50:00.003-04:00

I've been avoiding the whole My Space / Facebook thing for a while now... but now I'm checking it out. A little part of me is afraid that a public ill-prepared for the communications onslaught of web 2.0 toys like Facebook will fall prey to it. It may lead to implants that allow people to have every thought cataloged for later analysis. Before you know it, we'll all be Assimilated! (Resistance is Futile!)

Champions of Disruption

2008-09-14T13:57:00.006-04:00

I've been noticing lately that truely interesting things only happen on the "edge". Everything is energy, and everything happens at the point where energy flows are disrupted.

If you don't believe me, just ask Mother Nature. Take solar energy. Powerful energy flows from our sun and saturates our solar system... but all the amazing things happen where that energy flow is disrupted. The Earth disrupts it, and the result, in this case, is merely life as we know it.

It's so primal that we've abstracted the concept of energy flows, and call it (among other things) currency. When we sell a resource (a form of energy, in a sense), we even call that change "liquidation".

Sure, potential energy has value, but there are no edges in a region of potential energy. Potential energy is usually static, consistent, and only really exciting for what it could do or become, rather than what it currently is.

Likewise, it's where disruptions occur that there's business to be done.

According to this article on Information Week, CIO/CTO's appear to have generally become change-averse order takers. Surveys cited indicate that many shops are not actively engaged in strategy or business process innovation.

Perhaps they're still feeling whipped by the whole "IT / Business Alignment" malignment. Maybe they're afraid of having business process innovation through technology innovation come off as an attempt to drive the business. Ultimately, it seems many are going into survival mode, setting opportunity for change asside in favor of simply maintaining the business.

Maybe the real challenge for IT is to help business figure out that innovation is change, and change is where the action is.

In any case, it seems there's a lot of potential energy building up out there.

The disruptions must come. Will you be a witness, a victim, or a champion of them?

Retail IT in the Enterprise

2008-09-13T14:12:00.017-04:00

Lately, the projects I've been on have had me taking on roles outside my comfort zone. (I'm not talking about downtown-Boston... with the "Boston Express" out of Nashua, I'm ok with that.)

I've always been most comfortable, myself, in cross-discipline engineering roles, especially in smaller teams where everyone's got good cross-discipline experience. The communications overhead is low. The integration friction is low. Everyone knows how it needs to be done, and people are busy building rather than negotiating aggressively.

These types of tight, focused teams have always had business focused folks who took on the role of principal consultant. In this type of situation, the principal consultant provides an insulation boundary between the technical team and the customer.

This insulation has made me comfortable in that "zone": I'm a technologist. I eat, sleep, dream software development. I take the ability to communicate complex technical concepts with my peers effectively and concisely, very seriously.

So like I said, lately the projects I've been on have yanked me pretty hard out of that zone. I've been called on to communicate directly with my customers. I've been handling item-level projects, and it's a different world. There is no insulation. I'm filling all my technical roles, plus doing light BA and even PM duty.

Somewhat recently, I emailed a solution description to a CFO. The response: "Send this again in user-level English."

It killed me.

I've gotten so used to having others "protect" me from this sort of non-technical blunder. In contemporary projects, the insulating consulting roles are simply not present.

Makes me wonder about the most important lessons I learned during my school days... In high school days, maybe it was retail courtesy, and retail salesmanship in a technical atmosphere ("Radio Shack 101"). In college days, the key lessons might have been how to courteously negotiate customer experience levels, (from "help desk 101").

Compact and Full .NET Frameworks

2008-09-10T19:58:00.006-04:00

One of the things I've been intrigued by for a while now is the fact that code compiled for the .NET Compact Framework (all versions) executes very nicely on the full .NET Framework.

For example, my personal hobby project, "Jimmy Sudoku", is written in C# for the .NET Compact Framework 2.0. There are actually two install kits. The first is a .CAB file for Windows Mobile devices. The second is an .MSI for Windows 9x, XP, and Vista. The desktop install kit even serves two purposes. First, it installs the program on the desktop. Second, it leverages ActiveSync to push the .CAB up to the Windows Mobile device.

It's a .NET Compact Framework app especially for Windows Mobile devices, but many 'Jimmy' fans don't have a Windows Mobile device to run it on.

The coolest part is the ease in which all of the components inter-operate. The .EXE and .DLL's that are delivered to the mobile device are the very same as the ones that are delivered to the desktop. Like Silverlight to WPF, the Compact Framework is a compatible subset of the full framework, so interoperability is a given.

Even better, you can reference CF assemblies in Full framework assemblies. One immediate offshoot of this in my hobby project... the web service I built to service "Game of the Day" requests actually references the CF assembly that implements the game state model and game generator code. The assembly that generates games on Windows Mobile PDA's & cell phones is the very same assembly that generates games in the ASP.NET web service.

Admittedly, there are some bothersome differences between the CF and the Full .NET Framework. The CF does not support WPF. The CF has no facilities for printing. Also, while the CF does supports some of the common Windows Forms dialogs, it does not support File Save and File Open dialogs on Windows Mobile Standard Edition (Smart Phone / non-touchscreen) devices.

These differences can be overlooked to some extent, though, for the fact that one compiled assembly can execute on so many very different machine types. Further, with interoperability, one can extend a CF-based core code with full-framework support. For example, I'm currently playing with desktop print functionality for my hobby project.

Something that I'd really love to see, some day, is a good excuse to develop a Windows Forms app for a client that had shared components between the desktop and a mobile.

I can imagine that this model would be superb for a huge variety of applications, allowing a fully featured UI for the desktop version, and an excellent, 100% compatible, very low risk (almost "free") portable version.

I've often thought this would work great for apps that interface hardware, like:
field equipment,
mobile equipment,
vehicles of all sorts,

...simply plug in your PDA (via USB or Bluetooth), and it becomes a smart management device for the equipment, using the very same code that also runs on the desktop.

Semi-IT / Semi-Agile

2008-09-04T21:15:00.003-04:00

While working on-site for a client, I noticed something interesting. On the walls of some of my client's "users" offices, along with other more classic credentials, are certifications from Microsoft... SQL Server 2005 query language certifications.

I've heard a lot about the lines between IT and business blurring. We talk a fair amount about it back at HQ.

Interestingly, this case is a clear mid-tier layer between classic IT (app development, data management, advanced reporting) and business in the form of ad hoc SQL querying and cube analysis. In many ways, it's simply a "power-user" layer.

The most interesting part about it is the certification, itself. The credentials that used to qualify an IT role are now being used to qualify non-IT roles.

Another trend I'm seeing is development ceremony expectations varying depending on the risk of the project. Projects that are higher risk are expected to proceed more like a waterfall ceremony. Lower risk projects proceed with more neo-"agility".

The project I was on was apparently considered "medium" risk. The way I saw this play out was that all of the documentation of a classic waterfall methodology was expected, but the implementation was expected to develop along with the documentation.

In many ways, it was prototyping into production. Interestingly, this project required this approach: the business users simply did not have time to approach it in a full waterfall fashion. Had we been forced into a full-fledged classic waterfall methodology, we might still be waiting to begin implementation, rather than finishing UAT.

Power and Control: Fusion Report 13 June 008

2008-09-04T21:08:00.001-04:00

Power and Control: Fusion Report 13 June 008

Thanks for the report, M. Simon. My post about Energy Productization was actually a bit of a response to an article I read that covered both the WB7 and ITER. Your report hits the points much more concisely.

Energy Productization

2008-09-02T21:29:00.005-04:00

I think the ITER project, a grand-scale fusion project, is interesting. I'm troubled with it for a few reasons, though. I can't help but think that there's only a few reasons that we need it, and most of them have to do with power... of a controlling nature.

There's already a great big huge fusion reactor throwing more energy at us than we can collect, let alone use... every single day... the sun.

Efforts to create fusion reactors here on earth are great for these reasons:
1) Energy for space exploration
2) Productization of energy
3) Weapons innovation

Once we learn how fusion's done, we can build space craft from it that could potentially get us somewhere in the galaxy. That's all well and good, but will it happen before we poison our existing biosphere?

Once we have fusion reactors, energy moguls can sell it. Oh, great! Instead of energy productization through oil, we get energy productization through fusion... because we can't all have one of these great big huge reactors in our basements. At least it's renewable. If the moguls are benevolent, it might even be cheap.

Finally, once we have fusion reactors like this, we'll learn new ways to blow ourselves out of our collective misery... so I suppose the first two points are mute once this comes along.

Economic Detox

2008-09-01T10:06:00.013-04:00

While contemporary headlines bode poorly for the U.S. economy, I see them as signs of hope...

I keep hearing high-pitched alarms about the weakening U.S. dollar, inflation, energy prices, the housing market bubble burst. We all see the ugly face of the these conditions.

Global trade has been a bitter (but necessary) pill for the U.S. Perhaps the Clinton-detonated U.S. economic nuclear winter (of global trade, NAFTA, etc.) is finally starting to give way to a new economic springtime in the States.

In the late 90's US market, there were a lot of excesses in the technology sector. Then the bubble burst. When the dust settled, we (the US IT industry) found ourselves disenfranchised by our sponsors... corporate America beat us with our own job hopping. U.S. Engineers hopped off to the coolest new startup, and rode their high salaries into the dirt, while enduring companies went lean, mean, and foreign. We had become so expensive, we were sucking our own project ROI's completely out of sight. By hooking foreign talent pools, the ROI's were visible again.

Nearly a decade later, look what's happening around the world... Many foreign IT job markets are falling into the same salary inflation trap that the U.S. market fell into... They are going through the same inflation we experienced. Their prices are rising.

Combine their salary inflation with our salary stagnation and a weakening dollar, and what do you get?

A leaner, meaner domestic competitor.

In a sense, it's like that in many sectors of the U.S. economy.

So let the U.S. dollar weaken... It means that America can go back to being product producers (rather than mindless consumers) in the global market!

If It Looks Like Crap...

2008-08-30T08:02:00.008-04:00

It never ceases to amaze me what a difference "presentation" makes.

Pizza Hut is airing a commercial around here about their "Tuscani" menu. In the commercial, they show people doing the old "Surprise! Your coffee is Folgers Crystals!" trick in a fancy restaurant, except they're serving Pizza Hut food in an "Olive Garden"-style venue.

It clearly shows my point, and that the point applies to anything... books, food, appliances, vehicles, and software, just to name the first few things that pop to mind. You can have the greatest product in the world... it exceeds expectations in every functional way... but any adjective that is instantly applied to the visual presentation (including the environment it's presented in) will be applied to the content.

If it looks like crap, that's what people will think of it.

(Of course, there are two sides to the coin... What really kills me are the times when a really polished application really IS crap... it's UI is very appealing, but not thought out. It crashes at every click. But it looks BEAUTIFUL. And so people love it, at least enough to be sucked into buying it.)

Good engineers don't go for the adage "It's better to look good than to be good." We know far better than that. You can't judge the power of a car by its steering wheel. Granite countertops look great, but they're typically hard to keep sanitary.

When it comes to application user interfaces, engineers tend to make it function great... it gives you the ability to control every nuance of the solution without allowing invalid input... but if it looks kludgy, cheap, complex, or gives hard-to-resolve error messages, you get those adjectives applied to the whole system.

So what I'm talking about, really, is a risk... and it's a significant risk to any project. For that reason, appearance litterally becomes a business risk.

For any non-trivial application, a significant risk is end-user rejection. The application can do exactly what it's designed to do, but if it is not presented well in the UI, the user will typically tend to reject the application sumarily.

That's one thing that I was always happy about with the ISIS project. (I've blogged about our use of XAML and WPF tools in it, before.) The project was solid, AND it presented well. Part of it was that the users loved the interface. Using Windows Presentation Foundation, it was easy to add just enough chrome to impress the customers without adding undo complexity.

I blew it...

2008-08-25T17:17:00.003-04:00

While working on the site, I accidentally deleted a core portion of the Jimmy Sudoku 2.5 puzzle generator web service. (Don't ask me how... it was apparently so bone-headed that it took me a while to realize I'd done it.)

I've exhausted all my backup options... the backups were either too new (and therefore producing the wrong content format), or so old that the backup itself was corrupt.

The good news is 2.5 supported local game generation if the service was on the fritz. I guess that means its covered.

Anyway, if you send me proof of purchase of any rev prior to 3.0, I'll send the fresh bits along.

Send it to me in email to jimmysoftware (at) kataire.com, and I'll reply with a copy. (Your order number & date will probably suffice...)

Compromise & Capitulation

2008-08-17T19:21:00.002-04:00

There's three different flavors of Windows Mobile in the 6.x line. Standard, Classic, and Professional.

Standard = Smart Phone, no touchscreen
Classic = PDA w/touchscreen
Professional = PDA / Phone with Touchscreen

One of the other interesting little gotchas is that the .Net Compact Framework 2.0 compiles the same for all three editions. Unfortunately, once in a while, you get a "NotSupportedException" out of the Standard edition.

A few days ago, in order to get my sudoku program published, I decided to simply avoid a problem I had with the Standard edition's lack of a SaveFileDialog and OpenFileDialog. My avoidance manifested in a "not supported" message of my own, if the user tried to save / load a file in that environment.

Today, I capitulated... I implemented an alternative file save/load functionality which kicks in automatically when the program gets a "NotSupportedException" on the common dialogs.

It's in 3.0.3, which I've re-published on PocketGear.

Jimmy SuDoku 3.0 Released

2008-08-15T23:14:00.006-04:00

Those of you who have worked with me on a project in the past few years probably know of my hobby project. It's an implementation of SuDoku. It's made for Windows Mobile devices (cell phones, etc.), but it also runs on Windows XP (et al).

The old version, 2.5, had been published on PocketGear. This last update was published in January, 2007, just before I started with Edgewater.

I've been hacking at it here & there since then, but the project suffered from lots of maladies... most significantly lack of time.

So after more than a year and a half, I'm happy to finally announce Jimmy SuDoku 3.0!

3.0 has a whole new game state model, based on CLR classes rather than an XML DOM. This means the puzzle generator's fast enough on hand-held devices that it doesn't need a web service to do the work for it. Another side-effect of this change is a smaller run-time memory footprint, though I'm not sure by exactly how much.

I also figured out how to leverage the hardware controls on WM6.0 & 6.1 devices so that non-touchscreen devices can play, too.

SSIS Bug / Workaround

2008-07-21T09:07:00.003-04:00

I ran across a "feature" (bug) in SSIS today that I figured I'd record in case I ran across it again...

Error: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80040E14. An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".". An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "The OLE DB provider "BULK" for linked server "(null)" reported an error. The provider did not give any information about the error.". An OLE DB record is available. Source: "Microsoft SQL Native Client" Hresult: 0x80040E14 Description: "Reading from DTS buffer timed out.".

According to this MSDN forum thread, it's a bona-fide SSIS bug (I'm running SQL Server SP2 with the latest known patches as of this date.)

So the problem is in the SQL Server Destination. Changing it out for an OLEDB Destination seems to do the trick.

WALL•E and Enterprise Data Landfills

2008-07-15T23:05:00.001-04:00

"Life is nothing but imperfection and the computer likes perfection, so we spent probably 90% of our time putting in all of the imperfections, whether it's in the design of something or just the unconscious stuff. "
-Andrew Stanton, director of Disney/Pixar's WALL-E, in an interview on the topic of graphic detailing.

I'm enough of a sci-fi geek that I had to take my kids to see WALL*E the day it opened. I found it so entertaining that, while on vacation, I started browsing around the internet... digging for addititonal tidbits about the backstory.

I came across the quote, above, initially on Wikipedia's Wall-E page.

The simple truth carries across all applications of contemporary computer technology. Technology tools are designed for the "general" cases, and yet, more and more often, we're running into the imperfect, inconsistent, outlying, and exceptional cases.

To follow the thought along, perhaps 90% of what we do as software developers is about trying to get a grip on the complexities of... everything we get to touch on. I guess the remaining 10% would be akin to the root classes... the "Object" class, and the first few subordinates, for example.

Andrew Stanton's quote reminds me of the 90-10 rule of software engineering... 90% of the code is implemented in 10% of the time. (conversely, the remaining 10% of the code is implemented in the remaining 90% of time). I tend to think of this one as a myth, but it's fun thought.

It's dealing with the rough fringes of our data that's among the industry's current challenges, but it's not just corporate data landfills.

I recently heard a report that suggested that technology will get to the point that commercially available vehicles with an auto-pilot will be available within the next 20 or so years. What's really the difference, to a computer, between financial data, and, say, navigational sensor data?

So to flip that idea on its head, again, and you could have more intelligent artificial agents spelunking through data warehouses... WALL-DW ? (Data Warehouse edition)

Then again, I wonder if the 80-20% rule isn't what gets us into our binds to begin with.

Real Software Heroes

2008-07-14T16:47:00.001-04:00

While scanning the channels looking for an interesting show to watch, I came across a show on the Science channel... "Moon Machines". I couldn't have been luckier than to see the chapter "Navigation".

I'd heard bits about the technology that went into the Apollo missions, and how there were some of the first "modern" IC-based computers on board, but I never really thought about the implications of what they were doing. Having these computers aboard meant they had software. There wasn't exactly COTS systems for navigating to the moon.

The episode focused quite a bit on the experience of the software development team, including some at the personal level. There were quotes like "Honey, I'm going to be in charge of developing something called 'software'."... (and the response: "Please don't tell the neighbors.")

I've felt pressure on projects before... stuff I was working on that represented millions of dollars in its success, and presumably millions lost in its failure. I've even worked on software projects where runtime production logic errors could send people to jail. I've never written software that human life directly depended on.

My hat is off to the folks who took up this monumental challenge for the first time in software history, and made it work. To me, that's every championship sports victory... ever... combined.

All I can say is... wow.

They knew what a monumental victory it was, too... 40+/- years later, and the engineers they interviewed were still moved by the awe of their own accomplishment, and the personal sacrifices they made to pull it off.

As well they should be. Fantastic!!

SSIS: Unit Testing

2008-06-26T06:20:00.007-04:00

I've spent the past couple days putting together unit tests for SSIS packages. It's not as easy to do as it is to write unit & integration tests for, say, typical C# projects.

SSIS Data flows can be really complex. Worse, you really can't execute portions of a single data flow separately and get meaninful results.

Further, one of the key features of SSIS is the fact that the built-in data flow toolbox items can be equated to framework functionality. There's not so much value in unit testing the framework.

Excuses come easy, but really, unit testing in SSIS is not impossible...

So meaningful unit testing of SSIS packages really comes down to testing of Executables in a control flow, and particularly executables with a high degree of programability. The two most significant control flow executable types are Script Task executables and Data Flow executables.

Ultimately, the solution to SSIS unit testing becomes package execution automation.

There are a certain number of things you have to do before you can start writing C# to test your scripts and data flows, though. I'll go through my experience with it, so far.

In order to automate SSIS package execution for unit testing, you must have Visual Studio 2005 (or greater) with the language of your choice installed (I chose C#).

Interestingly, while you can develop and debug SSIS in the Business Intelligence Development System (BIDS, a subset of Visual Studio), you cannot execute SSIS packages from C# without SQL Server 2005 Developer or Enterprise edition installed ("go Microsoft!").

Another important caveat... you CAN have your unit test project in the same solution as your SSIS project. Due to over-excessive design time validation of SSIS packages, you can't effectively execute the SSIS packages from your unit test code if you have the SSIS project loaded at the same time. I've found that the only way I can safely run my unit tests is to "Unload Project" on the SSIS project before attempting to execute the unit test host app. Even then, Visual Studio occassionally holds locks on files that force me to close and re-open Visual Studio in order to release them.

Anyway, I chose to use a console application as the host app. There's some info out there on the 'net about how to configure a .config file borrowing from dtexec.exe.config, the SSIS command line utility, but I didn't see anything special in there that I had to include.

The only reference you need to add to your project is a ref to Microsoft.SqlServer.ManagedDTS. The core namespace you'll need is

using Microsoft.SqlServer.Dts.Runtime;

In my first case, most of my unit testing is variations on a single input file. The package validates the input and produces three outputs: a table that contains source records which have passed validation, a flat output file that contains source records that failed validation, and a target table that contains transformed results.

What I ended up doing was creating a very small framework that allowed me to declare a test and some metadata about it. The metadata associates a group of resources that include a test input, and the three baseline outputs by a common URN. Once I have my input and baselines established, I can circumvent downloading the "real" source file, inject my test source into the process, and compare the results with my baselines.

Here's an example Unit test of a Validation executable within my SSIS package:

[TestInfo(Name = "Unit: Validate Source, duplicated line in source", TestURN = "Dupes")]
public void ValidationUnitDupeLineTest()
{
    using (Package thePackage = _dtsApp.LoadPackage(packageFilePath, this))
    {
        thePackage.DelayValidation = true;
        DisableAllExecutables(thePackage);
        EnableValidationExecutable(thePackage);
        InjectBaselineSource(GetBaselineResource("Stage_1_Source_" + TestURN), thePackage.Variables["SourceFilePath"]);
        thePackage.Execute(null, null, this, null, null);
        string errorFilePath = thePackage.Variables["ErrorLogFilePath"].Value as string;
        //throw new AbortTestingException();
        AssertPackageExecutionResult(thePackage, DTSExecResult.Failure);
        AssertBaselineAdjustSource(TestURN);
        AssertBaselineFile(GetBaselineResourceString("Baseline_Stage1_" + TestURN), errorFilePath);
    }
}

Here's the code that does some of the SSIS Package manipulation referenced above:


#region Utilities
protected virtual void DisableAllExecutables(Package thePackage)
{
    Sequence aContainer = thePackage.Executables["Adjustments, Stage 1"] as Sequence;
    (aContainer.Executables["Download Source From SharePoint"] as TaskHost).Disable = true;
    (aContainer.Executables["Prep Target Tables"] as TaskHost).Disable = true;
    (aContainer.Executables["Validate Source Data"] as TaskHost).Disable = true;
    (aContainer.Executables["Process Source Data"] as TaskHost).Disable = true;
    (aContainer.Executables["Source Validation Failure Sequence"] as Sequence).Disable = true;
    (aContainer.Executables["Execute Report Subscription"] as TaskHost).Disable = true;
    (thePackage.Executables["Package Success Sequence"] as Sequence).Disable = true;
    (thePackage.Executables["Package Failure Sequence"] as Sequence).Disable = true;
}


protected virtual void DisableDownloadExecutable(Package thePackage)
{
    Sequence aContainer = thePackage.Executables["Adjustments, Stage 1"] as Sequence;
    TaskHost dLScriptTask = aContainer.Executables["Download Source From SharePoint"] as TaskHost;
    dLScriptTask.Disable = true;
}


protected virtual void EnableValidationExecutable(Package thePackage)
{
     Sequence aContainer = thePackage.Executables["Adjustments, Stage 1"] as Sequence;
    TaskHost validationFlow = aContainer.Executables["Validate Source Data"] as TaskHost;
    validationFlow.Disable = false;
}

protected virtual void EnableValidationExecutable(Package thePackage)
{
    Sequence aContainer = thePackage.Executables["Adjustments, Stage 1"] as Sequence;
    TaskHost validationFlow = aContainer.Executables["Validate Source Data"] as TaskHost;
    validationFlow.Disable = false;
}

Another really handy thing to be aware of...

IDTSEvents

I highly recommend you implement this interface and pass it into your packages. Of course, in each event handler in the interface, implement code to send reasonable information to an output stream. Notice the call to thePackage.Execute, way up in the first code snippet... the class that contains that method implements that interface, so I can manipulate (when necessary) how to handle certain events.

Interestingly, I haven't needed to do anything fancy with that so far, but I can imagine that functionality being very important in future unit tests that I write.

Here's a visual on all the resources... the image shows SSMS over VS, with both database tables and project resources with common URNs to relate them.

I won't get into the details of the framework functionality, but I found it useful to be able to do things like set a flag to rebuild baseline resources from current outputs, and such.

I modeled some of my framework (very loosely) functionality on the Visual Studio Team System Edition for Testers, which we used on the TWM ISIS project.

Another interesting lesson learned: I can see that the folks who built SSIS were not avid unit testers themselves. SSIS Executables have a "Validate()" method. I encountered lots of problems when I tried to use it. Hangs, intermittent errors, all that stuff that testing should have ironed out.

Null Schema

2008-06-18T16:57:00.002-04:00

I've been contemplating the whole "unstructured" thing for a while now, and I've developed some new hypotheses about it. The discussion's been around the fact that Web 2.0 / Enterprise 2.0 generates a lot of "unstructured" data.

I'm not sure "unstructured" is really the most technically fitting word, though. It's the word that works if you're a technical person talking to a non-technical person.

I think the information we're seeing in these settings is typically better structured than what we've seen in the past. The structures are being defined by the provider, however, sometimes on an ad-hoc basis, and can change without notice.

If you're in the geek domain, I think "undefined" fits better. Maybe "unknowable structure". It's Null Schema.

I think we've all seen tons of this... it's a trend towards increasing structure with less defined schema. It seems to fit with the "agile" trend.

So the other aspect of this Web 2.0 thing is that the data doesn't have to just be of an unknowable format. It can also be communicated through a number of communications channels, at the provider's discretion. People define conventions to ease this. Interestingly, the convened-upon channels end up providing context for the content. In turn, it adds to its structure... more null schema.

It flies in the face of our tightly defined, versioned SOA end-point contracts. XSOA? :)

It's been said that SOA lives in a different problem space, but that may only be a matter of convention, moving forward.

My Windows Mobile PCS Phone

2008-06-06T17:00:00.001-04:00

I just gotta say... this thing's fantastic.

It's my own personal mobile phone (not company issued), but I really do get some hefty mileage out of it.

It's a Sprint Mogul 6800. Verizon has a similar unit (vx6800, I think). It's one of many Windows Mobile 6 devices out there... touch screen, with a stylus and a slide out keyboard. It was already a little dated when I bought it, but that just meant the price wasn't so high.

I'm just psyched because I bought this one back in September 2007, and I keep finding new uses for it.

My new assignment is in Boston. One of the rough edges of the assignment, for me, is the long commute... (1 hour minimum (if traffic's really good)). So after doing the math, I discovered that I can pick up the Boston Express bus ride from a neighboring town.

I'm on the return trip at this moment, using my phone for internet access, and it's really solid, EV-DO.A service. Connectivity is a USB port away.

While I'm doing that, I'm ALSO listening to 2Gb worth of music that I downloaded via subscription service from Napster using Windows Media Player Mobile.

The built in GPS was not software-enabled when I bought the phone, but a software upgrade released in January solved that. The GPS works great with the Google Maps for Pocket PC app I downloaded for free. This morning, it got me from South Station to NHP just like it got me from NYC's Port Authority bus terminal to the AT&T store on Madison Ave a few weeks ago to see MS Surface.

So I also toy with the .NET Compact Framework, and used it some time ago to write my own Windows Mobile implementation of sudoku. (I've been seriously considering refactoring to mess with LINQ in the .NET CF 3.5)

The built in camera's nice... I use it all the time when I've forgotten my "real" camera.

Wi-Fi is occasionally very handy... it's great when I need network access while talking on the phone.

So is bluetooth... I appreciate being able to talk on the bluetooth headset while taking notes or looking stuff up online... or just appreciate talking hands-free.

I sync the thing with Outlook, and rely on it too heavily for meeting reminders. That comes in handy at NHP, too, where network access is... limited. (I can even get RSS updates from the Tech Mill.) It's also integrated with Active Directory, so when I need a teammate's contact info, I can find it online as long as I have connectivity (pretty much always).

I've used the voice recorder a few times. It's been handy when I've needed it.

There's a whole list of "separate" items I don't carry because I have this thing... I don't wear a watch. I don't have an iPod. I don't carry a GPS handheld, or keep one in my car. I don't often carry around my camera. I seldom use paper to take notes, so I don't often need a pen. This one thing covers all that stuff, and more.

Did I mention that it's a phone? :)

Enterprise Reprogramming

2008-05-20T16:40:00.003-04:00

I found an interview of a Satyam Senior VP listed on LinkedIn relatively interesting (link below).

This senior VP talks about how Satyam and the IT industry is responding to new challenges.

One thing that stands out to me is the statement that they are moving from services to solutions. They make the implication that they are rebuilding or reprogramming businesses at the workflow / process level. They appear to be successfully applying technology build-out as a commodity service while implementing their solutions... Sounds like they're treating the enterprise as a sort of programmable platform, like SOA / BPM on a grand scale.

From the article:
"A solutions provider transforms business. The difference in business will happen when we change those business processes as well. That is where we are bringing in business transformation solutions — process optimisation, process reengineering, etc. "

My intuition doesn't quite square with Satyam's vision.

Lots of things have been pointing towards more innovation in the top layers of applications, built on a very stable technology base. To me, it still feels like there's an unspoken motivation for that: business leadership wants IT folks to make ruggedized app dev tools and hand them over to power users (and/or process owners). Business leaders want IT to get the C# off their business processes.

All of that is sorta where I started cooking up the hypothesis of metaware from.

I'm curious to know how Satyam's vision is really working. I guess we'll know in a few years.

‘Moving towards IP-led revenues’

Functional Expression

2008-05-18T07:52:00.015-04:00

So one more thing crossed my mind about implementing code with respect to art & science, and I had to express it...

I looked up the term "Art" in the dictionary. The first definition is:

the quality, production, expression, or realm, according to aesthetic principles, of what is beautiful, appealing, or of more than ordinary significance.

For me, regarding coding, it's a matter of remembering a few points:

implementation is expression
significance is subjective
beauty is in the eye of the beholder

So code can be expressed, fundamentally, in a bunch of ways:

Electronically,
Numerically,
mnemonically,
symbolically,
graphically,
gesturally,
audibly,
visually,
etc... ?

Simple, clever, elegant, seemingly natural expressions of all kinds are typically beautiful to a programmer, when they function correctly.

Of course, to me, the most beautiful implementations are implementations that elegantly express its business in a way that's very clear to anyone familiar with the problem domain at that abstraction level, and to the target platform(s).

Artless Programming

2008-05-17T10:00:00.010-04:00

So maybe I am strange... I actually have printed snips of source code and UML diagrams and hung them on my office wall because I found them inspirational.

Reminds me of a quote from The Matrix movies...
Cypher [to Neo]: "I don't even see the code. All I see is blonde, brunette, red-head." :)

It's not quite like that, but you get the point. There's gotta be a back-story behind the witty writing. I suspect it has something to do with a programmer appreciating particularly elegant solutions.

One of the hard parts about knowing that programming is an artful craft is being forced to write artless code. It happens all the time. Risks get in the way... a risk of going over budget, blowing the schedule, adding complexity, breaking something else.

It all builds up. The reality is, as much as we software implementers really want application development to be an art, our business sponsors really want it to be a defined process.

The good news for programmers is that every application is a custom application.

It really sucks when you're surgically injecting a single new business rule into an existing, ancient system.

This is the case with one of my current clients. At every corner, there's a constraint limiting me. One false move, and whole subsystems could fail... I have such limited visibility into those subsystems, I won't know until after I deploy to their QA systems and let them discover it. If I ask for more visibility, we risk scope creep. The risks pile up, force my hand, and I end up pushed into a very tightly confined implementation. The end result is awkward, at best. It's arguably even more unmaintainable.

These are the types of projects that remind me to appreciate those snips of inspirational code.

Don't get me wrong. I'm happy there's a fitting solution within scope at all. I'm very happy that the client's happy... the project's under budget and ahead of schedule.

The "fun" in this case, has been facing the Class 5 rapids, and finding that one navigable path to a solution.

See also:
politechnosis: Art & Science

Art & Science

2008-05-10T12:19:00.021-04:00

Fire & Ice... Day & Night...

This question, Art vs. Science, has come up a million times in software development circles. Reading Paul Johnson's (Paul's Pontifications) blog post, in conjunction with a discussion in the Tech Mill at Edgewater, (thanks, Jason!) I have come to see that art and science are not as opposite as I once viewed them to be.

What hit me was that Paul makes the statement that there's no process to implementing software. I still disagree. There are many processes.

The number of processes that an implementer can choose from to write his/her code is often vast, and depends on the problem set. A problem set includes many things, including requirements, tools, target platform, development platform, existing code, and even the implementer's mood and frame of mind. That is what makes implementing code, like painting, or creating a recipe, an art.

Within a common implementation problem set, there can be a large number of processes which can be applied to derive valid solutions. In fact, there are so many, that some distinct processes may actually render the very same code. So, to be more clear, it's not that there's no process... it's that there's no single valid process.

Knowing that there's no one single valid process doesn't mean that we can't pick a needle from the haystack... if the process produces a solution within the problem set, it's good.

Now consider what happens when you start to narrow a problem set. There's lots of things you can do. Frameworks, platforms, clear-specific requirements, best practices, coding standards, well structured architectures... these things are all factors that limit the problem set. By narrowing a problem set, you narrow the number of valid processes. By narrowing the number of valid processes that a developer can choose from, lots of interesting things start to happen. You achieve more predictable results, and are more likely to achieve repeatable schedules... and you reduce overall project risk.

This is what's so interesting about contemporary trends in software development, such as Ruby on Rails... use of these tools narrows problem sets that developers face. This means the implementer can spend less time figuring out where the blanks are, and more time filling them.

Now let's take this further. What happens when you reduce the problem set dramatically...? Take a single, relatively well known problem, on a very specific platform, using a very small set of unambiguous expressions. You get a very tightly defined process. By doing this, you wring the art out of creating something, to the point where it becomes machinable. The process becomes realized as a factory.

So to answer the question... Art or Science?

It's a trick question... art and science are not exclusive opposites. Art is about freedom to choose your creative process. Science is about knowing what processes are available, and the pros and cons of each. So programming, like all creative activities, is usually art (except in single-processed cases), and usually science (except in cases of serendipity and true miracles).

Paul's Pontifications: An Under-Appreciated Fact: We Don't Know How We Program

Hands on Surface

2008-05-08T11:57:00.006-04:00

There's a developing UI paradigm growing, and some of it has been released in the wild.

Multi-point touch screen systems are starting to take shape out of the ether, and it really feels like it's going to usher in a new era of computing. We've talked about a few of them here in the Tech Mill. It's "Minority Report" without the goofy VR glove.

Microsoft's offering in this arena is Surface (formerly "Milan").( http://www.microsoft.com/surface )

From available marketing materials, Surface is much like the other offerings that are under development, with a few interesting differences. Rather than being an interactive "wall", it's a "table". In addition to interacting to a broad range of touch-based gestures, Surface also interacts with objects. Some of it's marketed use-cases involve direct interaction with smartphone, media, and storage devices.

This week, I'm on a training assignment in New Jersey, but within a bus ride to one of very few instances of Surface "in the wild".

I made it a secondary objective to hit one of the AT&T stores in mid-town Manhattan.

I had a lot of high expectations for it, so actually getting to play a bit with it struck me as a touch anti-climactic. The UI was great, but it was clear they cut costs on hardware a bit: responsiveness wasn't quite as smooth as the web demos. It did impress me with the physics modeling of the touch gestures... dragging "cards" around the table with one finger mimicked the behavior of a physical card, pivoting around the un-centered touch point as a real one would.

I was also a bit concerned that the security devices attached to the cell phones they had around the table were some sort of transponder to hide "vapor-ware" special effects. My own phone (an HTC Mogul by Sprint) was ignored when I placed it on the table.

All in all, I was happy to finally get to play with it. Between technology advances and price drops, this UI paradigm will start to make it into the power business user's desk.

I mean, can you imagine, for example, cube analysis.... data mining... report drilling... and then with a few gestures, you transform the results into charts and graphs... then throw those into a folder on your mobile storage / pda device...

I'm still loving the idea of interactivity between physical and virtual (and/or remote) logical constructs...

Imagine bringing up the file server and your laptop on a "Surface" UI, and litterally loading it with data and installing software with the wave of your hand....

or...

Having a portable "PDA" device with "big" storage... in fact, big enough to contain a virtual PC image... In stand-alone mode, the PDA runs the VPC in a "smart display" UI. When you set it on a Surface, the whole VPC sinks into it. You get access to all the Surface functional resources including horsepower, connectivity, additional storage, and the multi-touch UI while the PDA is in contact. When you're done, the VPC transfers back to the PDA, and you can take it to the next Surface UI in your room at the hotel, or the board room (which has one giant "Surface" as the board room table.)

The preview is over at AT&T today. According to Wikipedia, Microsoft expects they can get these down to consumer price ranges by 2010 (two years!).

Multiprocessing: How 'bout that Free Lunch?

2008-05-04T23:40:00.002-04:00

I remember reading an article, a few years back...

The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software

Its tagline: "The biggest sea change in software development since the OO revolution is knocking at the door, and its name is Concurrency."

Mr. Sutter's article suggests that because CPUs are now forced to improve performance through multi-core architectures, applications will need to typically employ multi-threading to gain performance improvements on newer hardware. He made a great argument. I remember getting excited enough to bring up the idea to my team at the time.

There are a number of reasons why the tag line and most of its supporting arguments appeared to fail, and in retrospect, could have been predicted.

So in today's age of multi-core processing, where application performance gains necessarily come from improved hardware throughput, why does it still feel like we're getting a free lunch?

To some extent, Herb was right. I mean, really, a lot of applications, by themselves, are not getting as much out of their host hardware as they could.

Before and since this article, I've written multi-threaded application code for several purposes. Each time, the threading was in UI code. The most common reason for it: to monitor extra-process activities without blocking the UI message pump. Yes, that's right... In my experience, the most common reason for multi-threading is, essentially, to allow the UI message pump to keep pumping while waiting for… something else.

But many applications really have experienced significant performance improvements in multi-processor / multi-core systems, and no additional application code was written, changed, or even re-compiled to make that happen.

How?

Reduced inter-process contention for processor time
Client-server architectures (even when co-hosted, due to the above)
Multi-threaded software frameworks
Improved supporting hardware frameworks

Today's computers are typically doing more, all the time. The OS itself has a lot of overhead, especially Windows-based systems. New Vista systems rely heavily on multi-processing to get performance for the glitzy new GUI features.

The key is multi-processing, though, rather than multi-threading. Given that CPU time is a resource that must be shared, having more CPUs means less scheduling collision, less single-CPU context switching.

Many architectures are already inherent multi-processors. A client-server or n-tier system is generally already running on a minimum of two separate processes. In a typical web architecture, with an enterprise-grade DBMS, not only do you have built-in “free” multi-processing, but you also have at least some built-in, “free” multi-threading.

Something else that developers don’t seem to have noticed much is that some frameworks are inherently multi-threaded. For example the Microsoft Windows Presentation Foundation, a general GUI framework, does a lot of its rendering on separate threads. By simply building a GUI in WPF, your client application can start to take advantage of the additional CPUs, and the program author might not even be aware of it. Learning a framework like WPF isn’t exactly free, but typically, you’re not using that framework for the multi-threading features. Multi-threading, in that case, is a nice “cheap” benefit.

When it comes down to it, though, the biggest bottlenecks in hardware are not the processor, anyway. The front-side bus is the front-line to the CPU, and it typically can’t keep a single CPU’s working set fresh. Give it a team of CPUs to feed, and things get pretty hopeless pretty quick. (HyperTransport and QuickPath will change this, but only to the extent of pushing the bottle necks a little further away from the processors.)

So to re-cap, to date, the reason we haven’t seen a sea change in application software development is because we’re already leveraging multiple processors in many ways other than multi-threading. Further, multi-threading options have been largely abstracted away from application developers via mechanisms like application hosting, database management, and frameworks.

With things like HyperTransport (AMD’s baby) and QuickPath (Intel’s), will application developers really have to start worrying about intra-process concurrency?

I throw this one back to the Great Commandment… risk management. The best way to manage the risk of intra-process concurrency (threading) is to simply avoid it as much as possible. Continuing to use the above mentioned techniques, we let the 800-lb gorillas do the heavy lifting. We avoid struggling with race conditions and deadlocks.

When concurrent processing must be done, interestingly, the best way to branch off a thread is to treat it as if it were a separate process. Even the .NET Framework 2.0 has some nice threading mechanisms that make this easy. If there are low communications needs, consider actually forking a new process, rather than multi-threading.

In conclusion, the lunch may not always be free, but a good engineer should look for it, anyway. Concurrency is, and will always be an issue, but multi-core processors were not the event that sparked that evolution.

Converging Metadata on XAML

2008-04-10T15:01:00.003-04:00

In a recent project, we used Excel spreadsheets to capture the data dictionaries for our functional requirements. The project actually had 22 data dictionary sections, each with about a hundred data points described in them. Each data point had on the order of twenty properties that the client needed to know about. It included not only how to present the data entry field, but how to bind it at runtime to business objects, how to transform the user input before presenting the data to various back-end systems, validation, and a few other rules.

Excel spreadsheets can be saved off as XML, a fact which we leveraged. Our analysts filled in the Excel spreadsheets, identifying all the data points and the typical properties of each. After that, we engineers filled in bindings and a few other more technical details. Later, we transformed our Excel-document data dictionaries into a custom XML format (a Domain Specific Language or DSL) using XSLT. Finally, we compiled those extracts into our client application as a resource (one for each dictionary) containing all the client metadata for that dictionary.

Divorcing ourselves from the broken notion that XAML is WPF, and with just a little more time (on the order of days), we could have codified our DSL into a set of .NET classes. By doing this, we could have converged our data dictionary metadata on XAML. In effect, this would have turned metadata directly into runtime business objects. These business objects would have served as the content for our WPF-based LOB data entry forms. In turn, this would have greatly simplified a number of complexities, especially around the problem of indirect data binding. The end result would have been a more maintainable code base, and even faster running code.

Metaware

2008-04-06T17:42:00.015-04:00

There's been a fair amount of buzz in the IT world about IT-Business alignment lately. The complaints seem to be that IT seems to produce solutions that are simply too expensive. Solutions seem to range from "Agile" methodologies to dissolving the contemporary IT group into the rest of the enterprise.

I think there's another piece that the industry is failing to fully explore.

I think what I've observed is that the most expensive part of application development is actually the communications overhead. It seems to me that the number one reason for bad apps, delays, and outright project failures, is firmly grounded in communications issues. Getting it "right" is always expensive. (Getting it wrong is dramatically worse.) In the current IT industry, getting it right typically means teaching analysts, technical writers, developers, QA, and help desk significant aspects of the problem domain, along with all the underlying technologies they need to know.

In the early days of "application development", software based applications were most often developed by savvy business users with tools like Lotus 1-2-3. The really savvy types dug in on dBase. We all know why this didn't work, and the ultimate response was client-server technology. Unfortunately, the client-server application development methodologies also entrenched this broad knowledge sharing requirement.

So how do you smooth out this wrinkle? I believe Business Analytics, SOA/BPM, Semantic web, portals/portlets... they're providing hints.

There have been a few times in my career where I was asked to provide rather nebulous functionality to customers. Specificially, I can think of two early client-server projects where the users wanted to be able to query a database in uncertain terms of their problem domain. In both of these cases, I built application UI's that allowed the user to express query details in easy, domain-specific terms. User expressions were translated dynamically by software into SQL. All of the technical jargon was hidden away from the user. I was even able to allow users to save favorite queries, and share them with co-workers. They enabled the users to look at all their information in ways that no one, not even I, had considered before hand. The apps worked without giving up the advances of client-server technology, and without forcing the user into technical learning curves. These projects were both delivered on time and budget. As importantly, they were considered great successes.

In more recent times, certain trends that have caught my attention: the popularity of BI (especially cube analysis), and portal/portlets. Of all the other tools/technologies out there, these tools are actively demanded by business end-users. At the same time, classic software application development seems to be in relatively reduced demand.

Pulling it all together, it seems like the IT departments have tied business innovation into the rigors of client-server software application development. By doing this, all the communications overhead that goes with doing it right are implied.

It seems like we need a new abstraction on top of software... a layer that splits technology out of the problem domain, allowing business users to develop their own applications.

I've hijacked the word "metaware" as a way of thinking about the edge between business users as process actors (wetware) and software. Of course, it's derived from metadata application concepts. At first, it seems hard to grasp, but the more I use it, the more it seems to make sense to me.

Here is how I approach the term...

As I've mentioned in the past, I think of people's roles in business systems as "wetware". Wikipedia has an entry for wetware that describes its use in various domains. Wetware is great at problem solving.

Why don't we implement all solutions using wetware?

It's not fast, reliable, or consistent enough for modern business needs. Frankly, wetware doesn't scale well.

Hardware, of course, is easy to grasp... it's the physical machine. It tends to be responsible for physical storage and high-speed signal transmission, as well as providing the calculation iron, and general processing brains for the software. It's lightening fast, and extremely reliable. Hardware is perfect in the physical world... if you intend to produce physical products, you need hardware. Hardware applications extends all the way out to wetware, typically in the form of human interfaces. (The term HID tends to neglect output such as displays. I think that's an oversight... just because monitors don't connect to USB ports doesn't mean they're not human interface devices.)

Why do we not use hardware to implement all solutions?

Because hardware is very expensive to manipulate, and takes lots of specialized tools and engineering know how to implement relatively small details. Turnaround time on changes makes it impractical in risk-management aspects for general purpose / business application development.

Software in the contemporary sense is also easy to grasp. It is most often thought to provide a layer on top of a general purpose hardware platform to integrate hardware and create applications with semantics in a particular domain. Software is also used to smooth out differences between hardware components and even other software components. It even smooths over differences in wetware by making localization, configuration, and personalization easier. Software is the concept that enabled the modern computing era.

When is software the wrong choice for an application?

Application software becomes a problem when it doesn't respect separation of concerns between integration points. The most critical "integration point" abstraction that gets flubbed is between business process and the underlying technology. Typically, general purpose application development tools are still too technical for user domain developers, and so quite a bit of communications overhead is required even for small changes. This communications overhead is becomes expensive, and complicated by generally difficult deployment issues. While significant efforts have been made to reduce the communications overhead, these tend to attempt to eliminate artifacts that are necessary for the continued maintenance and development of the system.

Enter metaware. Metaware is similar in nature to software. It runs entirely on a software-supported client-server platform. Most software engineers would think of it as process owners' expressions interpreted dynamically by software. It's the culmination of SOA/BPM... for example BPMN (Notation) that is then rendered as a business application by an enterprise software system.

While some might dismiss the idea of metaware as buzz, it suggests important changes to the way IT departments might write software. Respecting the metaware layer will affect the way I design software in the future. Further, respecting metaware concepts suggests important changes in the relationship between IT and the rest of the enterprise.

Ultimately it cuts costs in application development by restoring separation of concerns... IT focuses on building and managing technology to enable business users to express their artifacts in a technologically safe framework. Business users can then get back to innovating without IT in their hair.

The Mature Software Industry, Corporate Specialization, p2

2008-03-24T15:32:00.001-04:00

In driving down I-293 around the city of Manchester one night this weekend, I noticed some of the old factories across the river were lit up so you could see the machinery they contained. Those machines have probably been producing goods reliably for decades.

In my last post, ("Corporate Specialization") I used an analogy of power plants to describe how software engineering groups might someday fit into the corporate landscape.

I found myself thinking that a more precise analogy would be to liken software application development to... hardware application development.

When it comes down to it, hardware, software... the end user doesn't care. They're all just tools to get their real jobs done.

I remember seeing this when I was a kid. I recall observing that when the Atari 2600 has a cartridge inserted, and it's powered on, the hardware and software components were functionally indistinguishable. The complete system might as well be a dedicated-purpose hardware machine. It became an appliance.

Modern platforms don't really change that appliance effect to the end-user.

So, aside from operators, I'm sure these classic B&M manufacturers have technical people to maintain and manage their equipment. I'd be surprised to find out that they keep a full team of mechanical engineers on the staff, though. It seems to make sense that a mature software development industry will start to look much more like that over time.

Further, take a look at computer hardware. It's undergone some maturing over the past few decades too. There really aren't many companies that actually bake their own. I remember tinkering a bit with chips & solder. Then, I started building PC's from components. While my current desktop at home is still one of my own "Franken PC's", I think it's been over a year since I even cracked the case on it. I suspect that when I finally break down & replace the thing, it will be 100% Dell (or maybe Sony) [and Microsoft].

With respect to software engineering, never mind all that FUD we're hearing about IT getting sucked into business units. That's good for "operators" and "maintenance" types, who become analytics and process management in the new world order. I suspect the heavy lifting software engineering will eventually be done mostly by companies that specialize in it.

With that, it might be more educational to look at the histories-to-date of the industrial-mechanical and electrical engineering groups to see the future of the software engineering group.

I think this might be where MS & Google are really battling it out, long term. As the software industry continues to mature, the companies with the most proven, stable software components will end up with the market advantage when it comes to building "business factories" out of them for their clients. ...or maybe it will just be about the most matured development and engineering processes... or maybe both... ?

Corporate Specialization

2008-03-24T15:27:00.002-04:00

There's an old adage: "Anything you can buy, you can produce yourself for less."

In our business, we're well aware that there are a few fundamentally flawed assumptions with that sentiment. Despite the barriers to entry and many other recurring costs, somehow the idea seems pervasive in the business world.

I started my career in a consulting shop that insisted it was a product company. Then I moved, through market forces, into products based companies. I stayed on board with some of them long enough to help shut out the lights and unplug the servers when the sales didn’t hit the marks. The other big problem I’ve seen with product shops was that it’s engineering group typically went through its own “release cycles”… once the product was released, my team was cut to the bone.

I've never been in a classic IT shop, per se, but I’ve definitely been on tangent IT-support projects within “product” oriented companies. In IT groups, I’ve often thought that companies might see IT projects as frills and overhead. At some level, I think the pervasive IT-alignment is a counter measure to that idea. Still, it seems IT projects are typically viewed as liability, rather than asset. When it comes time for the company to refocus on its core competencies, (which is inevitable in the ups & downs of business) IT projects are prime targets for cutbacks.

Since the “.COM bust”, in these types of companies, an engineer on the staff for just three years is often seen as a “long-timer”… but no one feels bad about that, since a lot of these companies fold in that time frame, as well.

After experiencing the product based companys' ups & downs, and seeing many colleagues who had already been through the IT shops, I’m convinced… the outsourced project / consulting route is really the wave of the future, even more than it has been in the past. It’s only a matter of time before business admits this necessity as well.

It makes sense... I wouldn’t figure on every company to own & operate their own power plant.

Why should they typically want their own software engineering group if they don't have to?

[Edit: See also Part 2 Here]

The Wetware Bus

2008-03-15T14:05:00.003-04:00

A colleague pointed me to this post about display size improving productivity.

It brings up a favorite topic of mine... With my current laptop (@1024x768), it's a bit of a pet-peeve.

It totally makes sense that tuning bandwidth to wetware improves productivity. Display real-estate is the front-side bus to wetware!

That's why I also like the Dvorak keyboard layout... Personally, I never achieved the level of proficiency with twenty +/- years of QWERTY (with formal training) that I have with the Dvorak layout over the past three years. Dvorak's a hard curve to get past, but I committed to it in the interest of tuning bandwidth (and maybe hoping to reduce RSI).

All Exceptions Are Handled

2008-02-21T21:24:00.002-05:00

A recent colleague of mine was fond of pointing out that “all software is tested.” This truism is a simplification, of course, but the basic premise is this: the bugs will be exposed. The only question is “by whom?”

I submit for your consideration another truism, perhaps a corollary to the former: “all exceptions are handled.” Yes, ALL exceptions… and I’m not just talking about a top-level catch-all exception handler.

That requires further explanation. In order to fully back that up, let's talk about what a typical exception handling mechanism really is.

According to Bjarne Stroustrup in his classic (“The C++ Programming Language”), “the purpose of the exception-handling mechanism is to provide a means for one part of a program to inform another part of a program that an ‘exceptional circumstance’ has been detected.”

Unfortunately, Stroustrup’s book was published a decade ago, and the consensus on the subject among software engineers has hardly been less vague since.

Really, the typical exception-handling mechanism enables a delegator to effectively reconcile an expression of non-compliance in its delegated tasks.

So exception-handling mechanisms usually have several parts. To begin with we have a task or process. Then, there’s a task supervisor. The supervisor is the method that contains the typical try-block. This try-block generally wraps or makes calls to the task. There’s the exception object, which is really the information that describes an occurrence of non-compliance… the excuse, if you will. An instance of an exception object is not the actual exception itself, but merely the expression of the actual exception. Finally, there’s the handler, which is the reconciliation plan that is hosted by the supervisor.

In many languages, supervisor code can have zero or more reconciliation plans, each for a particular class of exception. A handler can also throw or re-throw, meaning a succession of supervisors could potentially be informed of an insurrection.

So it can be said that there are actually zero or more reconciliation plans or handlers… within the program. How do “zero handlers” square with the bit about all exceptions being handled?

We developers easily forget that software is always a sub-process of another task. Software doesn’t run for its own entertainment. Software is social. It collaborates, and blurs the boundaries between processes. Software processes that are not subordinate to other software processes are subordinate to… (I’ll borrow a term) wetware. When software experiences a circumstance that it cannot reconcile, the software will invariably express non-compliance in some fashion to a supervisor. In many cases, the supervisor is appropriately software. In many other cases, the supervisor is wetware.

Regardless… whether the program goes up in a fireball of messages, locks hard, blinks ineffectively, or simply disappears unexpectedly; these acts of insubordination are expressed to wetware. Wetware always has the final reconciliation opportunity. A user can take any and all information he or she has on the subject of the exception to handle that non-compliance.

The user could do a number of things: They could report it to their superior. They could ask someone else to investigate. They could try again. Even if they choose to do nothing, they are making that “informed” decision… and there. That’s what I mean when I say “all exceptions are handled.”

Now don’t worry. I’m certainly not about to advocate unprotected code.

Unfortunately, we developers aren’t trusting types. We go out of our way to protect against exceptions everywhere. We write code in every procedure that secretively skips over processing something if there’s a broken dependency. We write exception handlers that quietly log the details or completely swallow exceptions.

There are occasions when “do nothing” is a reasonable reconciliation. Much of the time, however, we engineers are engaging in illicit cover-up tactics. Handling decisions get co-opted from real process authorities and made at levels where the best interests of the stakeholders are not considered. Often cover-ups add redundant checks that make code harder to understand, maintain, and fix. It actually becomes complexity that exceeds that of proper exception handling.
Where does this get us? You gotta be fair to your stakeholders.

Keeping all stakeholders in mind is critical to building an exception throwing and handling strategy, so let’s dig in on that a bit. The first stakeholder most everyone thinks of in just about any project, is the project sponsor. The business requirements most often need to support the needs of the project sponsor. The end user comes next, and things like client usability come to mind. Sometimes end users fall into categories. If your program is middleware, you might have back-end and front-end systems to communicate exceptions with. In some cases, you not only have to consider the supervisory needs of the end user, but also, potentially, a set of discrete back-end system administrators, each with unique needs.

Remember, however, that during project development, you, the developer, are the first wetware supervisor and stakeholder of your program’s activities. You’re human; you’re going introduce errors in the code. How do you want to find out about them?

Here’s a tip: the further away from the original author that a bug is discovered, the more expensive it is to fix.

You want mistakes to grab you and force you to fix them now. You do not want them to get past you, falsely pass integration tests, and come across to QA, or end users, as a logic flaw in seemingly unrelated code.

Again, in the typical cover-up strategy, dependency-checking code becomes clutter and adds complexity. You may find yourself spending almost as much time debugging dependency checks in cases like this, as actually fixing functional code. Part of a well-designed exception strategy is writing clean code that keeps dependency checking to a minimum.

One way to make that possible is to set up encapsulation boundaries. This is also a good practice for other reasons, including managing security. Top level processes delegate all their activities so they can supervise them. Validate resources and lock them as close to the start of your process as possible, and throw when it fails. Validate data and its schema when and where it enters the process, and throw when it fails. Validate authentication & authorization as soon as you can, and throw when it fails. Once you have your dependencies sanity-checked, clean processing code can begin processing.

Don’t forget that the UI is an edge. Not only should input be validated, but due to important needs of the user, reconciliation code that respects the user should be set in place.

Thrown exceptions, of course, need to be of well-known domain exception types that can be easily identified in the problem domain, and well handled. Don’t go throwing framework exceptions. It will be too easy to confuse a framework exception for an application exception. Some frameworks offer an “ApplicationException” designed for the purpose. I might consider inheriting from ApplicationException. Throwing an “ApplicationException” is probably not unique or descriptive enough within the problem domain to make the code understandable.

By taking an encapsulation boundary approach, lower level tasks cleanly assume their dependencies are valid. If a lower level task fails due to a failed assumption, you’ll know instantly that it’s a bug. You’ll know the bug is either right there on the failure point, or it’s on the edge where the dependency or resource was acquired.

Another important consideration in an exception-handling strategy: code reusability. It often makes sense to decouple a supervisor from its process. In this way, it becomes possible to apply consistent supervision over many different processes. Another interesting possibility that becomes possible in this manner is the idea of code execution with design-time or even run-time configurable supervision. Each different “supervisor” construct could respond to the informational needs of a different set of stakeholders. Finally, handling can be decoupled from supervision. This provides another way to make stakeholder needs configurable.

Ok, so I'm getting carried away a bit there... but you get what I'm saying.

By designing encapsulation boundaries, and by decoupling the various parts of exception handling, you can support effective supervision and reconciliation through each phase of the project life cycle via configuration. In this way, you can honor all the stakeholders during their tenure with the application, and write better software.

[3/12/08 Edit: I originally liked the term "trust boundaries", because it focused on being able to trust dependencies, and also brought security as a dependency to mind, but "encapsulation boundary" is much more precise. Thanks, Kris!]