Dylan Does Digits

Blog is moving

2010-04-17T14:20:00.001-07:00

This blog has moved to http://digits.transatlantical.me

Parameterized Query with LIKE using Oracle.DataAccess

2010-04-10T10:05:00.000-07:00

If I have to figure this out yet again for the nth time, I'll get ticked off. So here is how to do a LIKE query in C# using an OracleParameter:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Oracle.DataAccess;
using Oracle.DataAccess.Client;
using System.Data;

namespace Oracle
{
    class Program
    {
        static void Main(string[] args)
        {
            const string searchTerm = "bd";
            const string cs = @"Data Source=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=thehostname)(PORT=1521)))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=the.service.name)));User Id=user;Password=pass.word;";
            const string sql = @"select rm_id, ref_fac_id || ' - ' || rm_num as name
                        from asbestos.rooms
                        where rownum < 51
                          and ref_fac_id is not null
                          and( lower(ref_fac_id) like :q
                            or lower(rm_num) like :q)
                        order by rm_num asc";
            var conn = new OracleConnection(cs);

            var cmd = conn.CreateCommand();
            cmd.CommandText = sql;
            cmd.CommandType = CommandType.Text;
            var param = cmd.CreateParameter();
            param.DbType = DbType.String;
            param.Direction = ParameterDirection.Input;
            param.ParameterName = ":q";
            param.Value = string.Format("{0}{1}{0}", "%", searchTerm.ToLower());
            cmd.Parameters.Add(param);
            conn.Open();

            using (var reader = cmd.ExecuteReader(CommandBehavior.CloseConnection))
            {
                while (reader.Read())
                {
                    Console.WriteLine(reader.GetString(1));
                }
            }

            Console.Read();
        }
    }
}

Port blocking in Windows 7 makes for unhappy WCF testing

2010-03-26T13:29:00.000-07:00

I typically do something like this when I'm testing a WCF service (using NUnit here):

namespace Tests
{
    [TestFixture]
    class When_running_as_hosted_service
    {
        private ServiceHost _host;
        private IWhoKnowsService _client;

        [SetUp]
        public void Before_any_test()
        {
            // 1.
            // Start a WCF service hosted in a generic ServiceHost
            // This is exactly the same as hosting in IIS or a console app
            const string port = "http://localhost:12345/";
            var uri = new Uri(port);
            _host = new ServiceHost(typeof(WebService), uri);
            _host.AddServiceEndpoint(typeof(IWhoKnowsService), new WSHttpBinding(), uri);
            _host.Open();
            
            // 2.
            // Now dynamically build a client to that service.
            // This is the same as 'Add Service Reference' but without all
            // of the code generation
            _client = new ChannelFactory(new WSHttpBinding())
                .CreateChannel(new EndpointAddress(port));
        }

        [TearDown]
        public void After_all_tests()
        {
            // Tear it down so that the TCP port is release correctly
            _host.Close();
        }

        [Test]
        public void Can_get_a_list_of_descripts_for_a_skill_level() 
        {
            var descriptions = _client.DescriptionsByLevel(SkillLevel.Padwan);
            Assert.IsNotNull(descriptions);
            Assert.Greater(descriptions.Length, 0);

            foreach (var item in descriptions)
            {
                Console.WriteLine(item);
            }
        }
    }
}

I've used this technique many, many times on Windows XP but I just made the switch to Windows 7 for development, having skipped Vista entirely. When trying to run that test I get this error back from the test runner:

AddressAccessDeniedException: HTTP could not register URL http://+:12345/<…>.  Your process does not have access rights to this namespace

That's not good. So I hit the Internet and found an excellent post by Nicholas Allen that set me straight. Basically Windows 7 really locks down your usage of TCP ports over HTTP. Using the trusty old netsh command he shows how to open up the port of choice in one easy step:

netsh http add urlacl url=http://+:12345/ user=dev2010\Dylan

Problem solved! I reran the test in Visual Studio and got a pass.

NOTE: You must be running your Command Prompt as an administrator in order for this to work. In other words, right click on Command Prompt in your Start Menu and choose the appropriate option.

Ruby strings on Windows. Backlashes again!

2010-03-18T09:12:00.000-07:00

There are all kinds of discussions about how to correctly use single or double quoted strings in Ruby, especially where character escaping is concerned. I was faced with just such an issue today and thought I'd share my experience.

Basically I'm building a rake script to automate my .NET project setup. Part of that setup involves doing some search and replace is SQL scripts so that a developer can store some settings in a config file and have that populate variables in files. For example:

YAML file config.yml contains a setting called production -> db -> backup_location. The text in the file is "\\\\some\\unc\\path\\name". You see how the backslashes in the Windows path are doubled up? That's to satisfy Ruby and it is in double quotes.
The rake script uses this value as a parameter to a method that I wrote that is heavily influenced (copied with simplification!) by the albacore project, which is a set of rake tasks geared towards .NET projects. The important lines of the script are these:
```
replacements.each do |search,replace|
    search_term = "%#{search}%"
    template_data.gsub!search_term, replace
end
```

My SQL file has this in it that should have various paths inserted:

FROM  DISK = N'%backup_file%' WITH  FILE = 1
 , MOVE N'Cityworks_Production' TO N'%db_dir%\Cityworks_WCS.mdf'
 , MOVE N'Cityworks_Production_log' TO N'%db_dir%\Cityworks_WCS_1.ldf'
 , MOVE N'sysft_CityworksFtCat' TO N'%db_dir%\Cityworks_WCS_2.CityworksFtCat'
 , move N'ftrow_CityworksFtCat' to N'%db_dir%\CityworksFtCat.ndf'
,  NOUNLOAD,  STATS = 10
GO

Unfortunately when I run the script using a UNC path rather than a typical C:/something/or/other path I could never get two backslashes to appear at the start of the path. I tried (literally!) every combination I could think of: doubling up, adding an extra backslash, single quotes, double quotes, you name it. Along the way I wrote a little one liner that converts a string from Ruby path to a Windows path (winpath = ruby_path.gsub('/','\\')) but still no joy. Then I searched some more online and finally found an answer: it wasn't my various File.join or File.expand_path calls that were the issue: it was the gsub call itself that was the problem.

In fact the question was answered by Matz himself (inventor of the Ruby language). While my string with four backslashes actually did give me the right answer of two escaped backslashes, gsub also assigns special meaning to backslashes. So get this - you need to quadruple any backslash in a string if you are going to gsub it. Crazy but true. Here's what my fixed up code looks like now:

replacements.each do |search,replace|
    search_term = "%#{search}%"
    template_data.gsub! (search_term) { replace } unless replace.nil?
end

The trick is to use a block rather than tacking on the replace as a parameter. In this way gsub offloads the whole string interpretation to the block rather than doing its funky thing with interpreting the backslashes inline. Boy, am I glad I got that figured out.

A little coding test

2010-02-25T12:06:00.001-08:00

Do this without compiling...

Someone recently asked me about some questions to give someone during an interview. The fun part is having the interviewee do the problem on a white board without an IDE to help them. Or having them do it in Visual Studio, Eclipse, or whatever IDE they typically use for their language of choice. Here is one such problem. It’s not very complicated but it does force the interviewee to think about assumptions being made:

Given a certain amount of money that you need, and a set of coin denominations, tell me the minimum number of coins of each denomination I need to have that amount of money.

Having heard the problem statement I tried the whiteboard version. Then I opened up Visual Studio and typed out the following. I waited until I typed the whole thing before trying to build or run tests, just to keep myself honest about assumptions I was making.

STOP! Try to do this yourself without reading my approach. Not that I'm right, fast, wrong, or anything else, but see how you do first.

public class CoinCounter
{
  /// <summary>
  /// Assumptions made:
  /// 1. There's always a denomination of 1 unit
  /// 2. There are no fractional units like the 1/2 pence in England
  /// 3. The denominations are always in the same units (cents, not cents and dollars)
  /// </summary>
  public static Dictionary GetMinimumCoinsNeeded(int[] denominations, int targetValue)
  {
    var result = new Dictionary();
    
    // Sort in descending order so we get the biggest denomination first
    Array.Sort(denominations);
    Array.Reverse(denominations);
    int remaining = targetValue;
    
    foreach (int denomination in denominations)
    {
      // Turns out there's a Math.DivRem function that gets both the
      // coins and remaining value in one go.
      int coins = remaining / denomination;
      
      // Forgot to add this first time through - don't need to
      // do the rest if there aren't any coins
      if (coins == 0) 
        continue;
      
      // Got this backwards first time - denomination % targetValue
      remaining = targetValue % denomination;
      result.Add(denomination, coins);
      Console.WriteLine("Adding {0} coins of denomination {1} leaving {2} to work with...",
            coins, denomination, remaining);
      
      if (remaining == 0)
        break;
    }
    
    return result;
  }
}

And see if it works when you finally build it

When I saw this problem I ended up with a solution that resembles the one above. I did make some goofy mistakes that involved some wiping out and re-“coding” on the whiteboard. For example, I'd just been using Excel to do some modulus work and actually wrote in the Excel Mod function rather than the C# modulus (%) built in function. Embarassing! The other things I found when I actually tried to compile then run the code are listed in C# comments. For example, I got the order of the two parameters in the % call backwards. Here is the final part where I report the number of different coins to use:

var result = CoinCounter.GetMinimumCoinsNeeded(new[] { 2, 1, 100, 500, 10, 25 }, 2341);

foreach (var item in result)
{
 Console.WriteLine("Denomination: {0}, Coins: {1}", item.Key, item.Value);
}

Which returns this output:

Denomination: 500, Coins: 4
Denomination: 100, Coins: 3
Denomination: 25, Coins: 1
Denomination: 10, Coins: 1
Denomination: 1, Coins: 1

Try it yourself. It’s a good test of your logic ability and knowledge of your core language and platform.

It only took me 3 hours to get Rails running on Ubuntu

2010-02-22T20:29:00.000-08:00

I wish! I've used Ruby (the .NET version IronRuby at least) for building a project, and tinkered with Rails in a non-serious way, but I find myself wanting to test something real for work these days. Well, it's a long shot, but I'm wondering what the cost in man hours would be to move our public Subversion subversion from a Windows 2003 VM to Ubuntu. The reason? I like VisualSVN Server for its ease of use and no brainer installation, but I need to get a decent web GUI for svn up and running. There's the PHP-based WebSVN project that presumably I can plug in to svn, but I honestly don't want to mess with co-hosting the private Apache server that VisualSVN uses on port 80 with a full blown Apache install...also on port 80...just for the PHP stuff.

Maybe that really isn't too big of a deal, but I also find myself wanting to try out the truly beautiful and useful svn GUI that is Warehouse. It is a Ruby on Rails application, and that is the real impetus for trying Ubuntu and Apache: Rails is a bugger to get working correctly on Windows in anything other than play mode. I've tried mongrel packs (!), WebBrick (can't believe I actually did that), and other approaches and it just seems like I'm trying to force the Rails round peg into the Windows square hole. So on to Ubuntu.I've been using Sun's (Oracle's?) VirtualBox for a while now, ever since VMware completely hosed my laptop and external USB drive with all my backups. I really like it because it just works and has great bridged networking support. What with all the crazy eth0 and other intuitive network adapter names on Linux, that out-of-the-box networking experiences is truly a blessing. So without further ado, here's a screenshot of my desktop about 10 minutes ago. I'm pretty pleased with progress thus far, even if most of it was trolling through online "HOW-TOs" and the phenomenal TekPub Linux for Softies screencast series (totally worth $14).

I have to say, the apt-get package manager for Ubuntu is truly amazing. I was floored when I just typed sudo apt-get install apache2 and it just worked. Anyway, nothing useful to mention like how I actually got this far, but again, I'm pretty pleased with the end result: a scaffolded Rails app that actually serves up HTML. Woohoo!

Oracle Spatial SIG group

2010-02-22T11:08:00.001-08:00

I just finished setting up a new place on the web for the Independent Oracle User Group (IOUG) Oracle Spatial Special Interest Group (SIG). That’s a mouthful so luckily we abbreviate it to Spatial SIG and still know what we are referring to. Anyway, if you are interesting in Oracle Spatial technology then head on over to the new group site and take a look at membership options, our charter, and the newly relaunched discussion list. Okay, so the discussion list has one post in it so far (from yours truly) but it will get more content as the days and weeks pass.

Lessons learned on a project...time to 'fess up

2010-02-09T19:31:00.000-08:00

Here's an internal presentation I gave at one of our ‘DevCasts’ a few months back. I shared my successes and failures with the team in return for a great discussion on effective use of tooling, when is too much design, and some virtual back slapping. Nothing earth shattering but I learned a lot...meaning it hurt sometimes but felt good at the end of the project!

Dev Cast - Lessons from a Complex Application

View more presentations from Dylan Thomas.

Markdown, Pandoc, Love

2010-02-09T16:32:00.000-08:00

I'm in the middle of writing a lengthly software requirements spec this week. It's actually been fun to go through the process again on a fresh project with a clean slate. The requirements gathering meetings were great: business leader, sales people, and developers all in one room and surprisingly united in their goals. I used the usual tools of the trade—like FreeMind and Unable (a clone of a great little note taking application called Notable)—and was able to get a lot of the requirements and user story details knocked out on site.

Then I was sitting back at the office wondering how to keep all of my SRS edits in version control and hit upon the idea of using HTML rather than Word. No good reason other than wanting to keep my changesets small. I'm very comfortable with HTML and can whip out a mean CSS/HTML combo in no time flat. The beauty of it all, of course, is that I can just write what I want without fussing over headers, page numbers, indenting bullets the right amount, and all that guff. I like a good looking document as much as the next guy but frankly I just needed to focus on getting the requirements done, not worrying about the font color of my headings.

Enter Markdown

Being no stranger to sites like Stack Overflow I know that people can write pretty complex stuff using HTML. But also that it's a total drag when again, all I really need is the write text with some minimal markup. So if I want to highlight something I might type this is *important* stuff and I might want to create a bulleted list like so:

* item one
* item two
* and something else

This is pretty much how I write notes anyway, and it turns out it's very, very close to how a simple 'HTML lite' syntax called Markdown does things. Essentially is a dead simple syntax that can be transformed into HTML by any number of tools. So I did a little test. I started to write my SRS in Markdown format using a text editor rather than Word, all the while committing my changes to Subversion. It was actually pretty nice and I did indeed focus on writing instead of formatting. But then what? I've got a text file in some goofy format that only a programmer could love (ahem! guilty as charged!!!). I needed to deliver something usable to my client like a web page. Here's what a typical section looks like:

#### Trademarks
All brand names and product names are trademarks or registered trademarks of their respective companies

Software Requirements Purpose and Process
-----------------------------------------

A Software Requirement Specification--hereafter called an SRS--is an initial attempt to describe the functionality of a custom software product or custom system. It can be likened to the blueprint for building a new house (or perhaps remodeling an existing house). Just as an architect guides a homeowner through their needs for the new house, so Woolpert technical staff guide clients though the requirements process. This implies two very important points that the client project management team and stakeholders should remember through the software requirements process:

* *Continuous client involvement* is crucial to success. A homeowner generally does not sign off on architectural plans early on and then just hope that the architect understood perfectly your meaning. Woolpert technical staff are highly skilled, but they cannot possibly know all the needs or business processes of the clients as well as they do.
* *Requirements are a moving target*, not a set of mandates that are set in stone. To be sure, some requirements are so central to the project that they are unlikely to change. However, changing requirements...

Ignore the actual content for a second, and focus on the simple rules that Markdown uses. If I want a heading I underline the text. If I want bullets I put an asterisk as the first character and leave some white space around the 'list'. If I want a highlight ( in HTML) I put a couple of asterisks around the text. Simple. To be sure there are complicated parts but at that point I just drop in the HTML, e.g., a div is just a div.

Enter Pandoc

In the 'Holy Cow! How did he do that?' category I found a wonderful little program called Pandoc written in Haskell (?!) by an associate professor of philosophy from the University of California at Berkeley (!!!!???!?!). Not exactly a normal, run of the mill combination there, but Pandoc is a beautiful little program. It essentially does one thing, and does it exceptionally well: it takes one or more Markdown text files and spits out HTML or PDF (or DocBook and a few other formats). So in order to turn my srs.markdown text file and various screenshots in to an HTML page complete this those hyperlink-thingies, I just double-click the batch file I wrote and in less than a second I've got an excellent SRS document in HTML format that my client can open on any computer, share with anyone on the team, and click around in. From my script you can see that I'm including some CSS stylesheets, I'm asking for HTML out, and I've overridden a default HTML template because I needed to add some autogenerated stuff to the HTML HEAD section. Oh, and I also wanted it to generate a hyperlink table of contents for me based upon my H tags. Short and sweet.

pandoc -o srs.html
  -c "assets/css/blueprint/print.css"
  -c "assets/css/blueprint/screen.css"
  -c "assets/css/woolpert.css"
  -w html srs.markdown
  --template html.template
  --toc

I wish I could show you some of the output but I'm under a non-disclosure agreement. But I promise, it's really...neat. Not to mention, when I've made changes, my client simply gets the latest from my employer's public svn repository, double-clicks the same batch file, and gets the very latest docs. [Smile]

Whew!

2010-02-02T20:47:00.000-08:00

Holy tamales, what a day! I just wrapped up a marathon session getting a new "product" ready for a demo on the east coast (yes, the one that's not the West Coast). I spent all of yesterday struggling with 3rd party vendor software and finally deciding that it sucked and I can do it better myself. So I ripped that stuff out, wrote some handy dandy SQL, and had a WAAAAAAY faster solution by 7:00am this morning.

I fed Lyra and reunited her with her pacifier to grant us some peace and quiet for an additional 30 minutes, and managed to knock out some other bugs that had be...well, bugging me. All before 8:00am :-) And a few hours later it's looking really spiffy. OK, so actually it's not that exciting to look at, just a couple of ASP.NET MVC sites with dead simple controllers. But man, oh man, is the code behind it all sweet. We've got messages, we've got reliability, we've got factories, we've got...whatever: it's freaking cool because it took me ages and the prospective client is going to be very excited when I show them a web page with their logo on it.

The real bottom line is that I can actually look forward to my long flight tomorrow as a time for tweaking and tinkering and polishing the demo rather than sweating my 'nads off while trying to just get the code working. “I love it when a plan comes together”

Poor man's spatial query against ArcSDE

2010-02-02T07:39:00.000-08:00

I pretty much forget this every single time that I need to do a quick minimum bounding extent (MBE) or 'bounding box' query in ArcSDE. The ArcObjects alternative is just too painful to deal with when you simply need the extent of a feature, and SQL does the trick.

use sde_database
go
declare @meter nvarchar(20)
set @meter='MN13'

select layer_id from sde.SDE_layers where [owner] = 'GIS'
  and table_name='WMETER'

-- Answer is 155, and this is the table name for the next query
-- Notice the f155 here...

select b.OBJECTID, f.eminx,f.eminy from gis.f155 f
inner join gis.WMETER b
 on f.fid=b.Shape
where b.SerialNumber = @meter;

Like I said, stupid simple but I still have to look it up every darn time.

There are many ways to skin a cat

2010-01-30T08:24:00.000-08:00

A colleague just sent me this link. It's pretty funny on one hand since I can see myself in a few of the samples (like "Enterprise Developer" when I was doing my one and only Java project early in 2009 - that was an XML-eye opener!). But the other aspect is that there are always multiple ways to achieve the same goal in software development. That's why I like our internal peer review program: I get to see every conceivable approach and discuss the merits of each. Sure, some are smellier than others. But on the whole I like that fact that we can use our tool of choice is such a variety of ways. I just learned a bunch about Python by simply looking through the listing!

Idiots Guide To UML Class Diagrams

2010-01-14T17:03:00.000-08:00

I put this together and figured I'd share it. I always draw the wrong little diamond for aggregation vs. composition so maybe this will help me to use the right pen for the job next time! Please ignore the formatting. This is a cut'n'paste job from an HTML document that has very pretty styles in it. Blogger kind of hosed it.

Basic UML for Class Design

Introduction

Unified Modeling Language (UML) is a graphical notation covering a wide variety of steps in software and system design. One of the core uses of UML is to design object oriented systems using classes, interfaces, relationships, and other simple graphical elements. The purpose of this document is to describe the most common elements of UML diagrams geared towards class design.
All diagrams were created using the yUML website.

The Elements

A Class

Just the class - no properties or other information

A class with details. The name is at the top. The public fields (or properties) come next. They have a plus sign if they are public and a minus sign if they are private. Finally come the methods, again marked public or private with signs.

An Interface

Just the name - no other contract details

With some details

Relationships

There are a few different kinds of relationships and deciding which one to use can be a bit confusing at first because it depends upon the intent of the relationship. Simple associations sort of ignore the two main types of associtions and can be used for higher level UML diagrams. But being more specific is important as the design evolves and so understanding the difference between aggregation and composition is key. It really comes down to ownership and object lifecycle

Aggregation—This is known as a has a relationship because the containing object has a member object. But here is the critical part: The member object can survive or exist without the enclosing or containing class, so it can have a meaning beyond the lifetime of the enclosing object.
Example: A room has a table and the table can exist without the room.
Composition—This is known as a is a part of or is a relationship because the member object is part of the containing class and cannot existing or survive outside the context of the containing class. This also means that the lifetime of the member object ends with the lifetime of the enclosing object.
Example: The IT Department is part of the Company. The IT Department cannot exist without the Company and has no meaning after the lifetime of the Company.

Simple Association

Customers have a Billing Address

The relationship between a Customer and and their Orders is described by the action orders. So it read a Customer orders an Order.

Cardinality

A Customer can have zero or more Addresses. An Address must have exactly one Customer. Notice how the numbers are positioned on the line: the customer fact (zero or more) is written next to the Address class.

Directionality

An Order has an Address called billing and an Address called shipping. This does not say anything about the Address-to-Order relationship.

Aggregation

A Company has exactly one Location. A Location has a Point.

Composition

The Company has exactly one Location. The Location does not exist outside of the context of the Company.

Other Dependencies

Inheritance

Both the Contract and Salaried classes inherit from the Wages class.

The NightlyBillingTask class inherits (implements) the ITask interface

Dependents On

The HttpContext class is dependent upon the Response class. The relationship reads as the HttpContext class uses the Response class. There is no has a or is part of a Response implied here. It is just plain usage.

A Complete Example

Here it all comes together with some extra annotations in the form of notes.

Dev Cast Dependency Injection

2010-01-13T20:58:00.001-08:00

This is pretty basic stuff, but that's sort of the point. I'm giving this presentation before a bunch of code samples at my company tomorrow. We call these ad hoc "share some ideas" meetings DevCasts (props to Joe for the name).

Dev Cast Dependency Injection

View more presentations from Dylan Thomas.

Running a bunch of SQL scripts in C#

2010-01-05T20:31:00.000-08:00

Here is how I did it. Read in any script called *.sql from a directory that does not have the word 'test' somewhere in it, then use the Microsoft.SqlServer.Management.Common and Microsoft.SqlServer.Management.Smo namespaces for the following:

public bool RestoreFieldDatabaseFromBackup(string connectionString)
{
 List scripts = this._sqlDirectory.GetFiles("*.sql", SearchOption.AllDirectories)
  .Where(s => !s.Name.Contains("test"))
  .Select(s => s.FullName).ToList();

 if (scripts.Count() == 0) return false;

 var connection = new SqlConnection(connectionString);

 try
 {
  for (int i = 0; i < scripts.Count; i++)
  {
   using (var reader = new StreamReader(scripts[i]))
   {
    var server = new Server(new ServerConnection(connection));
    var db = new Database(server, connection.Database);
    string scriptText = reader.ReadToEnd();
    Log.For(this).Debug("Running: " + scripts[i]);
    db.ExecuteNonQuery(scriptText);
   }
  }
 }
 catch (Exception ex)
 {
  Log.For(this).Error(ex);
  return false;
 }
 return true;
}

LINQ to objects: gettting the SQL version into C#

2009-12-24T10:41:00.000-08:00

In my last post I described a SQL query that grouped records by an ID (non unique obviously otherwise what is the point of grouping in the first place?), and then selected only the most recently created record from each group. I have been struggling with the LINQ query to do that same thing in C# and finally hit upon an answer. Not the answer maybe, but it works so I’ll call in 'an' answer

void Main()
{
 var list = new WorkorderStatusChangeEvent().Build();
 var ids = from w in list
   group w by w.WorkorderID into g
   let latestDate = g.Max(w => w.TimeOfEvent)
   select g.Where(w => w.TimeOfEvent == latestDate)
    .Select(w => w.Id).First();
 var results = list.Where(x => ids.Contains(x.Id));
 results.Dump();    
}

public class WorkorderStatusChangeEvent
{
 public virtual Guid Id { get; set; }
 public virtual string WorkorderID { get; set; }
 public virtual string OldStatus { get; set; }
 public virtual string NewStatus { get; set; }
 public virtual DateTime TimeOfEvent { get; set; }
 
 public IList Build()
 {
  var earliest = new WorkorderStatusChangeEvent
     {
     Id = Guid.NewGuid(),
     OldStatus = string.Empty,
     NewStatus = "NO",
     TimeOfEvent = DateTime.UtcNow.Subtract(TimeSpan.FromSeconds(120)),
     WorkorderID = "123"
   };
  var earlier = new WorkorderStatusChangeEvent
       {
        Id = Guid.NewGuid(),
        OldStatus = "BLAH",
        NewStatus = "NO",
        TimeOfEvent = DateTime.UtcNow.Subtract(TimeSpan.FromSeconds(60)),
        WorkorderID = "456"
       };
  var latest = new WorkorderStatusChangeEvent
       {
        Id = Guid.NewGuid(),
        OldStatus = "BLAH",
        NewStatus = "YES",
        TimeOfEvent = DateTime.UtcNow,
        WorkorderID = "456"
       };
  var anotherone = new WorkorderStatusChangeEvent
       {
        Id = Guid.NewGuid(),
        OldStatus = "BLAH",
        NewStatus = "NO",
        TimeOfEvent = DateTime.UtcNow.Subtract(TimeSpan.FromSeconds(300)),
        WorkorderID = "456"
       };
  IList list = new List();
  list.Add(earliest);
  list.Add(earlier);
  list.Add(latest);
  list.Add(anotherone);
  
  return list;
 }
}

Basically there are two parts. First getting the groupings by WorkorderID and extracting the PK of the object (Id) for the most recent record. This uses the let statement. The second part is a simple contains query that does the equivalent of a SQL IN to pull only those records from list whose Id is in the list of grouped/max date matches.

And by the way, I absolutely could not have ever figured this out without the great LINQPad tool by Joseph Albahari. It is free and yet absolutely priceless. Here are the results from the query as presented by the Dump extension method that LINQPad provides:

Id	WorkorderID	OldStatus	NewStatus	TimeOfEvent
5IEnumerable<WorkorderStatusChangeEvent> (2 items)
8191ab38-e070-4fd0-a8ea-83a7090c16d6	123		NO	12/24/2009 6:29:51 PM
a5cee31a-643c-4642-b0f6-3c8f6e54cbfe	456	BLAH	YES	12/24/2009 6:31:51 PM

And finally, have a great holiday. I've promised not to do any more techie stuff until the New Year. Well, I'm not going to blog until the New Year anyway!

SQL: grouping by maximum date and ID

2009-12-23T10:16:00.000-08:00

Another reminder to myself. I have an audit table that is populated by a trigger on a table. I use this table to drive another process and that process involves finding the most recent addition to the table for a specific work order. So basically if a work order has it’s status changed multiple times—which is perfectly valid—I need to know the last change only. Here is what the source data looks like after changing status on a couple of work orders:

Id	WorkorderId	OldStatus	NewStatus	TimeOfUpdate
F0FE4F81-4CEF-DE11-8442-001C23437026	10	null	CLOSED	2009-12-22 22:51:14.377
F1FE4F81-4CEF-DE11-8442-001C23437026	10	CLOSED	OPEN	2009-12-22 22:51:14.390
F2FE4F81-4CEF-DE11-8442-001C23437026	10	OPEN	CLOSED	2009-12-22 22:51:14.390
F3FE4F81-4CEF-DE11-8442-001C23437026	9	null	CLOSED	2009-12-22 22:51:14.390

Because I created these by firing the trigger in a SQL script for testing the times for two of the records for work order 10 are identical. That’s OK; in fact I need to deal with this possibility in my application so I’m glad it happened that way. Anyway, here’s the SQL that I wrote to get the information that I need. Not rocket science but it is just one of those things that had me stumped for a while until it dawned on me that I needed an inner join rather than a subquery.

select w.Id
 , w.NewStatus
 , w.OldStatus
 , w.TimeOfUpdate
 , w.WorkorderId
from Test.WCS.WorkorderStatusChange w
 -- Inner join on the max date grouped by workorder id
 inner join (select w.WorkorderId as woi, MAX(w.TimeOfUpdate) as tou
   from Test.WCS.WorkorderStatusChange w
   group by w.WorkorderId) as t
  on t.tou = w.TimeOfUpdate
  and t.woi = w.WorkorderId

And that gives me what I need. Again, notice that the two results for work order 10 are valid because both have the MAX(TimeOfUpdate) value:

Id	NewStatus	OldStatus	TimeOfUpdate	WorkorderId
F3FE4F81-4CEF-DE11-8442-001C23437026	CLOSED	null	2009-12-22 22:51:14.390	9
F1FE4F81-4CEF-DE11-8442-001C23437026	OPEN	CLOSED	2009-12-22 22:51:14.390	10
F2FE4F81-4CEF-DE11-8442-001C23437026	CLOSED	OPEN	2009-12-22 22:51:14.390	10

Of course the real question on my mind now is how the heck to do that with HQL or the Criteria API in NHibernate? I’ve wrestled with it for some time now and have basically given up. I am going to grab the whole lot of data from SQL Server and use LINQ to objects to filter out what I need. This is acceptable in this particular case because I delete the rows in the status table once I have read them in; they are only there long enough for the application to add a message to a message queue using the information. Still, it’s bugging me that I cannot figure out the query in NHibernate.

Building WSDLs from .NET assemblies

2009-12-19T09:10:00.000-08:00

More of a reminder to myself than anything else, here's a batch file that creates proxy classes that can talk to WCF services. The WCF service interfaces are defined in the Visual Studio project build output. Then it's a simple matter of using the svcutil command line utility from the .NET SDK to go through the motions. This turns out to be a great way to automate the creation of WSDLs for non-.NET clients, and a good way to create proxy classes that you have a lot more control over than the “Add Service Reference” approach in Visual Studio.

set outDir=..\build\wcs-wsdl
set wsdlDir=..\lib\wcs-wsdl
set svc="C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin\svcutil.exe"
set msbuild="C:\WINDOWS\Microsoft.NET\Framework\v3.5\msbuild.exe"

rmdir /s /q %outDir%
mkdir %outDir%

%msbuild% ..\source\Woolpert.Cityworks.Services\Woolpert.Cityworks.Services.csproj
%svc% ..\source\Woolpert.Cityworks.Services\bin\Debug\Woolpert.Cityworks.Services.dll
%svc% *.xsd www.woolpert.com.wis.cityworks.wsdl /language:C# /out:%outDir%\CityworksClient.cs
%svc% *.xsd www.woolpert.com.wis.security.wsdl /language:C# /out:%outDir%\SecurityClient.cs
%svc% *.xsd www.woolpert.com.wis.spatial.wsdl /language:C# /out:%outDir%\SpatialClient.cs
xcopy *.xsd %wsdlDir%\ /I /Y
xcopy *.wsdl %wsdlDir%\ /I /Y
xcopy *.config %wsdlDir%\ /I /Y
del /q *.xsd
del /q *.wsdl
del /q *.config

In my other projects that need access to clients I can then run a subset of this script and be all set for consuming the web services:

set output=MyMiddleware.cs
del %output%
set svc="C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin\svcutil.exe"
%svc% *.wsdl *.xsd /language:C# /o:%output%

NHibernate Events to the rescue

2009-12-06T15:00:00.000-08:00

The Issue, in which Dylan gets bit in the ass by a legacy database

I'm building some message-oriented-middleware (MOM?) for a 3^rd party vendor product. The database has evolved over many years and has all of the quirks that you might expect of such a system: primary keys that are implied but not made explicit with constraints, relationships that are implied but not explicit, columns that are not nullable in the business logic but are nullable in the tables, etc. I’m not knocking the vendor: I am keenly aware that keeping a multipurpose database absolutely clean of cruft is a tough challenge indeed. But it does make for some really tough integration points. My major challenge over the past week has been auditing. The vendor system makes very heavy use of sprocs and embeds the creation of the audit trail in those sprocs rather than using what, to my mind, is the more obvious approach of triggers.

Walking through an example, let’s say that my MOM or some other end user creates a new workorder record. The steps go like this:

Call the sproc to add the workorder—This itself calls a number of other sprocs with all kinds of logic in there based upon magic strings. Hairs standing up on the back of my neck!
Call the sproc to do the auditing
1. Get the name of the table you’re adding a record to. In this case it might be something like "WORKORDERS".
2. Query a table that has a listing of specific columns in tables to keep an audit trail on. So the query might look something like this: select a.auditcolumn from auditable a where a.tablename = 'WORKORDERS'. The list of resulting auditcolumns tells us what values from our workorder records we have to add in to the audittrail table.
3. For each auditcolumn, get the before value and after value from the inserted or updated record and store them in the audittrail table.

The ugly part is that I have to explicitly call the sproc to do the auditing rather than leverage a trigger to do the work for me. And the really ugly part is that I need to avoid user the vendor’s sprocs and replicate the business logic in C# code in the MOM. There are some historical reasons for this that are not relevant to this discussion, but the point remains: because I have to avoid the vendor sprocs “business logic interface” and because the auditing is not handled by triggers, I’m faced with figuring out the auditing with my tool of choice, NHibernate. The selection of NHibernate was driven primarily by the need for my MOM to have transparent support for both Oracle and SQL Server. SubSonic appears to have some holes related to Oracle that my co-workers have found, and Entity Framework is too new for me to be building a cross platfrom product core on top of. I am concerned that some of the wackier edge cases like auditing and composite keys will trip me up.

In summary, I have a legacy database with a wonky auditing feature, I cannot use the current set of vendor sprocs, and I need to be absolutely sure that when my MOM is creating and updating records that I create the audit trail.

NHibernate Events and Aspect Oriented Programming

Of course this sounds like a perfect place to use some AOP jujitsu because the auditing function is cross cutting in nature, just like logging or security authorization. In other words, the act of creating a workorder or updating an employee record are tasks that a developer would need to understand and account for. But the auditing itself is orthogonal to the main task at hand and thus should be handled elsewhere in the code. Frameworks like Spring in the Java world and PostSharp for the .NET crowd handle all of the plumbing to enable AOP. But luckily, so does the container that NHibernate uses—the Castle project’s Windsor Container.

Here’s the basic idea of AOP then. I should be able to create a class or implement and interface or listen for an event in my application code and tell the AOP framework to catch that event and do something with it. One of the canonical examples for AOP is logging; rather than litter my code with Log.Debug("some message") or Log.Fatal("boom!") I would rather tell the AOP framework to listen for any method call and write a debug statement, or listen for any unhandled exception and send an email to the system administrator. The Windsor Container (I’ll just call it Windsor from here on out) can do exactly that for me, and the NHibernate developers have built extensibility points in to the persistence lifecycle to help me. Enter NHibernate events.

These NH events were added at v2.0 and are a set of interfaces that I can implement to do things like, oh, I don't know...auditing! For example, there is an interface called IPostInsertEventListener that I can implement. If I then tell NH that I want to use my implementation of IPostInsertEventListener by adding it during the configuration stage of NH startup, then my code in the OnPostInsert method will run every single time a record is inserted or updated in the database. As you might imagine, when I realized the consequences of having this capability at my fingertips, I got a little giddy! After all this turned out to be the magic bullet I was looking for to solve my crazy auditing problem. Seems like I should be able to run through the steps I described above in that OnPostInsert method and add my audit records using the before and after values. Now, I'm more of a look-at-someone-elses-implementation-and-modify-it kind of guy, so I trolled around on the Internet and found a couple of great resources beyond the standard NHibernate docs.

Ayende has a great blog post describing almost precisely what I need to do. He’s one of the committers on NH as well so I felt like his advice was pretty sound.
Rob Reynolds also read Ayende’s post and applied it specifically to an auditing event listener so I read his approach with great interest as well.

The pieces are all there. How to actually do it.

Having done all of the research I was able to whip up my cross-cutting auditing event handler in pretty short order. Here’s the complete code, with some of the vendor specific stuff ellided and obfuscated for non-disclosure reasons:

As you can see I capture every single insert or update event in my system and then look for the IAuditable marker interface that I created for that purpose. If I actually have an IAuditable class then I proceed to handle the specific cases. Right now I just need to do workorders but expanding this is trivially easy. You can see where I then use some attributes to map between the properties/fields in my class and the column names in the database. I did try to mine the NHibernate metadata to avoid this (you can read about my frustrations elsewhere) but ultimately went with a bit of duplication of metadata to avoid a much bigger headache in the long run. This is, after all, a consulting gig and I can only futz around for so long before I just have to get the darn thing working.

How to configure NHibernate to use this? Well I’m using Fluent NHibernate for this project and luckily for me Rob Reynolds had again beat me to it. I essentially copied his code and arrived at this to hook it all together. Yep, just one extra line of code in my config setup :-)

Configuration cfg = GetDbConnectionConfiguration(Resources.CityworksDatasource);
  Fluently.Configure(cfg)
   .ExposeConfiguration(c =>
    c.EventListeners.PostInsertEventListeners = new IPostInsertEventListener[] { new AuditEventListener() })
   .Mappings(m => m.FluentMappings
    .Add()
    ... // Lots more mappings
    .Add()
   .BuildConfiguration();
  return cfg;

So there you have it, a very clean way to leverage the Windsor/NHibernate AOP extensibility point to make a messy auditing story a lot easier to swallow.

I love unit test results that are green

2009-11-24T09:54:00.000-08:00

I've been working on a fairly complex legacy database integration using NHibernate. Several "implied" (a.k.a. missing) relationships, composite keys, magic strings where "B" means projected and empty means...not sure actually. So my unit tests are really a life saver. Every time I add a feature I have the benefit of knowing that I'm getting closer without doing things that are going to bite me later. Yes, my tests could be poorly written in some cases, yes I'm hitting the database way more than I should, but I'll take it as the price for my TDD comfort blanket.

Bad Explanation

2009-11-20T14:42:00.000-08:00

I have officially discovered the worst slide describing REST. Not because the explanation itself is particularly poor. But the visual noise is just too much to bear. I actually stopped watching the presentation immediately. Here's the offending slide at 0:21

Visual Studio theme - Ruby Blue knockoff

2009-11-16T16:07:00.000-08:00

In the “can’t possible face another line of code” category for today I spent a few minutes changing colors in Visual Studio. It's not exactly rocket science but sometimes you just need to do something different to refresh the brain cells. I kind of like this darker-but-not-black look to reduce eye strain. It's a total knockoff of John Long's TextMate theme. Download the theme from GitHub and using your Import/Export Settings option to pull them in. I just exported the colors and fonts so you should not see any other settings change.

UPDATE: And you can probably tell that I like the Monaco font for my fixed-width viewing pleasure.

Circuit Breaker of the non-electrical variety

2009-11-13T19:19:00.000-08:00

You all know about circuit breakers. They protect electrical networks and devices from surges by opening up when the bad stuff happens. Basically they are a binary toggle: a circuit breaker that is closed ensures that electricity can flow unimpeded through a medium like good old copper wires. A circuit breaker that is open...well...doesn't. It protects our expensive gadgets like computers and toasters from spikes in the supply. I may work for an engineering company, but that just exhausted my knowledge of what an actual, physical circuit breaker does.

But what about protecting software from volatile resources in their own systems? I can't remember how many times I've assumed that a website was running when attempting to call it, or how many #*(#%$#@! WCF services I've depended on in a "loosely coupled" system for critical things like, oh I don't know...security! I guess my loosely coupled systems are still prone to massive failures when some innocuous part decides to puke. I'm sure you've had this happen before: a database that didn't work, an file transfer that was just too slow. Enter the Circuit Breaker.

Release It!

In most excellent book Release It! Michael T. Nygard presents many approaches to deploy fault tolerant software into the wild. We all know how easy it is to build software. We all know that it can bring a business to it's knees when said software doesn't work right. Personally I'm thinking about an embarrassing tool I wrote that crashed and burned when access to an FTP site wasn't there. Totally missed it during coding, got bit in the ass when a deployed system failed spectacularly. No details necessary. Suffice it say that shame caused me to seek out ways to make my deliverable more tolerant of the daily hiccups we all identify with. But Nygard's book is a real revelation me and many others in the pragmatic advice he gives to protect systems against just such failures. And of course, one "pattern" he advocates is the use of Circuit Breakers (big C and B).

It goes something like this:

Software calls an unreliable service like "Bob's automated electronic knife sharpening service". For some unfathomable reason this works the first time around. All good.
The next time we call the service it times out. Oh no! Disaster! Except we handle the error nicely and increment our "screw up" counter by one.
We try again (darn user just keeps clicking the "sharpen my knife on the internet button") and the thing fails again. No problem - we again catch the error and tell the use "Whoops!" and increment the "screw up" counter again. We now have two service failures. We have also configured our system to stop the madness if a service fails three times in a row.
Of course, the user clicks the button again immediately, thus ignoring all of our strongly worded advice that "it ain't going to work". Our "screw up" counter hits three and guess what? The circuit trips. Somewhere in our calling code we decide that enough is enough and we should just open the circuit and stop these requests to the unreliable service for a while. We hope beyond hope that just waiting a while, like maybe 30 seconds in a web application, will give Bob's Knife Sharpening Service time to recover.
Wouldn't you know it...the user still presses the darn button. But this time we stop the madness and immediately and instantly report back that the circuit is open and request cannot go through. Or a popup saying "Knife sharpening is busy...try again in a minute or two". Or we disable the offending button. Whatever we present to the user, the important part is that a timer is fired up when the circuit trips open and no amount of button clicking will induce our cleverly protected system to call the unstable service until the timeout period has expired.

Do you get the idea? We are essentially trying to stop a bad situation from getting worse. We all know that when a remote service is down, the last thing we should do is to just keep right on calling it. We should give it a chance to catch it's breath. That's what the Circuit Breaker gives us.

C# Circuit Breaker, Anyone?

I've been burned enough time that I wanted to implement CB on my current project. Hey, who am I to ignore 100 years of sound electrical advice, let alone 5 year old software engineering wisdom? I found some really excellent implementations that all approach the problem from slight different angles. In particular the canonical (or first, at least) version came from Tim Ross. This was followed up earlier in 2009 by the prolific Davy Brion in a great blog post that laid it all out there. Davy uses the Castle Windsor inversion of control container to do some fancy aspect oriented stuff using interceptors (and if you understand that sentence they I think we would enjoy drinking a beer together). For my current project that is all a bit too much, plus I'm using StructureMap, so I made a couple of very minor tweaks to his code to get up and running. Here's the implementation. Notice how the CB used a state pattern and failure counter but is really all about invoking an Action.

In my heartbeat service (health monitor for a web site) I do something very, very simple and test resources. The crux of the whole thing is to wrap that call to the potentially unreliable remote resource inside the circuit breaker:

I've included a couple of unit tests so that you can see how it all ties together

Summary

The circuit breaker protects a system from potentially unreliable dependencies. It does not bury the errors but actually let's the calling application still be fully aware of the exceptions. But it does draw a line in the sand by way of a failure threshold and stops an excessive number of calls to the bad resource. That is the essence of controlled failure. Let the thing break, but don't let the break cause a cascading meltdown to your carefully crafted calling system. Handle the unpredictability by opening the circuit.

Programming Opinions

2009-11-12T20:09:00.001-08:00

Few things start flame wars faster than an open ended question addressed to a bunch of programmers. There's a really great question on Stack Overflow that appears to be breaking the mold on that front. The contention is that most developers will have a controversial opinion or two up their sleeve. As it happens, many of the "controversial" opinions so far seem to be quite sensible. A couple of my favorites:

If you only know one language, no matter how well you know it, you're not a great programmer.
There seems to be an attitude that says once you're really good at C# or Java or whatever other language you started out learning then that's all you need. I don't believe it- every language I have ever learned has taught me something new about programming that I have been able to bring back into my work with all the others. I think that anyone who restricts themselves to one language will never be as good as they could be.
It also indicates to me a certain lack of inquisitiveness and willingness to experiment that doesn't necessarily tally with the qualities I would expect to find in a really good programmer.
My Take
Amen! I was just telling Kirk that the only reason my C# is doing as well as it is (I'm modest too) is that I read a lot of Ruby and Python recently. Lambdas and other functional bits like list comprehensions and map/reduce have deepened my understanding of LINQ and the whole Func/Action thing. "I'm a .NET Developer" to me sounds a bit silly. How about "I've done a lot of .NET work but give me a web connection and some good source code to look at and I could do the project in [INSERT LANGUAGE HERE]."
If you're a developer, you should be able to write code
I did quite a bit of interviewing last year, and for my part of the interview I was supposed to test the way people thought, and how they implemented simple-to-moderate algorithms on a white board. I'd initially started out with questions like:
Given that Pi can be estimated using the function 4 * (1 - 1/3 + 1/5 - 1/7 + ...) with more terms giving greater accuracy, write a function that calculates Pi to an accuracy of 5 decimal places.
It's a problem that should make you think, but shouldn't be out of reach to a seasoned developer (it can be answered in about 10 lines of C#). However, many of our (supposedly pre-screened by the agency) candidates couldn't even begin to answer it, or even explain how they might go about answering it. So after a while I started asking simpler questions like:
Given the area of a circle is given by Pi times the radius squared, write a function to calculate the area of a circle.
Amazingly, more than half the candidates couldn't write this function in any language (I can read most popular languages so I let them use any language of their choice, including pseudo-code). We had "C# developers" who could not write this function in C#.
I was surprised by this. I had always thought that developers should be able to write code. It seems that, nowadays, this is a controversial opinion. Certainly it is amongst interview candidates!
My Take
Jesus!!! I would have crashed and burned on that one. My math is rudimentary and based upon Google searches. I even bought a beginners guide to pre-calculus recently. Yes, we need to be able to write code, but lets stick to realistic problems. I'm much more likely to ask a question like the FizzBuzz one, or ask a candidate to walk me through pseudo code for an elevator control system. If they can actually THINK through that design exercise then I can teach them how to code it.

So yeah, that's a really thought provoking question that has spawing lots of interesting discussion. Have a troll through the responses and let me know what you think is "controversial" and what is just common sense.

Rake for .NET Projects

2009-11-04T21:22:00.001-08:00

I'm working on a .NET project right now that involves 3 (or is it 4?) developers on two different teams in 3 different locations. The usual dance is being played with different dev settings for each person, like "use this GIS server for these layers" and "use this Oracle TNS for that database, but SQL Server for the other one". Not to mention the need to update a bunch of JSON configuration files depending on who is building the project. So I thought I'd take a look at the Rake build system that is written in Ruby. Actually, because I know there would be a revolution if the developers had to install Ruby just to update some files, I went with IronRuby instead. This is the version of Ruby that is being built to run on the .NET Framework. The beauty of it (apart from a slow startup time) is the speed of execution, and an xcopy deployment for any machine that already has .NET installed. Which is every Windows PC out there.

So how does this work? Well, I first downloaded IronRuby (get the ZIP file not the MSI installer) and unzipped it into the trunk\tools directory of my project. I followed the excellent advice of Evan Closson and created a little batch file that sets up my Ruby environment such that commands like iirb and irake actually cause the expected thing to happen. Ignore the stuff about 7-zip for now. I added that later and will explain.

After getting what amounts to a zero-installer, xcopy deployment of the entire Ruby language running in 3 minutes, we then need to get Rake. Ruby has a wonderful package management system called Gems. By typing igem install rake (don't miss the "i" there...that's IronRuby you're running!), Rake is downloaded and installed in to my trunk\tools\ironruby\lib\ruby\gems\1.8\gems\rake-0.8.7 directory. I had unzipped my IronRuby download in to the ironruby directory, and the rest was done for me. Test thinks out by typing irake. It'll barf at you because it's got nothing to do, but at least you now have the environment all squared away.

Next I compressed my IronRuby directory, complete with Rake, into a 7-zip file called ironruby.7z and then deleted my ironruby directory. The resulting file is about 2MB is size and suitable to put in subversion. Large files need not apply! I also have the 7-zip executable in my trunk\tools directory so that my init.bat file can uncompress it on demand when a new developer downloads the code for the first time. (note: if you're not using 7-zip yet then drop everything, go and install it, then carry on reading).

That leaves us with this directory structure (omitted the irrelevant parts). You can see how the JSON resources that I want to update per user are in the resource directory of my websites, and that the ConnectionStrings.config.template is at the top of the source directory. I link to this file from every project in the solution that needs access to databases so that the Web.config or App.config simply has in it and updating the one file automatically squares away any other application.

Rake

Big deal. I have Ruby and rake. So what? Again following Evan's advice I created a few files. Rake expects a file called rakefile.rb or just plain rakefile to live in the directory from which it is executed. In my case this means in my trunk directory. But I don't want to clutter things up so I also created a trunk\source\tasks directory to hold the two build scripts that really do the work: build.rb and rakefile.rb. First comes the rakefile:

As you might be able to tell, this is reading in some settings in the YAML (YAML ain't another markup language, I think) format. The yaml format is essentially the de facto Ruby serialization format, so by using yaml I get free conversion from a text file to a ruby object simply by calling YAML.load_file. Exactly the same as if I were to read a JSON file in to JavaScript - implicit understanding of the content. Here's a yaml file:

The structure is very simple and can be created in a text editor. Assuming the user starts the build process with irake DEPLOY=production then the production values are used. If they enter irake DEPLOY=development or just irake without a parameter then the dev settings are used. Amazing! Astounding! Just so much better than msbuild that I've struggled with.

Full disclosure: I'm not just spouting off here. I've written a few fairly ambitious and complex build scripts using msbuild. It works, it's powerful, and it's a bugger to use and learn. This rake business, on the other hand, is the first time I've ever touched Ruby and I'm not looking back. So there.

OK, we've got our configs. Now what do we do with them? My use case is simple: I need to update a boat load of JSON files and a single web.config file with some fresh server names, connection strings, and the like. Here's what build.rb looks like:

Rake is geared towards completing tasks. It is a build system, after all. First thing to notice is the default task. Just like every other system out there like make, ant, nant, msbuild, bake, psake, etc., it can have dependencies. So when I run irake it looks for a rakefile, then looks for a task called default. Finding that it then runs update_user_configs which in turn runs create_user_files which... It's obviously turtles all the way down from there - just keep following the dependency trail.

You can, dear programmer, probably figure out the rest of what is happening because Ruby is pretty expressive and clear (mostly!). For example, notice the 'require rexml/document'. REXML is the core Ruby XML processing library and I use that in the update_connection_strings task on line 84. If you've written any .NET applications you can tell that I'm opening up app.config or web.config files in the solution and doing a search and replace for a series of database connection strings. Look back at the YAML to refresh your memory. It's simple and I can literally cut'n'paste this task in to any one of my other .NET projects that need to have developer-specific connection strings in them.

Summary

I'm getting tired of typing here so I'll summarize briefly. IronRuby is a version of Ruby that is written in C# and runs on the CLR. It can use standard Ruby plugins or Gems such as Rake. Rake is a build system that uses regular Ruby code and files instead of XML to make writing build scripts less painful. I'm sure I've missed some key details in describing my setup. Let me know if you're confused by any steps and I'll do my best to update this post.

StructureMap Registry

2009-10-16T09:48:00.000-07:00

This is more for my information but feel free to look. I have centralized all object dependencies in my current project and have settled on the StructureMap IoC container as the coordinator. It took me a while to figure this out because I have two databases configured with NHibernate and depending on which repository I'm using I need to point to different NHibernate ISessionFactory instances. This was getting complicated so I created a simple unit of work class to manage the details, and was left with a much cleaner config. Admittedly this may look complicated to the uninitiated, but it's a lot easier that spreading all this stuff throughout my entire code base.

As usual, if you can't see the code it's because I'm lazy and I'm using github. You have to open the actual blog page for the good stuff :-)

SQL Server and firewalls

2009-10-15T20:18:00.000-07:00

I keep forgetting to open up port 1433 to access SQL Server 2008 on the network. Did a little searching and found this post that shows how to use the netsh command to do it all in a script. For example:

netsh firewall set portopening TCP 1433 "SQLServer"

Much better than going through the Windows Firewall user interface because you can set a bunch of exceptions in one script.

RegEx to the rescue: finding “between” matches

2009-10-14T12:45:00.001-07:00

I grabbed the source code of a web page to extract a bunch of train station locations in Chicago. After a little bit of vim’ing I had cleaned up the majority of the guff that I didn’t need. But I had hundreds of lines of Google Maps JavaScript to parse out so that I could get a CSV file to map my station locations. The problem was some goofy HTML mixed in with the JavaScript. I needed everything except the text between the colon after the station name and the two or three character station identifier.

41.7266667, -87.5477778, 93rd St. (South Chicago) Station:</dt><dd>93rd St. near Baltimore St.</dd><dd>Chicago,IL</dd><ul><li class=\"line me
41.7055556, -87.6558333,103rd St. - Washington Hts.  Station:</dt><dd>10335 S. Vincennes Ave.</dd><dd>Chicago,IL</dd><ul><li class=\"line ri
41.7069444, -87.6072222,103rd Street (Rosemoor) Station:</dt><dd>103th St. & Cottage Grove Ave.</dd><dd>Chicago,IL</dd><ul><li class=\"line me

So, I cracked open RegexBuddy and started to work. I really don’t get regex at all in spite of many false starts, but RegexBuddy makes life a little easier. After looking at this great answer on StackOverflow and working in RegexBuddy I was able to quite easily figure out the solution:

I love that product!

NHibernate HQL Queries I always forget

2009-10-12T16:24:00.001-07:00

The easy one is the first one - counting rows in a table. Stupid simple but I always forget the UniqueResult bit:

And the not-so-obvious but equally useful way to query an object with a composite primary key. You just pass an anonymous instance of the class with the primary key fields set:

Windows Services the Easy Way

2009-10-04T18:40:00.001-07:00

For a long time now I’ve been a big fan of creating Windows service projects in .NET using console applications rather than the canned Visual Studio Service Project template. My primary reason initially was to make sure I could debug the service application easily by setting a command line switch rather than monkeying around with attempting to debug a live Windows service. So my typical code would look something like this:

Notice how the first check is to see whether the command line contains a switch called –console? That is something I set up in the DEBUG properties of the Visual Studio project. So whenever I launch the executable during development it acts just like a console app, and I can in fact use that in production as well for testing purposes (-CONSOLE spits out a lot more logging information). If the –console switch isn’t present then the service class—GeneratorHost in this case—starts as a service as usual.

TopShelf

But I’ve been looking at TopShelf recently and prefer what I see here as a cleaner approach for testable code. It came from the MassTransit open source .NET service bus project as a way to quickly create service hosts for the large number publishers and subscribers involved in a typical MassTransit implementation, and was then refactored out in to its own project that has, interestingly enough, been coopted by Udi Dahan in his own nServiceBus project.

I came across TopShelf mostly because I’ve been pleased with how the MassTransit and nServiceBus approach to distributed systems is shaping up on my current project. I’m using a hand-rolled message queue with a database backend (NHibernate for Oracle, SQL Server, and SQLite support) and making all of my information exchanges be self contained messages that are simply added to the message queue for further processing. The WCF services for web service endpoints and other backend processing code are all hosted as Windows services for ease of deployment and configuration (app.config rather than messing with IIS 6.0). And the nice thing about TopShelf is that it support the IoC container that I’m using these days—StructureMap—and makes debugging and installation/deinstallation a piece of cake. Not to mention XML config of the user account for the service to run under which is a bonus when dealing with ArcGIS Server direct connections.

So what does it look like? Simply put, there is a fluent configuration API that talks to StructureMap and dynamically builds a Windows service for you without tinkering with ServiceInstaller classes and the like. Here’s an example from my current project where the main purpose of the Windows service is simply to serve as a host to a WCF HTTP service that in turn runs some ArcObjects bits:

I got a bit turned around with the StructureMap / Microsoft.Practices.ServiceLocation facade but the TopShelf source code has a good example of how to implement this. In fact, it was a cut’n’paste job:

Debug, Install, Uninstall

The truly great thing about TopShelf, though, is using it in production. Putting the configuration in a console application means that by default running the EXE such as GisDirectQueryHost.exe runs the “service” without having a Windows service. Running GisDirectQueryHost.exe /install installs the service and (you saw this one coming) /uninstall removes it. But in a real hat-tip to flexibility you can install multiple separate instances of the service by adding /install – instance InstanceName. Now that’s just plain cool. Thank you Kevin Miller for pointing this out.

Transforming ArcGIS Server REST JSON to NetTopologySuite

2009-10-03T17:17:00.001-07:00

Another quickie: I just wrote some code to get results from a Query against an ArcGIS Server REST server. I put a few classes together than represent fields, layers, and maps. Here the json output from a query (like this) is being passed to my helper function along with a list of fields that I’m interested in keeping, plus the geometry type (like esriPolygon). The wonderful part about this code is the way that Json.NET assist me with it’s great LINQ implementation and fast forward-only JsonTextReader. In the method below I’m using the JsonTextReader.

In the next example I’m ripping through the JSON result with a few lines of LINQ and getting fully built NetTopologySuite geometries that I can convert to GML, WKT, or another useful format. Or simply keep them as-is for analysis purposes.

Well, I haven't the polygon stuff yet (blush!) but you get the picture. Getting usable polylines from the JSON is literally two LINQ queries and one foreach . The simplicity is just a thing of beauty. Oh, and props to this guy for helping past a hurdle.

Converting ESRI geometries to GeoAPI geometries

2009-10-03T17:02:00.001-07:00

a.k.a. making ESRI data useful.

I need to do a little lightweight processing on some spatial data. The data is coming from an ESRI geodatabase in the form a IGeometry objects. One of the things I need to do is figure out the “middle” of points, lines, and polygons. Points and polygons are easy: the centroid. Lines are a little more problematic because what I want the “middle” point of a line to be is 50% the way along the length of the line. This is a pain using ArcObjects (ok, not impossible but still annoyingly opaque to my lame ArcObjects skills) and I know how to do it using NetTopologySuite. So I found a nice little utility in an ESRI forums post (that I can no longer find) that shows how to convert ESRI IGeometries to GeoAPI geometries. GeoAPI, as you may know, is the format that NTS uses.

I’ve posted my slightly modified version of the Converter I found on Google Code, and here is how I’m using the class in a simple ArcGIS Server web service:

You can see the usage of NetTopologySuite’s LengthIndexedLine to do a simple bit of linear referencing based upon distance. Simple yes, effective, yes.

Open source and Microsoft culture

2009-08-31T22:44:00.000-07:00

I saw this blog post about the difference between the treatment of open source in the Java and Microsoft communities. I must say I see this a lot and while I see the value of the support provided by Microsoft, I really have been amazed at the degree to which the Java folks just don't even consider NOT using open source. The Oracle WebLogic Server project I'm looking at now has all kinds of open source software.

I guess the thing I've realized is that even when something like Hibernate or Apache or Eclipse is released under an open source license, some of the very biggest players in the technology world have highly skilled software engineers working on the code. Not because they want to give this stuff away, but because it gives them a competitive advantage in a world where the tool you use is less important than how you use that tool.

Of course this was prompted by me having to write a network trace solver using ArcObjects rather than something simpler like...say...SQL! 1,000 lines of code later and I know know which valves isolate the break in a water main. Ah, the joy of closed source! :-)

When you expect an exception in NUnit 2.5

2009-08-12T10:01:00.001-07:00

I keep forgetting this and it's a hassle to find an example. I have a piece of code that I know throws an exception during testing, but I need to then do some work after the exception has been thrown to make sure that my system is in a consistent state. Here's how I'm running some code to do the setup and throwing the actual exception. And then there's the part that does my second round of checking. Obviously in this case I don't want to see data in the database if an exception happening before my transaction was committed.

Event Driven Architecture

2009-08-03T21:18:00.001-07:00

Really interesting ideas that reflect my own current thinking on system architecture. Mostly because I've been looking at this guy's work and Udi Dahan's (The Software Simplist and author of NServiceBus).

Event Driven Architecture

View more presentations from Chris Patterson.

Using Python and Oracle with cx_Python

2009-05-20T06:44:00.001-07:00

I needed to process records in a table with 14.4 million records the other day. The database is on my laptop so performance on a table that large sucks – no matter how many indexes you throw at the thing, it’s still all in one data file on one disk and has only 1GB of RAM to work with. My friendly DBA helped me to eliminate the big bottlenecks that I could remove on a laptop, namely to increase the amount of memory assigned to client side SQL manipulation. Other than that it still crawled and then I realized that the Java functions I am calling in the update statements are the slow point, not Oracle itself. See, for each row I am doing a bunch of string manipulation related to turned crummy address data in to nice clean US Postal Service standard addresses, and that takes Java some time; not much time but enough to build up over 14.4 million iterations. And what eventually dawned on me was the JVM itself was getting bogged down and there wasn’t much I could do about it.

Or was there? Again with the help of my DBA, the approach taken was to batch the rows by some arbitrary subset to reduce the load on the JVM and give it a chance to catch up. I chose rip through a ZIP codes’ worth of data for each batch and those ZIPs live in a text file. Could I have written this in PL/SQL? Maybe, but I don’t know it well enough to be productive quickly so instead I reached for my new scripting tool of choice that seems to get this type of job done quickly: Python. I got the approved Oracle connection bits called cx_Oracle and installed them locally. I have a full Oracle install including TNS listener and database server so I didn’t have to mess with Oracle Instant Client, so here is the sum total of my code:

As you can see from the __main__ method at the bottom, I read in the ZIPs from a text file and eliminate duplicates, then run a couple of update statements using the current ZIP to filter my working set. Look carefully at the update statements and you can see the Java functions being called. Again, I’m sure this could easily be done in PL/SQL if you know how but I’m recording this approach here for posterity. It works. That’s good enough for me.

Writing Documentation Using reStructured Text

2009-05-06T15:17:00.001-07:00

I’ve been working on some documentation for a web application this week. I’m not a big proponent of doing this in Microsoft Word because the docs need to be in both HTML and PDF formats for the client. Plus Word tends to get pretty squirrely when it comes to precise formatting and doing what I want vs. what it thinks I want. Think about the need to do code listings: in Word I can define a style that gives it the right font, the right indentation, and all the other stuff to make a code listing just like the examples I use in this blog. In HTML I do something similar but use CSS and the <pre> tag, with similar results. Of course the issue is that Word all too frequently mixes up the format of the surrounding paragraph and elements with my style. Just remember for a second how many times you’ve had to mess about with the indentation of the second paragraph of text in a bullet? Too many.

Of course, I also don’t really want to write a boat load of documentation in HTML – I’ve done it quite recently and it’s not fun, even with a great HTML editor to lean on. What to do? Well, I looked on the internet tubes for possible answers to my requirements:

I want to have output in HTML and PDF
It needs to support version control in a friendly way, so text-based files are a bonus and binary files are a no-no
Non-technical (non-developer) people should be able to write the help files without needing more than a little bit of training and a text editor.
It needs to be free - Adobe's RoboHelp does this stuff but it costs $999. Obviously not going to work for me no matter how cool it is.

I quickly came across one documentation tool that does what I need: the Sphinx Python documentation generator. “Huh?” I hear you say, “Python document generator…how the heck does that fit your list of requirements?” Well, the interesting thing about Sphinx is that it uses a very simple text format called reStructured Text (reST) that lets you write simple markup that mostly looks like simple text without a ton of angle brackets (like hello). Sphinx then runs a process using the Python docutils utilities that turns the reST in to other formats like HTML and LaTeX. Yes, the whole purpose of Sphinx is to convert reST into exactly the formats I need! From the reST page:

“Docutils is an open-source text processing system for processing plaintext documentation into useful formats, such as HTML or LaTeX. It includes reStructuredText, the easy to read, easy to use, what-you-see-is-what-you-get plaintext markup language.”

So it turns out that docutils and Sphinx come from a Python heritage and are implemented using Python, but they are generic tools that can be used to generate any kind of documentation you want using reST. Which sounds like exactly what I’m looking for.

reStructured Text

Rather than rehash all of the bells and whistles in reST I thought I’d just post an example:

============
Introduction
============

The :abbr:`EMMA (Election Map Maintenance Application)` application enables election officials at multiple levels of government stay on top of changes in electoral boundaries. Specifically it is designed to support changes to precinct boundaries across the entire state of Ohio. Election officials at the county level can propose modifications to precincts that adhere to U.S. Census and |ohsos| rules, and then |ohsos| staff can take those modifications through an electronic approval process. The benefit of EMMA is the easy management of what can quickly become complex changes, and a good historical record of all proposed changes whether or not they eventually are accepted.

.. warning:: This is a warning.

.. note:: This is a note.

.. sidebar:: A sidebar

    with some content

Before diving in to the detailed workflow guides or specific tool descriptions, take a look at the glossary to make sure you understand the various terms that are used throughout this documentation. While many of them, such as *precinct*, may be familiar to you, others such as :term:`VTD` and :term:`TIGER` may not.

You can see that a couple of simple conventions like indentation and blank lines are used to give the document its structure. Also the main heading just has some lines around it to make it stand out. What else?

Directives — see the .. warning:: and the .. sidebar::? Those are simple tags that represent semantic parts of the document. I don't even really need to describe what they are.

:term:`VTD` — This is a reference to a definition. Elsewhere in my reST document(s) I have a .. glossary:: directive which has the description of what VTD actually is. reST is smart enough to link the two.

But the really great part is the reST is semantic – it describes the content and structure of my document in a way that HTML and Word should but often don’t. Way too much time worrying about the colors, the indentations, and all that other guff.

Generating Output

Yeah, yeah, so semantic content and structure are great concepts, but how does that help me achieve my goal. Well Sphinx has utilities to take the reST and output in a variety of formats. Actually I think the only two widely used formats are HTML and LaTeX. If you haven’t come across LaTeX before then you owe it to yourself to do a little research. The short version is that it is also a text-based way to describe typeset quality documents (think magazines, journals, documentation) plus tools to take those LaTeX documents and convert them to formats like PDF. Aha! Now you’re getting the point here. In a nutshell:

Write reST documents in simple markup

→ Run reST files through Sphinx

→ Sphinx produces HTML or LaTeX

→ For LaTeX, turn the TEX file in to PDF

The “Run reST files through Sphinx” part is straightforward. In fact Sphinx has a handy command to run that even creates a little batch file to help you do it:

make html
make latex

As long as Python is installed with the Sphinx and docutils bits, that is all you need to do for HTML. For LaTeX → PDF conversion you also need to have a LaTeX processor. I am using a distribution called MiKTeX that runs very well on Windows and gives me another simple one liner:

pdflatext SphinxOutput.tex

The result is a very professional looking version of the documentation in PDF. The great part is that the glossary of terms is still all hyperlinked within the document, as are the table of contents and index. Sweet.

The PDF output for the glossary. In the HTML this is a Definition List.

Here’s the proof that references and content structure turns in to something useful in the PDF

A Bit More Detail on Requirements

Because it took a little preparation to get up and running, I’ll share a quick README that I wrote myself so that I can remember how this works. Here you go:

Setup
=====
Building this documentation requires the following to be installed
on your system:

* Python 2.5.4
    * download from http://www.python.org/download/
    * Add the C:\Python25 directory to your Windows PATH
* EasyInstall
    * run setuptools-0.6c9.win32-py2.5.exe found in the same
      directory as this readme file
    * Add C:\Python25\Scripts to your Windows PATH
* Sphinx documentation tools
    * Open a command prompt and go to Project\docs\manual
    * Type 'easy_install Sphinx' from a command prompt, making sure
      that you have an active Internet connection to download dependencies
* MikTex - LaTeX/PDF document generator
    * Download the Basic MiKTeX 2.7 installer from http://miktex.org/2.7/setup

Building the documentation:
===========================
* Open a command prompt and cd to Project\doc\manual
* HTML version
    * type 'make html'
    * look in manual\build\html
* PDF version
    * type 'make latex'
    * cd build\latex
    * type 'pdflatex mydocs.tex' - NOTE: If you are asked to download additional
      components during the first run then say YES! MiKTeX needs some extra bits
      to support the features required by the Sphinx PDF documentation style.
    * look for the mydocs.pdf
    * If the Table of Context is not showing up in the PDF simply rerun the pdflatex
      command. That usually fixes it.

If you need to clean up and start from scratch:
* type 'make clean'

Gotchas

Of course the world is not perfect. Configuring the documentation build happens in a python file called conf.py. I have yet to see much of how the LaTeX output can be configured beyond a handful of modifications. This is turn means that the PDF layout is largely beyond my control unless I want to get my hands dirty with the tex file itself. Which I don’t! Also, to change the layout of the HTML beyond the simple cosmetics would mean getting in to the Sphinx templating approach (which is similar to Django templates but not worth it for my current project).

Talking to Oracle with Ant

2009-05-04T21:08:00.001-07:00

I’m working on my first Java project this month and wanted to automate some PL/SQL scripts. They do a bunch of setup and teardown on a local Oracle instance to get it ready for work. I’ll soon be adding a simpler task for pushing some disposable test data in to the same database, but for now I’ll keep it simple. Anyway, on .NET projects I like to use msbuild to get my initial post-subversion checkout environment all squared away. Here’s my first Ant build script to handle a little Oracle. Of course you can see that it’s not exactly rocket science. A little Googling got me on the right path, and I feel immediately more at home with this format than I did when starting MSBuild. I guess maybe I should have tried NAnt!

<?xml version="1.0" encoding="windows-1252" ?>
<project default="init" basedir="../../">

    <!-- Assumes the existence of a build properties file -->
    <property file="${basedir}/override.properties"/>
    <property file="${basedir}/build.properties"/>
           
    <target name="init"></target>
    
    <!-- Rebuild derived (non-census) data from scratch -->
    <target name="tiger.db.up" depends="tiger.db.down">
        <exec executable="${db.sqlplus}" dir="${project.persistence}/sql/tiger">
            <arg line="${db.sqlplus.sys} @doall.sql"/>
        </exec>
    </target>
    
     <!-- Drop any objects not coming directly from census, i.e., derived data -->
    <target name="tiger.db.down">
        <exec executable="${db.sqlplus}" dir="${project.persistence}/sql/tiger">
            <arg line="${db.sqlplus.sys} @dropall.sql"/>
        </exec>
    </target>
    

</project>

The basedir property simply points to a directory two levels up. Why? Because I have a few standard properties that developers will need in order for the build to work. The neat part is that they can also create an override.properties file to, well, override (!) any of the defaults that they want to, e.g., the name of the Oracle instance, a username, whatever. Note that the references to the property files in my example above are in a particular order; properties in Ant are immutable so once I’ve defined it in my override file, it cannot then be redefined in the build.properties file even if it’s listed there. Nice and simple. Here’s the build.properties and my override.

<pre>#If you need to override any of these properties, create an 
# override.properties file in the same folder and make sure
# that it is imported FIRST in any build scripts.

# Oracle server and connection info
# Note that the $ is escaped with a leading $, so $$$$ equates to $$
db.sys.password = pa$$$$w0rd
db.sqlplus = sqlplus
db.host = yadda.yadda.whatever.com
db.sid = yomamma
db.sqlplus.sys = sys/${db.sys.password} as sysdba@//${db.host}/${db.sid}

tiger.state.user = tiger2007_39
tiger.state.password = tiger2007_39

# Project directories
# base.dir is defined in each build script and describes
# the relationship between the build.xml location and
# the relative location of all of the project
# directories. For example, if an Ant file is located
# in SomeCommon\ant\build.xml then the base.dir value
# defined in that file might read "../../" meaning that
# the project directories are "two levels up from me"
some.common = ${basedir}/SomeCommon
some.persistence = ${basedir}/SomePersistence
some.security = ${basedir}/SomeSecurity

# WARNING: Make sure that you leave an EMPTY LINE at the end of this file.

And the override. Notice I just need to override a couple of properties: the rest are left untouched.

# Any values added here will override values
# that exist in build.properties. If a value
# is not present here the value from build.properties
# will be used by default.
db.sys.password = secret.word
db.host = my.pc.com
db.sid = oracle

On Visualization

2009-04-18T09:13:00.001-07:00

I'm rereading Genius: The Life and Science of Richard Feynman by James Gleick this week. Incidentally he’s written another one of my favorite books, Chaos: Making a New Science which was originally written in 1988 and just republished last year. If you enjoy popular science writing Chaos is a great read. Anyway, back to Genius.

One of the things I frequently find myself doing when I’m at the early stages of a software development project is staring at the wall for an hour at a time. Literally an hour plus. I read up on the “literature” which entails a ton of blog posts, maybe some best practices described by the software vendor or open source folks that who’s package I’m using, and then talking with my colleagues. After that research I try to internalize what goals the software has to achieve and then the wall-staring begins. If I were to try to describe what I “see”, it’s like a collection of boxes of different colors floating around and forming connections with each other. The connections have different thicknesses and sort of pulsate with light when I feel there’s a good link between two things, or when there are signals or messages between them. Finally, as things settle down and I get closer to a starting point (where I could write a little code) the boxes start to crystallize like the water-to-ice phase transition and a more rigid structure emerges. And then I crack open my coding tool and start typing frantically before I forget what I just thought out…only to discover it was crap and I begin the process again but with a slightly different set of beginning assumptions.

Can you believe what I do for a living? :-)

Anyway, I am always interested to hear how other people deal with this visualization issue for things that don’t really have a physical, visual representation. This morning I re-read a great quotation of Feynman, the Nobel winning theoretical physicist and bongo drum player as he attempts to describe his approach to visualizing the impossible:

“What I am really trying to do is being birth to clarity, which is really a half-assedly thought-out pictorial semi-vision thing. I would see the jiggle-jiggle or the wiggle of the path. Even now when I talk about the influence functional, I see the coupling and I take this turn—like as if there was a big bag of stuff—and try to collect it away and to push it. It’s all visual. It’s hard to explain.”

Chaos, p 244

Isn’t that just the best description ever of a visual thought process? It completely captures the indescribable nature of how we intuitive think about complex processes that really are difficult to describe. For a physicist it’s the way that you think about E- and B-fields and electron paths; for software developers it’s the admittedly much easier design process. The end result is something concrete: a set of equations or a class diagram which are blueprints for reproducing what’s going on inside our heads. But what I love about it is the leveling of the playing field because I think all of us go through the visualization process when we’re working on something that is hard to put down on paper until we have the final answer.

How to get the display name of a Windows login when you start from the login name

2009-04-17T16:45:00.001-07:00

I was surprised how complicated this is, but upon reflection it is just because I have never used LDAP or Active Directory before. Basically I have an ASP.NET Web Form and access to the currently-connected user, looking something like this: DOMAIN\doe32. I need to get the display name associated with that user, e.g., “John Doe”. Here’s one solution:

Utility Function for Creating Images on the Fly

2009-01-26T15:08:00.001-08:00

Really goofy, but I keep this handy for created files on the fly, especially when I’m testing system boundaries for files of various sizes. For example, in my current project I limited the size of an upload to 5MB – so I create a whopping big image on the fly then try to send it.

Compression using 7zip and .NET

2009-01-06T20:17:00.001-08:00

I needed to compress some data across the wire on a project. Basically sending very large files that, when compressed, get really small so choosing the right compression algorithm is important. I ran tests with SharpZipLib and the native .NET zip compression handlers with decent results. But I also use the 7-zip desktop application for my day to day usage instead of WinZip so I tried compressing the data files using that as well. Glad I did – the 7-zip compressed an additional 20% on top of what ZIP did. Well worth the effort to incorporate in to my WCF application.

I basically took the code wholesale from this post by Rob Reynolds and sprinkled in a little of my own meager magic. The solution revolves around a generic IZip interface and IZipFileUtility. Lot’s of comments:

The SevenZip implementation essentially calls the command line version of the 7-zip utility (7za.exe) with the appropriate parameters:

And of course the concrete implementation of the IFileZipUtility actually pulls it all together. The idea is that this could have a WinZip implementation as well if necessary so that there would be no non-Base Class Library dependencies. You can see this in the Subversion repository.

Here’s how it looks in a test case:

You can grab all of the source code from Google Code.

Parsing XML in a SQL Server Stored Procedure

2009-01-05T17:00:00.001-08:00

I had to send a variable number of columns and values to a stored procedure. So for one query I might need to do the equivalent of an IN query (all values in one search column), but in other cases I might need to do a search on a number of columns and values (WHERE a = 1 AND b = 2 OR…).

There are all kinds of ways of doing this, but for my application I opted to create a little XML snippet to send the data. Don’t ask me why except that it makes a lot of sense in the context of the code I need to execute.

Here’s the C# code where I’m calling my procedure:

using (var repository = new RepositoryBase(this._connectionStringName)) { IDbCommand command = new SqlCommand(storedProcedure) { CommandType = CommandType.StoredProcedure }; IDbDataParameter p = command.CreateParameter(); p.DbType = DbType.Xml; p.ParameterName = "?list"; p.Direction = ParameterDirection.Input; p.Value = auditTracking.GenerateCoreEditListAsXml().ToString(SaveOptions.None); command.Parameters.Add(p); DataSet edits = repository.ExecuteQuery(command, tableNames); affectedRecords = this.ParseResultsToAuditRecords(edits); }

Where the GenerateCoreEditListAsXml function sets the value of the p.Value parameter to something like this:

In regular SQL calling the procedure looks something like this:

use [CorridorField] declare @list xml; set @list = '<PKs> <row id="1"> <pk name="PATH_ID">10</pk> <pk name="ATTRIB_NAME">_Last Visit</pk> <pk name="ATTRIB_VALUE">20-Jun-2005</pk> </row> <row id="2"> <pk name="PATH_ID">10</pk> <pk name="ATTRIB_NAME">05. Service</pk> <pk name="ATTRIB_VALUE">Air Force</pk> </row> <row id="3"> <pk name="PATH_ID">12</pk> <pk name="ATTRIB_NAME">_Playground PA</pk> <pk name="ATTRIB_VALUE">Yes</pk> </row> </PKs>' DECLARE @RC int EXECUTE @RC = [CorridorField].[FCA].[uspAuditPath_Attrib] @list

The procedure I’m calling in turn has this inside it. Notice how I’m first transforming the XML parameter in to a temporary table? That is so I can use it as the filter on the WHERE portion of the main query using an IN statement:

set ANSI_NULLS ON set QUOTED_IDENTIFIER ON go /* Get all data associated with a batch of FUS_BLDGS records */ ALTER procedure [FCA].[uspAuditPath_Attrib](@list xml) as declare @results table(PKRow varchar(30), PKColumn varchar(30), PKValue varchar(30)) insert into @results select DataSource.keys.value('../@id','varchar(30)') as PKRow , DataSource.keys.value('@name','varchar(30)') as PKColumn , DataSource.keys.value('.','varchar(30)') as PKValue from @list.nodes('/PKs/row/pk') as DataSource(keys) select * from @results; -- paths select distinct(p.path_id) from [path] p where convert(varchar(30),path_id) in (select distinct(convert(varchar(30),pa.path_id)) from dbo.path_attrib as pa where convert(varchar(30),pa.path_id) in (select PKValue from @results))

So what do the results look like? Pretty boring actually. The purpose is to pass in multiple rows with multiple discriminating columns and use those to get records from a related table. In this case it’s kind of boring but I couldn’t think of a better or more flexible way of doing it. Maybe I’m just creating a rats nest of maintenance but it’ll work for now.

WiX and XML, Windows Services

2009-01-05T07:51:00.001-08:00

I was tripped up by a couple of WiX things this week while building an installation package for a Windows services project.

Windows Services

The first was getting the Windows service to be started correctly. I’ve done this before but couldn’t get it to work this time around. Here was my starting point:

I am using a 3rd party compression tool—7zip— to manage some aspects of my app so, being alphabetically inclined, I put that first on my list of files to include. The actual service is in the file FcaImportService.exe. Then I have my usual service setup and start bits in the same component like this:

As I mentioned I’ve used this exact approach before with success. But that darn service kept bombing when it was trying to start! It was a really easy fix though. Simply make sure that the first File in the Component is the one that holds the service implementation and WiX will then happily create the service while pointing to the correct executable. Just look in the Service Control Manager to see which file WiX is actually pointing at:

XML

Really annoying one. It turns out that the Windows installer database that is encapsulated in the *.MSI file treats square brackets as variable delimeters. So when you use a string that WiX is going to interpret and that string includes square brackets, you need to escape them. Naturally one example is in XPath. Here’s how it should look in WiX:

Hope that saves you a few minutes of head scratching.

jQuery and ASP.NET Validation Controls Disagree

2008-12-20T10:57:00.001-08:00

I spent the better part of a morning struggling to understand how ASP.NET Validation controls interact with non-ASP.NET elements in a page.

The Context

I have a number of ASP.NET server pages (aspx) that have one or more TextBox controls on them. Some of those text boxes need to have a popup calendar to allow a user to easily enter a valid date. I am using ASP.NET Validation server controls to inject so JavaScript to perform some simple user input checks like so:

The popup calendar uses jQuery’s UI project to provide a nice configurable widget without the overhead of a server control. Why use jQuery? Basically I wanted to simply add a CSS class to any TextBox to add a calendar rather than adding a bunch of server controls, one for each date field. So I have this at the top of the page to import the scripts:

The gui.js is my own little script that basically hooks up any TextBox with a class of datePick with the jQuery datepicker widget like so:

The Problem

This all works great in Firefox and Chrome, but bombs in IE7 and IE8 every time. Well, it doesn’t bomb unless I choose to debug, but there is definitely a JavaScript error. What I see is the calendar popping up as expected. I can choose a date, but then the calendar stays displayed instead of gracefully closing to leave my new date in the text field. Why?

After doing some Googling I found a great discussion on the jQuery discussion group about ASP.NET validation controls and how they interact with non-ASP.NET JavaScript. Basically the validation script is trying to fire every time the calendar widget calls it’s onSelect function, and because the calendar hasn’t yet put a value in the control the ASP.NET JavaScript errors out (thinks the length of the property array it’s checking is less than zero or just not there).

This means that I can either have a nice jQuery calendar or validation controls. Unless there’s a way around it all…

The Solution

Thanks entirely to the supportive jQuery user mt I got a great workaround. Simply put, you need to subclass (or in jQuery terms, $.extend) the datepicker and override it’s onSelect method by setting it to nothing. Having done that, the calendar popup no longer interferes with what the ASP.NET-generated JavaScript is trying to do. Now I can have the best of both worlds. Here’s the fix in my gui.js:

WCF Experimentation

2008-11-16T14:24:00.001-08:00

I’ve been working on a Windows Communication Foundation (WCF) service at work this past week or so, and it’s taught me a few things about structuring projects correctly, and supporting unit tests.

First, you can get the code here using your favorite Subversion client: http://code.google.com/p/dylansknowledgebase/source/browse/#svn/trunk/WcfReferenceProject

Key points:

Do not try to create a WCF service using the New Project wizard in Visual Studio and expect to have something flexible or testable! That’s probably the biggest lesson of the week. I learned this through my struggles with the out-of-the-box arrangement that Microsoft provides, and got all of the right answers from an excellent Dot Net Rocks TV episode by Miguel Castro.
Do not use any automated tools or cut’n’paste too much when working with the app.config or web.config for your services. I have a much better understanding of what the heck is going on by slowly and methodically adding my own settings and seeing what works and what doesn’t. It’s pretty painful at first but it pays off when you can quickly diagnose a problem that seems to be coming from nowhere, only to discover a simple fix in the system.serviceModel section of the file.
Don’t use a Add Service Reference option in Visual Studio to automatically create your client proxy to the WCF service. Instead use ClientBase<T> to create a strongly type proxy very easily in code. It’s less work, and let’s you know immediately if it no longer matches the server implementation. Plus, you can override the logic in your own client if you need to, e.g., adding user verification before uploading a large file. Look in ClientTests\HelloClient.cs for an example of this in action.

That’s it: pretty simple stuff, but it had a profound and positive impact on the way that I’m pursuing WCF.

Automated Builds

2008-11-12T16:44:00.001-08:00

For the past few months I’ve been ramping up on my software building skills. Having cut my teeth on the .NET Framework, my weapon of choice for writing code has long been Visual Studio. Consequently I’ve always considered pushing F5 to be the extent of building software. Of course now I’m a little wiser and know the difference between compiling it and building something useful. In fact, the biggest impetus for me to learn how to run a more elaborate build process was working in a distributed team and needing to keep in sync. Anyway, what do I mean by building anyway?

Reducing Friction

A build process is simply a repeatable way to take a bunch of inputs and finish with a desired set of outputs. Think of a typical web application. It probably has some pages, some resources like images, CSS, JavaScript files, etc. But it also usually has one or more configuration files, 3rd party components, a database or two, and possibly even some management tools. Now imagine having to join a team of developers who are already writing that web site, and being productive immediately. This is where the build process comes in. It should go something like this:

Open IDE
Connect to version control system and download the trunk (most recent buildable version of the project)
Read the Readme.txt file describing the build process.
Run the build script
Start working

Unfortunately my prior experience has been more along these lines:

Open IDE
Download the latest code
Press F5 and wonder what all the errors are
Ask the lead developer what I’m missing, and discover that there are three database I forgot to install, and a dependency for ASP.NET AJAX to XCOPY deploy first.
Do what is suggested, still get a bunch of errors. Discover that there are 5 configuration settings that work on Bob’s PC but not on mine so I have to change them.
Waste the rest of the day on this crap.
Waste most of the next day on similar issues.
Sometime towards the end of day three (I kid you not) I can actually start to write code and add value to the project.

Which do you think I prefer?

So the build process implies a couple of things: a build tool that actually does all of the little bits and pieces for me like compiling code, setting up databases, adding users to systems, etc., and also a build script that the build tool uses to do it’s thing. There are numerous build tools out there and they generally seem to be either code based, e.g., Rake uses a Ruby script, bake uses a Boo script, or heavy on the angle brackets, e.g., Ant (Java), Nant (.NET), and MSBuild (.NET) all rely on XML documents to describe the build process. Personally I really like the readability of rake (Ruby) scripts, and theoretically there’s no reason why I could use Rake on my .NET projects. However, I decided to tow the party line and use MSBuild for my first attempts and build scripts simply because is is guaranteed to be installed on every .NET developers machine: one less thing to worry about.

It doesn’t just happen

As I was reading all of the various ALT.NET and agile remonstrations for not having a build process, I wondered why on earth everyone didn’t just do this stuff in the first place. Then I tried to create a build script to do the things I need to do and had a very rude awakening. It turns out to be fairly tough to learn MSBuild in the first place. As with anything, the initial frustration level was high enough to make me curse Microsoft up and down, but once I got the hang of it I found it relatively painless (like maybe stubbing your toe badly as opposed to slicing a finger off). In other words, I still find it to be a bit of a chore to keep up with the build script as things evolve, but here’s the lesson:

Make sure your build script is one of the very first things you do on a fresh project. It’s a lot less painful to make small incremental steps than it is to figure it all out at the end of a project.

In fact, maintaining a running build has become so central to my last couple of projects for that very reason. Think about this: in order to set up my current project I need to…

Install 4 databases from scratch
Install one web application
Install two web service projects – one is mine, the other is a vendor solution
Create web proxies dynamically each time the code in the web services change
Update at least 10 config files
etc.
etc.

Things absolutely go wrong with the build, but the great thing is when I fix something, it always works in the future, and if it breaks again it’s because something has changed that I probably needed to know about in the first place. It’s kind of like test driven development but for your whole project, not just for one tiny piece of it.

Next time

In my next post I’ll describe what is happening in my latest build script and how it helps the team of 4 developers in 3 time zones and 4 cities to keep tightly in sync.

Getters and Setters

2008-11-11T14:39:00.001-08:00

I just burned about 30 minutes trying to figure out why my data transfer object (DTO) didn’t work. I have a C# Web Service (asmx) that accepts a simple class. There are two properties I care about:

A unique ID which is an integer
An array of another simple class with a few simple properties

No matter how hard I tried I could not get the array property to show up. I know I’ve done this a million times before, but my proxy class (created by the wsdl command line tool) just wouldn’t generate the damn thing.

Simple answer: I forgot to add a setter to the property, so while I could get the array, the serializer that .NET uses to XML-serialize the array could not be called. Rather than notify me of this, the wsdl command simply ignored the type when generating the proxy classes.

So there you have it. Make sure your DTOs have:

A parameterless constructor
Getters and setters on properties you need to use on the client

Capturing Edit Events in ArcMap

2008-10-23T20:41:00.001-07:00

Issue

You are trying to capture the Save Edits and Stop Editing toolbar actions in ArcMap through code so that different actions can be performed for each event. Unfortunately the underlying events associated with the actions (saving and stopping editing) fire for both actions so figuring out if the use Saved or Stopped Editing is a chore.

Resolution

Set up your code to listen for the events:

Then put in whatever code you need to run on the Save Edits action. In this case there is no special code apart from the magic part where a boolean field is being set to true. This basically lets the application know that “I just did a save action”.

With that field set to true, when the OnStopEditing event fires just afterwards, we can look at that field to see whether or not a Save just ran as well:

Note that we are always setting the _isSavedAction field to false after any special code has run, and our special code only runs if the field is already false in the first place.

The Trick

Turns out this approach works very well, but it does not capture edit events caused by background edits. By background edit I means something done in a cursor behind the scenes while an edit session is active, e.g., doing a little update cursor action in your own code while a regular ArcMap edit session is in progress. This is documented by ESRI but took me a while to figure out. Fortunately the answer is simple: force these kind of background, cursor driven events to fire like all the others:

Welcome to techno-world

2008-10-23T19:58:00.001-07:00

In the past I’ve posted a few articles related to my job as a software developer or more accurately a “technologist”. They have ended up on my other blog, much to the consternation of my family and non-geek friends (which is to say, most of them). So I’m finally biting the bullet and creating a corner of the web for my own meanderings in software developments trials, database tribulations, and generally anything sort of work related that baffles me.

In essence, then, this will be a memory dumping ground for all of the things that I keep forgetting shortly after having learned them. Similar to my collection of bookmarks, only with more context.

So don’t expect anything earth shattering here folks: just the musings of a barely competent propeller head trying to stay gainfully employed while continually getting in to trouble.

By way of closing, I suppose I should warn you that you are likely to read about these topics here:

.NET development with a focus on C#
Software configuration management, particularly version control and automated builds.
SQL Server and Oracle bits appropriate to a developer (I’m categorically not a DBA so I’m likely to get it all wrong)
Spatial stuff. No, not outer space but GIS and mapping kind of spatial.

Dylan Does Digits

Blog is moving

Parameterized Query with LIKE using Oracle.DataAccess

Port blocking in Windows 7 makes for unhappy WCF testing

Ruby strings on Windows. Backlashes again!

A little coding test

Do this without compiling...

And see if it works when you finally build it

It only took me 3 hours to get Rails running on Ubuntu

Oracle Spatial SIG group

Lessons learned on a project...time to 'fess up

Markdown, Pandoc, Love

Enter Markdown

Enter Pandoc

Whew!

Poor man's spatial query against ArcSDE

There are many ways to skin a cat

Idiots Guide To UML Class Diagrams

Basic UML for Class Design

Introduction

The Elements

A Class

An Interface

Relationships

Simple Association

Cardinality

Directionality

Aggregation

Composition

Other Dependencies

Inheritance

Dependents On

A Complete Example

Dev Cast Dependency Injection

Running a bunch of SQL scripts in C#

LINQ to objects: gettting the SQL version into C#

SQL: grouping by maximum date and ID

Building WSDLs from .NET assemblies

NHibernate Events to the rescue

The Issue, in which Dylan gets bit in the ass by a legacy database

NHibernate Events and Aspect Oriented Programming

The pieces are all there. How to actually do it.

I love unit test results that are green

Bad Explanation

Visual Studio theme - Ruby Blue knockoff

Circuit Breaker of the non-electrical variety

Release It!

C# Circuit Breaker, Anyone?

Summary

Programming Opinions

My Take

My Take

Rake for .NET Projects

Rake

Summary

StructureMap Registry

SQL Server and firewalls

RegEx to the rescue: finding “between” matches

NHibernate HQL Queries I always forget

Windows Services the Easy Way

TopShelf

Debug, Install, Uninstall

Transforming ArcGIS Server REST JSON to NetTopologySuite

Converting ESRI geometries to GeoAPI geometries

Open source and Microsoft culture

When you expect an exception in NUnit 2.5

Event Driven Architecture

Using Python and Oracle with cx_Python

Writing Documentation Using reStructured Text

reStructured Text

Generating Output

A Bit More Detail on Requirements

Gotchas

Talking to Oracle with Ant

On Visualization

How to get the display name of a Windows login when you start from the login name

Utility Function for Creating Images on the Fly

Compression using 7zip and .NET

Parsing XML in a SQL Server Stored Procedure

WiX and XML, Windows Services