Synonyms: This is in my eyes the most basic approach. You have a list of common translations to different words like common misspellings, nicknames and so on. This approach is of course very depending on heavy maintenance, and must be worked over for every language/country – and actually works better with English than other languages like the Germanic ones, where you use concatenated words (like ‘Main Street’ being ‘Mainstreet’).

Match codes: You find those from very simple ones to the more sophisticated ones – going from ignoring vowels, soundex and metaphone (for English) to proprietary findings of all kinds. In my eyes match codes works OK for selecting candidates for matching – but falls a bit short when coming to actually settling the case.

Algorithms: A complex algorithm is a more sophisticated way to settle if two different spelled records make up the same real world entity. You have to deal with truncations, non phonetic typos, rearranged words and letters and all that jazz. The “LevenshteinDistance” is an example of an algorithm you could use – but such a method is just a fraction compared to the commercial used algorithms around.

Probabilistic learning: This is if fact a variation of synonyms, but the collection is not based on up front maintenance but collection of users actual decisions when verifying automatic matching. The tool will register the frequency and context of the paired elements in the decisions. This of course requires a substantial collection. I have implemented such a feature at organisations, where several people every day do verify matching results.

And then parsing and standardisation is often supplementary methods used to improve the matching. Also bringing in more data to support the decision is in my eyes a key to actually settle if some records make up the same real world entity. Business and consumer/citizen directories are available in different forms, coverage and depth around the world.

We are still left with the problem that IT colonised the word information early while actually focussing on the technology. The result is that to this day many people, and particularly many senior people in organisations confuse managing the technology with managing information

As the technology matures still further, and the technical wizards increasingly make the “difficulties and intricacies” you speak of ever more hidden from the business, I believe we will see the rise of true Information Management functions in all successful organisations

The issue will not be of IT becoming BT (a scary thought for us Brits where BT stands for something rather different!) but of Technology functions coming to work hand in hand with Information Management to realise a true Information Age

I’ll need a dedicated blog posting to provide some background and offer a complete and unbiased answer.

But in the mean time, here is the condensed version: traditional matching rules are not good enough – even when you add probabilistic components to them. So, there’s demand for innovation – and that innovation is mathematical modeling.

Any ideas?