Netrics HD

For a long time, the conventional wisdom in enterprise software implementations seemed to follow the old adage:

“There is nothing that can be done in a short time that can’t be done just as well in a long time.”

However, another old adage is “time is money” and there are many time-sensitive costs associated with enterprise software implementations that greatly increase their total cost of ownership and greatly decrease their total return on investment, including:

• Software Installation and Configuration
• Application Development
• Production Deployment
• Application Maintenance
• Software Enhancements and Bug Fixes
• Software Upgrades

We have also become conditioned to expect that a typical enterprise information initiative requires between 6 to 18 months to deploy, even when employing a agile (or phased) approach to application development.  However, with Gartner estimating that the volume of enterprise data also doubles every 18 months, the very challenge for which the enterprise is implementing a solution becomes a moving target.  The ground is therefore constantly shifting beneath our feet.

Day-to-day business challenges are evolving faster than the applications under development can come on-line – and by the time they do, the underlying business requirements they were based on have changed, forcing end-users to attempt to manually close the gap.  Basically, the business is being forced to adapt to the technology.

The current economic recession is also forcing everyone to try to find ways of “doing more with less.”  Some have been looking into the potential cost savings and operational efficiencies offered by cloud computing and software-as-a-service (SaaS).  But perhaps we should also be challenging vendors to not only provide enterprise software that is easier to install, configure, develop, deploy, and maintain – but that is also adaptive to changing business logic.

In order to remain competitive in such a constantly evolving environment, we need technology that adapts to business needs.  We need adaptive software.

Related Posts

Humans adapting to computers – instead of the other way around

Innovation Recession?

Innovation – Do More with Less

The Cloud brings Commoditization

Recently, my wife and I were shopping for eye glasses at a popular national retailer.  After finding frames she liked, we walked up to the sales counter to make our purchase.

“Have you purchased from us before?” the sales clerk asked my wife.  In order words, he was asking if she was already a customer that he should be able to find in their customer database.

“Yes,” my wife responded, “in fact, I bought my current glasses in this very store last year.”

“Excellent,” the sales clerk said as he prepared to enter her information into the computer, “we really appreciate that you have returned to purchase from us again, what’s your last name?”

“Damianakis.”

“Uh-huh. Could I see your drivers license?”

No, my wife wasn’t getting carded before she could buy a pair of glasses.  The sales clerk wanted to make sure he typed in our last name exactly as it is spelled.

“Hmmm,” the sales clerk muttered while staring at the computer screen, “I can’t seem to find your account.”  To his credit, he decided (I swear that I just stood there quietly smiling) to try intentionally misspelling our last name – five different ways.

“Are you absolutely sure that you purchased your glasses from this store?”

“Yes, absolutely.”

“Oh well,” the sales clerk responded, “I guess I will just have to enter you into the system again.  For the inconvenience, I will take another 5% off the price.”

I couldn’t help but think to myself – I just witnessed the creation of a duplicate customer – despite the diligent efforts of a front line employee!

It can sometimes be difficult to make a compelling business case for data quality.  But what company doesn’t value repeat business?  However, if your current reports are telling you that only 15% of new sales this year have been from repeat customers, how many of those apparently new customers are in fact, already a customer?

Furthermore, isn’t it time that we get computer systems to adapt to us, instead of us always adapting to their limitations?In this particular case, the sales clerk knew to try several intentional misspellings but was unable to find the right record. That’s backwards – the clerk should have entered the information he knew and the computer should have done the hard work to find the right record.

There is a better way! What are we waiting for, let’s eradicate this problem!

Related Posts

Data Checker – Long Overdue

Apples and Oranges

In a recent blog post Your Friend the Algorithm, Paul Barsch explains:

“With exponential trends of data growth and computational power colliding…companies are using technology…and sophisticated mathematical procedures to analyze data and make better decisions.”

Data frequently contains numerous variations caused by different conventions, lack of standards, omissions, and other inconsistencies.

Traditional approaches to data matching have heavily relied on data standardization to prepare records for matching.  This preparation creates a consistent format that allows for more direct comparisons on parsed attributes with standardized values.

However, the problem with the traditional approach is that “the world is literally drowning in data” explains Barsch.  “There’s too much data, and not enough analysis.”

Advancements in data matching technology using sophisticated mathematical algorithms are providing the capability to make better data-driven business decisions – without the prerequisite correction of data’s inherent imperfections.

According to Gartner Research, the volume of enterprise data doubles every 18 months.  There is also a rapidly growing need for real-time analysis of these burgeoning data volumes in order for companies to remain competitive in a constantly evolving marketplace.

“Algorithms help tackle complicated challenges,” explains Barsch.  “As data volumes and decision options increase, algorithms and the systems that run them take on added importance.”

The need to make imperfect data perfectly usable is becoming more important than ever.

Related Posts

Apples and Oranges

A Sisyphean Task…

Bill James is a baseball writer, historian, and statistician, who is perhaps best known for pioneering the field of sabermetrics, which as he defined it is “the search for objective knowledge about baseball.”

James uses analysis of baseball statistics to evaluate the contribution of an individual baseball player’s performance to their team’s ability to win a game.  For hitters, he believed that “a hitter should be measured by his success in that which he is trying to do…create runs.”

To measure this, James created a new baseball statistic that he called Runs Created:

(Hits + Walks) x Total Bases / (At Bats + Walks)

At the heart of this formula is the premise that a player’s ability to get on base is crucial to their team’s ability to score runs and win games.

Although that may sound rather obvious, the formula’s emphasis on statistics not typically considered important (e.g. Walks) was antithetical to baseball’s “conventional wisdom.”

Traditionally, statistics such as Batting Average (Hits / At Bats) and RBI (runs batted in) were considered tried and true techniques for evaluating hitters.

Additionally, there were the “intangibles” observed by scouts and coaches who trusted their “gut” more than nerdy number crunching.

After all, as these experts would argue – baseball is played on a field, not on a calculator.

All of this was detailed in Moneyball: The Art of Winning an Unfair Game, the excellent 2003 book by Michael Lewis.

Matches Created

In data matching, where statistical properties of fields and their values are used to measure the contribution each field makes to the likelihood that a matching record has been found, success should also be measured by what we are trying to do…create matches.

Tried and true techniques continue to be sought for the complex challenge of creating matches, with many of these techniques coming from advanced mathematics.

When you look under the hood of some of these new approaches to data matching, you might find some fields and their statistical properties being used in ways antithetical to “conventional wisdom.”

Initially, your “gut” might tell you these approaches simply don’t sound like they could possibly create acceptable matches.

However, success is truly measured by evaluating the match results – not the data matching techniques.

In some ways, it brings to mind what the 19th century poet John Keats referred to as Negative Capability:

“Capable of being in uncertainties, mysteries, and doubts without any irritable reaching for fact and reason.”

Of course, Keats was advocating an open-mindedness to new concepts in literature and philosophy, where if something speaks to you of a truth that you could accept but not explain, why bother with trying to explain it?

Therefore, if a new approach to data matching creates matches that you can accept, does it really matter what algorithm was used?

Perhaps we should follow Bill James lead and create a new statistic called Matches Created?

Related Posts

Narrative Fallacy and Data Matching

In the 1970s, spell checking programs debuted on mainframe computers.  In the 1980s, personal computers arrived and spell checkers soon followed, especially embedded within word processors.  In the 1990s, spell checkers joined the Internet revolution and debuted in e-mail programs and web browsers.

Today, spell checkers are virtually everywhere, from mobile text messaging to social networking websites.  In fact, nowadays you would be surprised (and probably annoyed) if spell checking wasn’t available whenever and wherever you were entering text.

Spell checkers are far from perfect.  However, their greatest feature is not just telling you that you have typed a word it can’t find an exact match for in its dictionary – it also provides a list of potential matches that may contain the word you were trying to spell.

In my previous post Apples and Oranges, I asked how many times you have looked up a contact in your e-mail program only to get zero results back.  Now, this might be understandable if you were looking up the e-mail address for Apple by typing in Orange.  However, most of the time what you typed was off by just a single letter (i.e. Aple).

Practically all of the devices and applications we use on a daily basis provide only exact matching.  Isn’t this just as surprising and annoying as not providing a spell checker?

For goodness sake, it’s 2009. Why does this problem persist – especially since the core technology to virtually eradicate this problem exists. Shouldn’t we demand the standard functionality of a Data Checker?

A Data Checker wouldn’t be clairvoyant.  However, if it couldn’t find an exact match for your lookup, it would provide a ranked list of potential matches.

Shouldn’t all of our devices and applications provide the capability of overcoming data variations, lack of standards, omissions, and other inconsistencies in our lookups and searches?

Related Posts

Apples and Oranges