Netrics HD

After years of neglect, data quality is slowly moving to the forefront of business technology as both a discipline and a thriving industry.

However, given data quality license revenues are estimated at a relatively minuscule $400 million for 2009 (compared to $17 billion for DBMS license revenues), data quality is not quite center stage yet.

Therefore, in this post I want to discuss the increase in awareness by organizations that is necessary to give data quality its due.  I describe it as the three levels of Data Quality Enlightenment (DQE).

DQE Level 1 – Unaware

Organizations at Level 1 are blissfully unaware that slight discrepancies in their data create the potential for their business processes to fail.

Sometimes, the resulting failure is immediately visible.  Other times, it eventually becomes visible in a downstream application or after some period of time has passed.  Either way, the organization feels the impact of the failure as increased costs or decreased revenue, or both.

Upon finally recognizing the root cause of the problem to be data quality, organizations typically progresses to Level 2.

DQE Level 2 – Aware

Organizations at Level 2 have come to realize that they must implement data quality measures to avoid the costs of “bad data.”

The logic usually goes like this – if data is not perfect our business processes can fail, therefore we must make sure that our data is always perfect.  What wonderfully flawed logic!

No matter how hard an organization tries, their data can never be perfect.  Why?  Because by its nature, the data, and access to it, changes over time.

Existing records are updated.  New records are created.  Both of these actions can be performed by existing or new people and by existing or new systems.

With the additional reality that these people and systems can be both internal and external to the organization, the complexity grows exponentially.

Therefore, is it realistic to expect that all data throughout the enterprise will always be kept perfect and standardized the exact same way?

Will humans accessing the data know and use the standard methods?  Will humans always know the exact and correct data they want?  Will multiple applications (within and between organizations) that need to share data use the same standards for data perfection?

Of course not.  Simply put, perpetually perfect data is not possible.  Don’t believe anyone who tells you otherwise.

Yet despite these facts, the majority of the data quality industry is still focused on attempting to achieve data perfection.

The common belief is that the way to data Utopia is by writing rules to parse, standardize and match data.  Of course the different rules have fancy technical names like “deterministic” and “probabilistic” but they all boil down to manual, static rules that need to be created, maintained, and updated in perpetuity.

The rules an organization has in place today for “perfect data” will have to change (update old rules and add new rules) as the data changes.

Unlike Level 1, where organizations quickly realize they must change and progress to Level 2, most organizations at Level 2 get stuck here and never progress to Level 3.

DQE Level 3 – Enlightened

Organizations reach Level 3 when they achieve enlightenment via the “eureka moment” when they realize that getting and keeping data perfect at all times and forever is, fundamentally, an insane idea.

These organizations then seek to find a better way.

That better way is to enable all enterprise applications to function correctly despite the fact that the underlying operational data they use is not perfect.  And to do it without constantly updating and creating rules to parse, standardize, and match data.

The enlightened phase has only just begun with a select few organizations reaching Level 3.

Enlightenment is Inevitable

As is often the case,  enlightenment comes from a simple yet powerful idea that breaks away from the constraints of conventional thought.

It’s only a matter of time before every enterprise application will no longer assume and require “perfect” data in order to function correctly.

When this finally happens, and it will, everyone will benefit.

In his recent ebizQ.net article SOA, Phase 2: Toward a Loosely Coupled World, Joe McKendrick declared:

“I am a passionate believer in the power of technology, as an enabler of entrepreneurship and organizational transformation. I have long advocated flattening the organizational hierarchy, and pushing decision-making down to the managers and employees who deal with customers and production on a day-to-day basis.”

I couldn’t agree more.  Nothing has a more powerful effect on an organization’s ability to succeed than putting the right technology into the hands of front line employees.

There is an unstoppable industry trend gaining daily momentum where organizations are increasingly looking for solutions with cloud computing and software-as-a-service (SaaS) as the new paradigm for enterprise architecture.

“Cloud computing is pushing some software vendors to change their models to component delivery,” explains McKendrick.  “This makes plenty of room not only for small start-ups, but also for development shops within traditional enterprises that have great ideas.”

Historically, many of the most powerful new trends in technology originated from small entrepreneurial vendors.  By focusing on enhancing their highly specialized components, they can provide a great source of rapid innovation.  Therefore, small software vendors, whose solutions are designed for deployment using a loosely coupled service-oriented architecture (SOA), may be the industry’s small giants upon whose broad shoulders we will all be standing in the not-to-distant future.

And according to Mohan Sawhney, professor at Northwestern’s Kellogg School of Management:

“The best-run companies are becoming orchestrators of networks of services.  Five years from now, the concept of an application will be obsolete.  They will all be services, combined, mixed, matched and reused as needed.”

Therefore, when it comes to enterprise architecture — service-oriented is future-oriented.

Related Posts

The API and the Innovation of Enterprise Applications

Innovation Recession?

Innovation – Do More with Less

The Cloud brings Commoditization

“One of the bigger trends to come down the pike lately,” explained Jim Ericson in his recent Information Management blog post The API is the New Network, “is the proliferation of Web-based application programming interfaces, or APIs, and how network traffic is growing exponentially through APIs.”

More and more organizations continue to look to innovations in cloud computing, software-as-a-service (SaaS), and information as a service, as a new paradigm for enterprise applications.  In a recent press release, Gartner Research identified the Top 10 Strategic Technologies for 2010 and the list includes both cloud computing and client computing.

This is an almost stark contrast to the traditional approach taken by large technology vendors, who tend to innovate via acquisition in order to offer consolidated enterprise application development platforms with seamlessly integrated components for data quality, data integration, master data management and business intelligence.  This allows the large technology vendors to offer end-to-end solutions and the convenience of one-vendor information technology shopping.

However, does buying everything from one large vendor guarantee a best of breed solution for each individual component?

An API-oriented approach enables a plug-and-play enterprise application strategy.  Under this model, enterprise applications are assembled from best of breed individual components that are loosely coupled via a network of API calls.

Historically, many of the most powerful new trends in technology originated from small entrepreneurial ventures.  Small technology vendors tend to be specialists with a narrow focus that can provide a great source of rapid innovation.

Perhaps we are witnessing the beginning of the reversal of the recent trend of vendor consolidation, and a return to the earlier industry landscape where smaller vendors remained focused on enhancing and improving their highly specialized components.

If the API is indeed the new network, then the innovation of enterprise applications is to be found in collaboration and not consolidation.

Related Posts

Innovation Recession?

Innovation – Do More with Less

The Cloud brings Commoditization

In his absolutely fantastic 2006 Princeton University essay The Algorithm: Idiom of Modern Science, Bernard Chazelle pondered the Holy Grail quest of computer science:

“How to unleash the full computing and modeling power of the Algorithm.”

Chazelle describes how Moore’s Law, which states that computing power doubles every two years, has delayed the rise to prominence of the algorithm, in much the same way that an abundance of relatively cheap oil has delayed the emergence of alternative energy sources.

The Triumph of Mathematics

“To make sense of the world, we have math,” explains Chazelle, and therefore, some might ask: Who needs algorithms?

“It is beyond dispute,” continues Chazelle, “that the dizzying success of 20th century science is, to a large degree, the triumph of mathematics.  A page’s worth of math formulas is enough to explain most of the physical phenomena around us: why things fly, fall, float, gravitate, radiate, blow up, etc.”

As Albert Einstein said:

“The most incomprehensible thing about the universe is that it is comprehensible.”

“Granted,” says Chazelle, “Einstein’s assurance that something is comprehensible might not necessarily reassure everyone, but all would agree that the universe speaks in one tongue and one tongue only: mathematics.”

The New Language of Science

“The Algorithm’s coming-of-age as the new language of science,” declares Chazelle, “promises to be the most disruptive scientific development since quantum mechanics.”

Algorithms are thought by some to be simply a way to automate the rapid execution of a task.  Although speed is important and the exponential growth of computing power has allowed algorithms to execute faster, it is the quality of the work performed by the algorithm that is vastly more important, especially algorithms used for complex data analysis in support of critical business decisions.

“The algorithmic paradigm,” explains Chazelle, “is not about what but how to think.  Self-reference is associated mostly with self-replication.  In the algorithmic world, by contrast, it is the engine powering the complex recursive designs that give abstraction its amazing richness: it is, in fact, the very essence of computing.  Should even a fraction of that power be harnessed for modeling purposes, there’s no telling what might happen.”

For example, using graph theory (a branch of theoretical mathematics), algorithms can construct mathematical models for the ways that humans recognize patterns in data.  The goal of these algorithms is not to replace human decision makers.

These algorithmically constructed models can be used to automate the rapid execution of analytical tasks providing true decision support for humans to use while navigating today’s challenging business environment, which faces daunting data volumes and a constantly evolving marketplace.

“Some say the Algorithm is poised to become the new New Math, the idiom of modern science,” explains Chazelle.  “I say The Sciences They Are A-Changin’ and the Algorithm is Here to Stay.  One thing is certain, Moore’s Law has put computing on the map: the Algorithm will now unleash its true potential.”

I completely agree and wholeheartedly echo the closing remark of Chazelle’s essay:

“May the Algorithm’s Force be with you.”

Related Posts

The Growing Importance of Mathematics

Drowning in Imperfect Data

Matches Created

A more precise, but less certain world

Narrative Fallacy and Data Matching

For speaking at this year’s Enterprise Data World conference, I received a copy of Stephen Baker’s amazing book The Numerati, which was inspired by his Jan 23, 2006  BusinessWeek article Math Will Rock Your World (one of my all time favorites!).

“When it comes to producing data,” explains Baker, “we’re prolific.  The very air we breathe is teeming with motes of information.  People with the right smarts can summon meaning from the nearly bottomless sea of data.  The key to this process is to find similarities and patterns.  We humans do this instinctively.”

Humans Teach, Machines Learn

Advancements in machine learning technology using sophisticated mathematical algorithms are providing the capability to make better data-driven decisions.

“Learning machines swim in numbers,” explains Baker.  “The learning process starts with humans…the annotators.  Their work is…to teach the machine what we humans know at a glance.”

Therefore, these advancements are not an attempt to replace human knowledge workers.  The number crunching capabilities of these advancements will allow us to “gradually evolve from data serfs into data masters.”

Advanced Geometry

There are many mathematical disciplines involved in machine learning.  However, perhaps one of more surprising is advanced geometry.

“Scientists often describe the world of data as a domain of sharp angles, colliding planes, and vectors shooting along endless paths,” explains Baker.  “Imagine a vast multidimensional space [with] dozens of markers…each marker occupies its own patch of real estate.”

Imagine each marker representing an individual character within a string of text.  Machine learning using bipartite graphs to allow data to “produce a line – or vector – that intersects with each and every one of its own markers…it’s a little like those grade-school exercises where a child follows a series of numbers or letters with her pencil and ends up with a picture of a puppy or a Christmas tree,” explains Baker.

However, the picture that bipartite graphs are drawing are too complex for the three-dimensional world of the human imagination.

“The computer has no trouble depicting [data] as vectors,” continues Baker.  “They all run neatly from one dimension through countless others and, more important, through every one of their distinguishing markers.  [Data] that resemble each other, naturally enough, are neighbors in this vector space.  [Data] that have a lot in common tend to point at similar angles.  Each link shared is a line connecting them, a so-called edge.  The next step is to calculate the importance of each edge…[edges] given a higher score…those lines on the graph are thicker.”

A New Era of Applied Mathematics

“The information age that we’re in is going to be an emerging new era of what would be called applied mathematics,” concludes Baker.  “Mathematicians are going to dip into the sea of data to form…the mathematical modeling of humanity.”

From the beginning of civilization mathematics has been central to our advancement. It is after all the language of science. But our relatively new found ability to collect digital data has ushered in a new era for leveraging and benefiting from mathematics.

Related Posts

Drowning in Imperfect Data

Matches Created

A more precise, but less certain world

Narrative Fallacy and Data Matching

For a long time, the conventional wisdom in enterprise software implementations seemed to follow the old adage:

“There is nothing that can be done in a short time that can’t be done just as well in a long time.”

However, another old adage is “time is money” and there are many time-sensitive costs associated with enterprise software implementations that greatly increase their total cost of ownership and greatly decrease their total return on investment, including:

• Software Installation and Configuration
• Application Development
• Production Deployment
• Application Maintenance
• Software Enhancements and Bug Fixes
• Software Upgrades

We have also become conditioned to expect that a typical enterprise information initiative requires between 6 to 18 months to deploy, even when employing a agile (or phased) approach to application development.  However, with Gartner estimating that the volume of enterprise data also doubles every 18 months, the very challenge for which the enterprise is implementing a solution becomes a moving target.  The ground is therefore constantly shifting beneath our feet.

Day-to-day business challenges are evolving faster than the applications under development can come on-line – and by the time they do, the underlying business requirements they were based on have changed, forcing end-users to attempt to manually close the gap.  Basically, the business is being forced to adapt to the technology.

The current economic recession is also forcing everyone to try to find ways of “doing more with less.”  Some have been looking into the potential cost savings and operational efficiencies offered by cloud computing and software-as-a-service (SaaS).  But perhaps we should also be challenging vendors to not only provide enterprise software that is easier to install, configure, develop, deploy, and maintain – but that is also adaptive to changing business logic.

In order to remain competitive in such a constantly evolving environment, we need technology that adapts to business needs.  We need adaptive software.

Related Posts

Humans adapting to computers – instead of the other way around

Innovation Recession?

Innovation – Do More with Less

The Cloud brings Commoditization

All industries are feeling the effects of the current economic recession.  All industries are interested in the many benefits of innovation.  However, in the technology industry, not only is it a significant interest, innovation is the driving force and lifeblood of a rapidly evolving sector.

Historically, many of the most powerful new trends in technology originated from small entrepreneurial ventures.  Small technology vendors tend to be specialists with a narrow focus that can provide a great source of innovation.  Small vendors are usually less financially stable and rely mainly on funding for their early survival.  However, the current economy is limiting venture capital opportunities.

Large technology vendors tend to innovate via acquisition of these smaller vendors.  A continuing trend in information technology is the consolidation of acquired functionality into enterprise class application development platforms with integrated components for data quality, data integration, master data management and business intelligence.  This allows large vendors to offer end-to-end solutions and the convenience of one-vendor information technology shopping.

However, further innovation is typically delayed while the vendor prioritizes integrating the acquired technology into the existing suite of products and integrating the acquired people into their existing staff.  Additionally, training programs and sales strategies must be adjusted to reflect the updated platform and product offerings.  All of this requires significant time and effort.  The collateral damage is innovation can lose momentum or become stagnant.

Even before the economic recession, many in the information technology industry expressed concern about the effect of this vendor consolidation on innovation.  Now with the economy starting to claim some promising start-ups, are we looking at an innovation recession?

Of course, the situation is more complicated than small vendors vs. large vendors.  Vendors of all sizes are struggling with finding viable business models to pursue possibilities such as cloud computing and software-as-a-service (SaaS).

The trend with most innovations is that early adopters often spend more than they earn in these pursuits.  Innovation is often high-risk with no guarantee of high-reward. Entrepreneurial start-ups are usually more willing to go “all-in” and risk everything, but again venture capital is currently hard to come by.  Large vendors have more financial stability, but in a down economy it is often better even for them to play it safe.

Everyone wants to do more with less.  But some are settling to simply do less – long term (or even medium term) this is a losing strategy.  Innovation often stimulates the economy but the paradox is that an economic recession both spurs and suppresses innovation.

With many information business technology professionals feeling forced to find a way to survive, has do more with less become do without innovation or is it truly do with innovation?

My belief is that it’s the latter.

Related Posts

Innovation Video from BusinessWeek

Innovation – Do More with Less

The Cloud brings Commoditization

We believe in the importance of innovation – innovation is at the heart of our software products and it’s the essence of our company. The global recession has brought innovation to the forefront!

Here is an interesting short video on Innovation – enjoy!