Netrics HD

In my previous post The Challenges of Data Transparency, I discussed the news about the data preparation for Recovery.gov regarding how state and local recipients are spending federal stimulus money.

In the post, I talked about the juxtaposition of data transparency and data quality, and how although missing or incomplete data is a common problem, completeness without any regard for accuracy could possibly do more harm than good.

I asked whether data should be concealed until it has been verified to be of sufficient quality, or should be provided as soon as it becomes available without regard for quality.

This past week, we have been inundated with news reports from numerous media outlets regarding the glaring data quality issues found on Recovery.gov, which would seem to indicate many would answer my question by advocating concealment until the verification of data quality has been performed.

I don’t want to get into some of the more politically charged aspects of the current debate.  I would prefer to pose the question in a more general sense.  When it comes to data, does it fundamentally come down to transparency vs. quality?

From my perspective, the underlying struggle in this debate is the desire to achieve both total data transparency and perfect data quality.  As wonderful as it would be if this was possible, the reality is simply that it is not.

Perfection (especially in data) is impossible to achieve.  Transparency reveals the quality issues naturally inherent in data.  I am not advocating we simply accept the reality of poor quality.  We must take action to identify and overcome data quality issues.

The traditional approach is employing standardization and other data cleansing techniques in an effort to perfect data.  Continuing advancements in mathematics and machine learning algorithms provide the capability to adapt to (and overcome) data’s inherent imperfections.

We must strive for total data transparency balanced with a realistic perspective of data quality.  Transparency provides the necessary access and emerging innovations in quality provide the methods for transforming data into actionable information.

Related Posts

The Chaos Theory of Data Quality

Drowning in Imperfect Data

The Growing Importance of the Algorithm

The Growing Importance of Mathematics

In the recent Federal Computer Week article Stimulus spenders race to the finish line, Alice Lipowicz reported on the efforts of state and local recipients to prepare detailed reports on how they are spending federal stimulus money.

Among the common concerns cited in the article was dealing with the sheer volume of data, and more importantly, the quality of the data.

“Transparency advocates predict that data quality,” reports Lipowicz, “will be as much of a concern at Recovery.gov as it is at USAspending.gov, which Congress established in 2006 to provide visibility into federal spending.  That site has been plagued with problems such as errors, missing data and mislabeled data.”

Data transparency is definitely a laudable goal, and not just for the government.  Organizations in every industry and of every size need to do a better job of making available for review, the data that was used to drive critical business decisions, especially financial decisions.

Missing or incomplete data is a common problem, but transparency can not simply mean a massive dump of all available data.

Completeness without any regard for accuracy could possibly do more harm than good.  Data frequently contains numerous variations caused by different conventions, lack of standards, omissions, and other inconsistencies.

An excellent question raised in the article was:

“Data quality has been a problem for years, so why do we keep getting [more data] instead of addressing these priorities?”

I think that this question represents one of the most significant challenges for data transparency.

Should data be concealed until it has been verified to be of sufficient quality?  Or should data be provided as soon as it becomes available without regard for quality?

Please share your thoughts.

Related Posts

Drowning in Imperfect Data

A Sisyphean Task…

All industries are feeling the effects of the current economic recession.  All industries are interested in the many benefits of innovation.  However, in the technology industry, not only is it a significant interest, innovation is the driving force and lifeblood of a rapidly evolving sector.

Historically, many of the most powerful new trends in technology originated from small entrepreneurial ventures.  Small technology vendors tend to be specialists with a narrow focus that can provide a great source of innovation.  Small vendors are usually less financially stable and rely mainly on funding for their early survival.  However, the current economy is limiting venture capital opportunities.

Large technology vendors tend to innovate via acquisition of these smaller vendors.  A continuing trend in information technology is the consolidation of acquired functionality into enterprise class application development platforms with integrated components for data quality, data integration, master data management and business intelligence.  This allows large vendors to offer end-to-end solutions and the convenience of one-vendor information technology shopping.

However, further innovation is typically delayed while the vendor prioritizes integrating the acquired technology into the existing suite of products and integrating the acquired people into their existing staff.  Additionally, training programs and sales strategies must be adjusted to reflect the updated platform and product offerings.  All of this requires significant time and effort.  The collateral damage is innovation can lose momentum or become stagnant.

Even before the economic recession, many in the information technology industry expressed concern about the effect of this vendor consolidation on innovation.  Now with the economy starting to claim some promising start-ups, are we looking at an innovation recession?

Of course, the situation is more complicated than small vendors vs. large vendors.  Vendors of all sizes are struggling with finding viable business models to pursue possibilities such as cloud computing and software-as-a-service (SaaS).

The trend with most innovations is that early adopters often spend more than they earn in these pursuits.  Innovation is often high-risk with no guarantee of high-reward. Entrepreneurial start-ups are usually more willing to go “all-in” and risk everything, but again venture capital is currently hard to come by.  Large vendors have more financial stability, but in a down economy it is often better even for them to play it safe.

Everyone wants to do more with less.  But some are settling to simply do less – long term (or even medium term) this is a losing strategy.  Innovation often stimulates the economy but the paradox is that an economic recession both spurs and suppresses innovation.

With many information business technology professionals feeling forced to find a way to survive, has do more with less become do without innovation or is it truly do with innovation?

My belief is that it’s the latter.

Related Posts

Innovation Video from BusinessWeek

Innovation – Do More with Less

The Cloud brings Commoditization

The economic tsunami that’s hit the world is bringing change on many fronts. This one is most curious: Trip Chowdjry a tech analyst at Global Equities Research believes that in 2009 we’ll see  a four day work-week to cut salary costs by 20%.

For some companies that may make sense but I don’t think this is something that’s going to catch on. If you want 100% commitment from your employees you have to give them the same commitment.

So I guess it boils down to either:

(a) 100% of employees being 80% happy
(b) 80& of employees being 100% happy
(c) a mix of (a) and (b)

I believe the answer is clearly (b) for the majority of organizations. Then again (c) could work if there’s  voluntary opt in. If an organization chooses (c) then is an 80% employee less likely to get promoted, especially compared to a 100% employee?

The other question to ask is has the net amount of work that the company must do been reduced by 20%? Or do all the employees suddenly become more efficient?

What do you think?

For individuals, families, businesses and organizations the theme for 2009 will undoubtedly be “Do More with Less.” The Google count is a mere 794,000 right now – I’m curious to see where it ends up 12 months from now.

For business and organizations doing more with less in these fraught times is imperative for survival. On the IT side, technology professionals at all levels must stay focused on delivering business value more efficiently and effectively than ever before.

One of the key enablers for doing more with less is innovation. Innovation delivers advantage – in the the form of saving costs, saving effort and improving results.

As we are all asked to do more with less let’s not forget that driving innovation is one way to make it happen.

Netrics is all about delivering innovative enterprise software! I’ll write about some specific examples in upcoming posts.