The Cloud may have the Computing, but the River has the data….
Organizations around the world are busy extolling, exploring or arguing with the notion of Cloud Computing, but there is another metaphor that may grow in importance as Cloud Computing matures — the River of data that flows between organizations and their partners which can be tapped for power and profit.
Every day, companies exchange millions of electronic documents with each other that have some very special characteristics:
- the data matches actual transactions occurring in the “real world” (orders, shipments, payments)
- the data is structured in a way that provides meaning, it can be processed and turned into information
- the transaction data has relationships to other transactions in the River that can be correlated
Unfortunately, for most organizations, the potential to tap into this powerful source of information remains just that, potential. The challenges inherent in examining, correlating, and acting upon the massive flows of data generated by even modest enterprises have typically been overcome only rarely, and often not at a sufficient scale to truly exploit the opportunity. In this post I’d like to look at some of the challenges that make this difficult, and I will explore the opportunities in future posts.
Some of the Challenges….
Every day, companies exchange millions of electronic documents with each other that have some very special characteristics:
- the data matches actual transactions occurring in the “real world” (orders, shipments, payments)
- the data is structured in a way that provides meaning, it can be processed and turned into information
- the transaction data has relationships to other transactions in the River that can be correlated
Unfortunately, for most organizations, the potential to tap into this powerful source of information remains just that, potential. The challenges inherent in examining, correlating, and acting upon the massive flows of data generated by even modest enterprises have typically been overcome only rarely, and often not at a sufficient scale to truly exploit the opportunity. In this post I’d like to look at some of the challenges that make this difficult, and I will explore the opportunities in future posts.
Some of the Challenges….
- Context: even today, most B2B transactions utilize traditional EDI standards like ANSI X12 or EDIFACT. Despite the standardization of documents, it is frequently tricky to understand the context in which a given transaction is operating (e.g. for a given ASN — ship notice — what order process is it part of?). Without context, it is a free floating piece of data, requiring some associations to turn it into information. If a critical shipment is one day away from delivery to a store about to run a promotion, that is information — if Shipment #101 has been processed, and I don’t know what process it is part of, that’s just data
- Timeliness: while immediacy is a great benefit of the data flowing between trading partners, age is its enemy. With today’s more efficient supply chains, data starts to go stale very rapidly. The challenge is to connect data flowing between partners to other information and to business processes before it is too late to act upon the information. The data flow is not unlike electricity generated by a dam, which must be sent over wires to be consumed in real time. Data warehouses can use the last three years of data to help forecast, but a logistics system has to act upon what is happening the chain today to deliver ROI
- Exceptions: until “the perfect order” is actually achieved, some of the most critical issues between partners are those that generate exceptions, either technically (cannot process an order or logistics document) or on the business side (wrong product, amount, price). It is truly scary at the latency involved in resolving these exceptions, with most of them basically “failing out” to a manual process. Despite the fact that for the “success case” there may be 100% automation, the most common “automated” exception handling is an email alert (how full is your inbox?)
- Integration: the “father of all challenges” when it comes to automation (and its role in resolving the previous three challenges), is the ability to integrate the flow of data into the many systems that are capable of rapidly providing context, and identifying and handling exceptions. This challenge has always been daunting, but recent developments in technology and the ongoing march of technology have started to improve the integration possibilities
- Scale: the sheer volume of data, while being its greatest strength, may also be its biggest challenge. Since millions of transactions flow between partners on a given day, to understand what is happening, all of those transactions have to be interpreted, put into context and analyzed for meaning. RIGHT NOW! If the approach is to methodical, you succumb to the timeliness challenge, but if the approach is too loose, you may generate so many “false exceptions” that you actually degrade business performance, rather than improving it