Last year’s survey was primarily concerned with the relevance of the classical Business Intelligence value proposition in the face of a rapidly-changing data environment – big data, real-time analytics and the incursion of statistical methods into what has traditionally been a data-aggregation business.
Through the survey we found that the classical value proposition was still highly relevant – data warehouses still need to provide one source of the truth and strong visualisations must make insight visible. We also found though that increasingly in South Africa these classical approaches are starting to meld with the analytical, statistical and big data environment. This trend was seen as highly relevant, although actual uptake and spend had yet to reflect it.
This year our starting point is still that the whole world of data is changing. We’d like to determine which of the new aspects of change are having an impact in our market.
1. OT and IT convergence
Operational Technology (OT) can be defined as the monitoring and control of processes by means of instrumentation and physical devices. This has always been the domain of process engineers. Examples are mine and mineral processing, production line automation, fire and hazard management and many others. Information technology is technically defined as applying to all forms of computerised data, but is generally applied to transactional systems, ie anything that could fall into the ERP category. These systems have seldom integrated or even exchanged data, but a major wave of convergence is underway, which is accelerating with the meteoric rise of the Internet of Things (IOT). Good examples of this can be found in logistics, insurance and health care. Integrating OT brings new challenges to the BI practitioner. OT architectures and are highly proprietary and can be inaccessible to enterprise’s BI practitioners. Transaction rates can be very high, which can overwhelm infrastructures. And security in the OT environment is often rudimentary.
2. Data enrichment
Data enrichment has been generally thought about as a form of data cleansing, used for requirements such as address correction or content interpretation. Typically this would need a normalisation process to get formats standardised, followed by extraction algorithms. However, new ways are developing of thinking about data, by enriching transactional data with that from other sources. The first widely-used example was the use of third-party maps as a basis for data visualisation. Another example is image collection and recognition combined with GPS for asset location. Enrichment technology has the potential to transform many business processes, but the collection and storage of the data can be problematic in the BI environment. For example, where evidence of inspection is required, an image must be provided. Where a learning algorithm has provided strategic insight, the way it arrived at that insight needs to be testable. BI is currently not well geared for these requirements.
3. Data partnerships
We recently became aware of the data partnership between Facebook and Whatsapp. Two separate (although mutually owned) organisations exchange data with the purpose of enhancing the customer experience. The world is filled with specialist pockets of data ownership, pertaining to every aspect of human life. The exchange of data, either purchased or for mutual data enrichment, is becoming increasingly exploited by market leaders. However this thinking hasn’t yet started amongst many companies, and there is a real risk to them that a key differentiator and market advantage may be seized by a competitor. Just how this will affect Business Intelligence, regarding collection, storage and display of other parties’ data, is unknown. For example, the integrity and accessibility of third party data will be governed by strict agreements, for which compliance must be shown.
4. The technical impact of these developments
Business Intelligence has usually reacted to changing environments, rather than being an agent for change in its own right. Perhaps it is a good moment for BI architects and practitioners to take a peek into the future, and to start pre-empting the technical challenges that will arise over this watershed period, in data collection, storage, visualisation, security and integration.