Posted by: pauditore | March 21, 2013

In-Memory: Big Data Innovation for the Millennium

innovatoinIs In-Memory database technology the next major innovation in the world of business intelligence? Is it the game changer that will provide the unfair advantage that leads to the competitive advantage that every CEO wants today? There is certainly a lot of hype around this (not new technology) and many vendors are touting it as the next greatest advance in the business intelligence tool chest. In the course of this treatise I will avoid the technobabble of In-Memory computing, but instead look at it from a practical business perspective. One of the most amazing experiences of my enterprise computing career has been when technology optimizes business processes and In-Memory has great potential to enable real time decision making in many aspects of business, especially operational business intelligence and performance management applications.

What is In-Memory?

There are many definitions of In-Memory but I like Wikipedia’s Definition the best:

“An in-memory database is a database management system that primarily relies on main memory for computer data storage. It is contrasted with database management systems, which employ a disk storage mechanism. Main memory databases are faster than disk-optimized databases since the internal optimization algorithms are simpler and execute fewer CPU instructions. Accessing data in memory reduces the I/O reading activity when querying the data which provides faster and more predictable performance than disk.”

The real magic here from my perspective is the ability to replicate and OLTP database even on multiple nodes, this enables lighting fast analytics performance and will be a major enabling technology for running data centric enterprise applications in the Cloud.

In-Memory is here now in various forms:

  • As a standalone diskless database.
  • As an appliance with built in analytical modules, IBM-Netezza, SAP-Hana, Oracle Exalytics.
  • Embedded in many ERP systems and other business applications that are data intensive.
  • Embedded in business intelligence platforms.
  • Embedded in data centric Cloud applications and infrastructures.
  • Open Source: WebDNA7
  • Strong set of analytical tools and modules integrated with In-Memory often optimized for industries and business functions such as operational reporting.

The ability to replicate, store and perform analysis on real time data even from OLTP systems enables organizations to make “real time decisions” on nearly all aspects of business operations that have been digitized. Way back in 1999 when I ran the BI/DW Research Program at, our research and surveys found that business intelligence managers wanted the ability to data mine real time transactional data bases; In-Memory now makes this is a reality.  However, the rigorous disciplines of business intelligence like extraction, loading and cleansing of data in addition to the appropriate application of analytical tools still remain. Remember the old business intelligence adage of the 1990’s garbage in garbage out, decisions made on unreliable or incomplete data sets lead only to bad decisions, real time or not.

The Business Value of In-Memory

The inherent business value of In-Memory revolves around the ability to make real time decisions based on accurate information about seminal business processes. An Oxford Economics study ( of 500+ C-level executives conducted in the spring of 2011, focused on real time business in the oil and gas, retail, consumer products, high technology industries. This research had some great findings and insights about how companies can gain competitive advantage with a real time business approach enabled by In-Memory:

  • “At an operational level: by speeding up data capture and simplifying

processes executives can reduce inventories, minimize business risks, lower

  • operational costs, accelerate speed to market, foster productivity and better

meet customer needs.”

  • “At the management level: by accelerating decision-making and planning

executives can exploit market opportunities faster, identify competitive

threats sooner, cope with market shifts more quickly and transform stagnating


There are a significant number of academic researchers studying In-Memory in North America which I would like to mention for reference: they include, Sam Madden at MIT, Ousterhout at Stanford, Abadi at Columbia, Mack Franklin at Berkeley and Peter Weil at MIT’s CISR, as well as Tom Davenport at Babson. Outside of Davenport, however, the majority of the research focus is on the technical aspects of In-Memory and not the business application and value.  Davenport authored another interesting article related to In-Memory: “How Fast and Flexible Do You Want Your Information, Really,” MIT Sloan Management Review, Spring 2011.

The business intelligence arena is strewn with acronyms from hell, which most business people don’t understand and that unfortunately it is the language of computers. Line of business people, managers and even CEOs will derive the most business value from In-Memory and could care less about the technology and the BI acronym jungle. There are many business processes and business intelligence applications that will be innovated with In-Memory, such as query and reporting, online analytical processing (OLAP), predictive analytics, scenario planning, simulation, price optimization and planning and budgeting for example.

Key Areas Where In-Memory Delivers Business Value

  • Finance
  • Performance Management
  • ERP Systems
  • Operational Business Intelligence
  • CRM
  • Mobile Business Intelligence

Finance & Performance Management

In my view these are the most important application areas that can benefit from In-Memory enabled business intelligence. Many organizations spend a great deal of time running financial queries and reports on accounts receivables for example, queries can take hours or even days depending on their complexity. One In-Memory vendor touts that it reduced an account receivable query from 77 minutes to 13 seconds. The ROI on this is somewhat debatable, would waiting an hour for a query to run vs. getting it in seconds make that much of a difference in decision making? Business analysts often spend 90% of their time running queries and significantly less time doing actual analysis. Faster queries, more time for analysis should ultimately lead to overall improved business performance.

Innovation in ERP

In-Memory is a huge innovation for all ERP customers (and a boondoggle for ERP vendors) who have been waiting decades for real time access to transactional ERP data, vendors will undoubtedly target industries with large embedded ERP systems that deal with huge volumes of data such as:

  • Retail
  • Manufacturing
  • Financial Hedge Funds, Brokerage, Insurance
  • Energy
  • High technology
  • Telecommunications
  • Consumer Product Goods
  • Cloud Infrastructure Service Companies


The ability to know and understand the customer experience is paramount in the new millennium as organizations strive to improve customer service, keep customers loyal and gain greater insights into customer purchasing patterns. This has become even more important as a result of social media and social media networks that are now the new “word of mouth platform.” In-Memory enabled BI tools will provide real time data not only from transactional systems, but will allow organizations to harvest and manage unstructured data from the social media sphere.

Mobile Business Intelligence

Demand for access to data from mobile devices is exploding especially in the retail industry and IT is hard pressed to deal with it. In-Memory databases, especially in the Cloud domain will play an increasing role in delivering on the promise of real time data, anywhere anytime. There are a number of new mobile business intelligence applications with easy user interfaces that will greatly benefit from this potentially game changing technology.


In the age of big data, In-Memory is a welcome addition to the business intelligence arsenal and it will undoubtedly significantly change and optimize many business processes. More importantly, it has the potential to allow organizations, line of business managers and the C-suite to spend more time on simulations, scenario planning and analysis, and less time on building and waiting for queries. This is important in the age of the Internet where information travels at a business velocity slowed only by geographical time zones. In-Memory will come at high cost for many organizations, more than $150,000 for a single implementation, not counting costs of data migration and the issues around failover and redundancy. Until next time I wish you great selling and marketing in the millennium.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: