Translating User Insight into Business Value: Application Integration and Control

By Omer Trajman,


WibiData Member of Technical Staff Omer Trajman

This blog post is the fourth and final in a series that explains how institutions can evolve from data miners to experience optimizers by building Big Data Applications using Kiji. The previous posts described the benefits of a customer data repository,  how organizations tackle the challenge of ingesting a wide variety of data into a single system and the model life cycle within Big Data Applications. This post describes how business users can control multiple application channels integrated into a single customer data repository.

A customer data repository is a powerful tool with which organizations can build highly personalized Big Data Applications. Using frameworks like Kiji, organizations can store and analyze user behavior, and score models as users interact with applications in real-time. In order to deploy these applications in production, organizations must integrate these capabilities into their applications and provide the business with transparency and control over the personalized algorithms.

Most enterprise applications are built on a service oriented architecture. This means that each component from the web server, to the application server, to the database, operates as an independent service that can store, process and retrieve information from other services. The user requests a page from the web server, which requests context from the application server, which requests data from the database. Each component returns an answer until the web server formats an HTML page and returns it to the user.

Kiji is run as a Big Data Application server. The existing application server, which is encoded with static rules for the application, sends information about user activity to Kiji over standard REST protocols. Kiji writes this information to the appropriate records in the underlying Big Data storage system (such as HBase). The application then requests information about what to display next to the user. Kiji transparently re-scores the predictive models, applying experimentation and segmentation logic as determined by analysts. The application server receives a freshly recalculated score based on the up to date information about the users’ actions.

Kiji can also store information collected from external systems such as product catalogs, transaction systems and third party data. All of this information may be streamed in via REST or loaded in batch using standard MapReduce based tools. The entire process from data loading to re-scoring and next action retrieval is fully automated in production environments.

All of this automation allows organizations to efficiently deliver personalized applications to users but it does not give businesses the transparency and control they often require. While Big Data Applications give organizations a powerful means of creating personalized experiences for customers, those experiences must still align with the overall business strategy, messaging and brand qualities of the organization. A personalized experience must be delivered in context and the business needs visibility and control over the algorithms that are delivering that experience.

The key to transparency and control lies in contextual business intelligence. The BI that we are familiar with is typically associated with long lag times and read-only dashboards that provide drill down capabilities at best. The BI for Big Data Applications is real-time and interactive. Each dashboard presents a snapshot view of live performance measured against KPIs and gives the business user controls over which KPIs to emphasize. The Big Data Application control dashboard gives users interactive feedback on how models are likely to perform when prioritizing one KPI over another. The dashboard also lets users decide which model experiments should be running and which are mandatory for a particular promotion.

This blog post series demonstrates why Big Data Applications built on an entity-centric Customer Data Platform are critical to today’s consumer facing organizations. Consumers are more engaged than ever and also have more choices and higher expectations for the level of personalization that businesses present to them. Still, a customer data repository is just the beginning. Customer Data Platforms built on frameworks such as Kiji give organizations a 360 degree view of their customers. In order to realize the value of this data, organizations must employ a real-time predictive model scoring, also available in Kiji, as part of a complete model lifecycle.

The only means of creating truly personalized experiences is to evaluate models based on each action a consumer takes, just as a trained sales associate is able to react to a live conversation. There is no single model that will serve all customers for all time. These models require consistent training, experimentation and evaluation. All of these capabilities must be integrated tightly into customer facing application channels such as websites, mobile apps, point of sale systems and call center terminals. Most importantly, business users need interactive, real-time transparency on model performance and control to ensure these models are inline with business strategy and goals.

If your organization is working with consumers, they are expecting a better experience and will spend time and money with companies that deliver on those expectations. The time to start realizing the full potential of Big Data is now and the key is to leverage Big Data Applications.

Business advice, Technology