Big Data & Analytics - the full view (upon request)

Upon request, I have put together the five parts of the previous published Big Data series and combined into one documents. This offers you to read everything in one place. Please share your thoughts and best practices with me. You always may email me directly to alexwsteinberg@gmail.com
Big Data Series - Part 1   Technical challenges
Big Data requires to learn much about data as an asset and analytics. Data is the most precious asset in an organization, the currency of the enterprise.
Companies’ data ecosystems have become complex and littered with silos. A large majority of companies is still not able to make full use of Big Data advantages.
There are many challenges with Big Data: Lack of knowledge, varying definitions & expectations, different views about data sources and use cases, ignorance about valuable data sources, technologies, etc.
Companies must understand data across the entire data supply chain and their individual stages: Identifying & leveraging data sources, importing, enhancement of data value, combination with other data, generation of insight, and taking of specific actions.
This means: companies must mobilize data across the enterprise; deeply understand, analyze and determine value of respective data; understand business use case and data patterns to determine appropriate actions.
It requires companies to commit to continuous discovery, experimentation, testing, learning, adapting and innovation.
There are many approaches, solutions and technologies presently offered in the Big Data domain and quickly evolving. Companies need to be aware of the different options and their pros & cons to combine those to an overall solution.
Continue part 2 out of 5  
Big Data Series – Part 2 - Traditional data approaches not enough anymore
Given the varying types, sources and sheer size of data today the traditional approach of collecting data in a staging area, transforming into desired format, loading in mainframe/ data ware house and then delivering requested data to users on a point by point query does not work well any more.
Companies must perform calculations, run simulations models, compare statistics at fast speed to generate insights. Real-time analytical tools able to pre-process streaming data and correlate data from internal and external sources, offer interesting opportunities, but also complex challenges.
Data acceleration enables massive amounts of data to be ingested, processed, stored, queried and accessed much faster. It ensures multiple ways for data to come into the company’s data infrastructure and be referenced fast.
Data acceleration leverages hardware and software power through clustering and helps correlate different data sources, including localization. It improves interactivity by enabling users and applications to connect to the data infrastructure in universally accepted ways and ensuring that user queries are delivered as quickly as required.
Continue part 3 out of 5
Big Data Series – Part 3 Six technology components for Data Acceleration
There are at least six key technology components to build a supporting architecture: Big Data platforms, Ingestion solutions, Complex event processing, In-memory databases, Cache clusters and Appliances. Each component helps with data movement (from source to where needed), processing and interactivity (the usability of the data infrastructure).
Big Data platform (BDP)
BDP is a distributed file system and compute engine. It contains a big data core, a computer cluster with distributed data storage and computing power. Replication and sharding partitions very large databases into smaller, more easily to manage parts in order to accelerate data storage.
Newer additions enable more powerful use of core memory as a high-speed data store. These improvements allow for in-memory computing. Streaming technologies added to the core can enable real-time complex event processing. In-memory analytics support better data interactivity.
Further enhancements to the big data core create fast and familiar interfaces with data on the cluster. The core stores structured and unstructured data, but requires map/reduce functionality to read. Query engine software enables the creation of structured data tables in the core and common query functionality (SQL etc.)
Ingestion
Collecting, capturing and moving data from its sources to underlying repositories used to be done traditionally through the extract, transform and load ETL method. Today the priority is not the structure of the data as it enters the system, but assuring that all data is gathered covering different increasing data types & sources and quickly transported to areas where it can be processed by users. Ingestion solutions cover both static and real-time data. The data the gathered by the publisher and then send to a buffer/ queue, where the user can request the data.
Complex Event Processing (CEP)
After data ingestion the CEP is responsible for preprocessing and aggregation (& triggering events). It tracks, analyzes and processes data of events and derives conclusions. CEP derives data from multiple sources and combines historic as well as fresh data in order to infer patterns and to understand complex circumstances. Its engines pre-process fresh data streams from its sources, expedite processing of future data batches, match data against pre-determined patterns and trigger events based on detected patterns.
CEP offers immediate insight and enables fast action taking. In-memory computation allows to run Data movement and processing in parallel, increasing speed. CEP solutions add computing power by processing the data before it is submitted to the data stores or file systems.
In-memory databases (IMDB)
IMDBs are faster than traditional databases, because they use simpler, internal algorithms and executive fewer central processing unit instructions. The database is preloaded from disk to memory. Accessing data in memory eliminates the seek-time involved in querying data on disk storage. The applications communicate through SQL, which receives records in the RAM and triggers the query optimizer.
IMDBs constrain the entire database to a single address space. Any data can be accessed within microseconds. The steadily falling RAM prices favor this solution.
Cache Clusters
They are clusters of servers in which memory is managed by a central software designed to transfer the load from upstream data sources (databases) to applications and users. They are typically maintained in-memory and can offer fast access to frequently accessed data. They sit between the data source and the user. Traditionally they accommodate simple operations such as reading and writing values. They are populated when a query is sent from a data user to the source. Prepopulating data into a cache cluster of frequently accessed data improves response time. Data grids can take caching a step forward by supporting more complex queries and using massive parallel processing (MPP) computations.
Appliance
Massive parallel processing sits between data access and data storage. Appliance here is a pre-configured set of hardware and software including servers, memory, storage, input/output channels, operating systems, DBMS, admin software and support services.
It may have a common database for online transactions and analytical processing, which improves the interactivity and speed. Appliances can perform complex processing on massive amounts of data.
Implementing and maintaining high performance data bases on clusters is challenging and few companies have the necessary expertise to do so themselves.
Custom-silicon circuit boards enable to develop their specific solutions. It enables development on devices for specific use cases and allows for network optimization (integrating embedded logic, memory, networking and process cores). This plug and play functionality offers interesting possibilities.
Continue part 4 out of 5
Big Data Series – Part 4 Creating a suitable Technology Stack/ Solution
All of these components bring their individual technology features. Companies must wisely put together an overall solution from among those components, leveraging their complementary advantages and customizing those to their particular needs.
There are four fundamental technology stacks (with their variations) offer possible solutions:

1. Big data core only or with enhancements (with complex event processing, with in-memory database, with query engine or with complex event processing and query engine)
   • This technology is the de-facto standard for exceptional data movement, processing and interactivity.
   • Data usually enters the cluster through batch or streaming.
   • Events are not processed immediately, but in intervals. Enables parallel processing on large data sets, and thus advanced analytics.
   • Applications and services may access the core directly and deliver improved performance of large, unstructured data sets.
   • Adding CEP enhances big data core processing capabilities, real-time detection of patterns in data and trigger events. Enables real-time animated dashboards. Could add machine learning program to the CEP.
   • IMDB can further increase computing power through placing key data in RAM.
   • Query engines can further open interfaces for applications to access big data even faster.
2. In-memory data base (IMDB) cluster only or with enhancements (with Big Data Platform, with complex event processing)
   • External data is streamed in or transferred as bulk to the IMDB
   • Users and applications can directly query the IMDB, usually through SQL like structures.
   • The incoming data is first pre-processed through the BDP before it goes to the IMDB
   • In case of CEP, the CEP first ingests the data; the processing is then done in the IMDB and then returned to the application for faster interactivity.
3. Distributed Cache only or with enhancement (with Application and Big Data platform)
   • A simple caching stack sitting atop of the data source repository. The application retrieves the data. The most relevant data subset is placed in the cache.
   • Processing of the data falls to the application (may result in slower processing speeds)
   • If BDP, the BDP ingests the data from the source and does the bulk of the processing, then puts data subset in cache.
4. Appliance only or with enhancement (with Big Data platform)
   • Data streams directly into the appliances; the application talks directly to the appliance
   • If BDP, the BDP ingests and processes data. The application can directly talk to the appliance for queries.

Continue part 5 out of 5

Big Data Series – Part 5 – 12 Immediate suggestions to build a data supply chain

• Consider data as perhaps the most important asset in your organization. Become data driven. Some people call it “data religious”.
• Research about Big data & Analytics best practices. It requires continuous learning. Refer to the different approaches offered in previous blogs (Data Acceleration Part 1 and 2).
• Do an inventory of existing data. Focus on most frequently accessed and time-relevant data.
• Identify, simplify and optimize inefficient data processes. Eliminate manual, time-consuming data curation processes (such as tagging and cleaning).
• Identify currently unmet business needs and develop solutions.
• Identify and overcome data silos.
• Simplify and standardize data access through a robust data platform
• Build an effective technology stack using one of the four suggested options while leveraging some of the described six components (Data Acceleration Part 1 and 2).
• Further explore API management, traditional middleware, PaaS and other possibilities
• Analyze current internal data sources and look for still hidden sources. Explore external sources to increase quantity and quality of available data.
• Identify and improve individual data supply chain streams
• Develop a systematic roadmap for building an effective overall data supply chain

Special thanks to Accenture Technology Labs and Analytics Group, whose thought leadership, best practices and white papers have served as inspiration and knowledge source for this Big Data series.

+++
To share your own thoughts or other best practices about this topic, please email me directly to alexwsteinberg (@) gmail.com.

Alternatively, you also may connect with me and become part of my professional network of Business, Digital, Technology & Sustainability experts at

https://www.linkedin.com/in/alexwsteinberg   or
Xing at https://www.xing.com/profile/Alex_Steinberg   or
Google+ at  https://plus.google.com/u/0/+AlexWSteinberg/posts

Big Data Series – Part 5 – 12 immediate suggestions to build a data supply chain

Immediate suggestions to build a respective data supply chain

-         Consider data as perhaps the most important asset in your organization. Become data driven. Some people call it “data religious”.

-          Research about Big data & Analytics best practices. It requires continuous learning. Refer to the different approaches offered in previous blogs (Data Acceleration Part 1 and 2).

-          Do an inventory of existing data. Focus on most frequently accessed and time-relevant data.

-          Identify, simplify and optimize inefficient data processes. Eliminate manual, time-consuming data curation processes (such as tagging and cleaning).

-          Identify currently unmet business needs and develop solutions.

-          Identify and overcome data silos.

-          Simplify and standardize data access through a robust data platform

-          Build an effective technology stack using one of the four suggested options while leveraging some of the described six components (Data Acceleration Part 1 and 2).

-          Further explore API management, traditional middleware, PaaS and other possibilities

-          Analyze current internal data sources and look for still hidden sources. Explore external sources to increase quantity and quality of available data.

-          Identify and improve individual data supply chain streams

-          Develop a systematic roadmap for building an effective overall data supply chain

 Special thanks to Accenture Technology Labs and Analytics Group, whose thought leadership, best practices and white papers have served as inspiration and knowledge source for this Big Data series.

Big Data Series – Part 4 Creating a suitable Technology Stack/ Solution

All of these components bring their individual technology features. Companies must wisely put together an overall solution from among those components, leveraging their complementary advantages and customizing those to their particular needs.

There are four fundamental technology stacks (with their variations) offer possible solutions:

1.       Big data core only or with enhancements (with complex event processing, with in-memory database, with query engine or with complex event processing and query engine)

o   This technology is the de-facto standard for exceptional data movement, processing and interactivity.

o   Data usually enters the cluster through batch or streaming.

o   Events are not processed immediately, but in intervals. Enables parallel processing on large data sets, and thus advanced analytics.

o   Applications and services may access the core directly and deliver improved performance of large, unstructured data sets.

o   Adding CEP enhances big data core processing capabilities, real-time detection of patterns in data and trigger events. Enables real-time animated dashboards. Could add machine learning program to the CEP.

o   IMDB can further increase computing power through placing key data in RAM.

o   Query engines can further open interfaces for applications to access big data even faster.

2.       In-memory data base (IMDB) cluster only or with enhancements (with Big Data Platform, with complex event processing)

o   External data is streamed in or transferred as bulk to the IMDB

o   Users and applications can directly query the IMDB, usually through SQL like structures.

o   The incoming data is first pre-processed through the BDP before it goes to the IMDB

o   In case of CEP, the CEP first ingests the data; the processing is then done in the IMDB and then returned to the application for faster interactivity.

3.       Distributed Cache only or with enhancement (with Application and Big Data platform)

o   A simple caching stack sitting atop of the data source repository. The application retrieves the data. The most relevant data subset is placed in the cache.

o   Processing of the data falls to the application (may result in slower processing speeds)

o   If BDP, the BDP ingests the data from the source and does the bulk of the processing, then puts data subset in cache.

4.       Appliance only or with enhancement (with Big Data platform)

o   Data streams directly into the appliances; the application talks directly to the appliance

o   If BDP, the BDP ingests and processes data. The application can directly talk to the appliance for queries.

Continue part 5 out of 5

Big Data Series – Part 3 Six technology components for Data Acceleration

There are at least six key technology components to build a supporting architecture: Big Data platforms, Ingestion solutions, Complex event processing, In-memory databases, Cache clusters and Appliances. Each component helps with data movement (from source to where needed), processing and interactivity (the usability of the data infrastructure).

Big Data platform (BDP)

BDP is a distributed file system and compute engine. It contains a big data core, a computer cluster with distributed data storage and computing power. Replication and sharding partitions very large databases into smaller, more easily to manage parts in order to accelerate data storage.

Newer additions enable more powerful use of core memory as a high-speed data store. These improvements allow for in-memory computing. Streaming technologies added to the core can enable real-time complex event processing. In-memory analytics support better data interactivity.

Further enhancements to the big data core create fast and familiar interfaces with data on the cluster. The core stores structured and unstructured data, but requires map/reduce functionality to read. Query engine software enables the creation of structured data tables in the core and common query functionality (SQL etc.)

Ingestion

Collecting, capturing and moving data from its sources to underlying repositories used to be done traditionally through the extract, transform and load ETL method. Today the priority is not the structure of the data as it enters the system, but assuring that all data is gathered covering different increasing data types & sources and quickly transported to areas where it can be processed by users. Ingestion solutions cover both static and real-time data. The data the gathered by the publisher and then send to a buffer/ queue, where the user can request the data.

Complex Event Processing (CEP)

After data ingestion the CEP is responsible for preprocessing and aggregation (& triggering events). It tracks, analyzes and processes data of events and derives conclusions. CEP derives data from multiple sources and combines historic as well as fresh data in order to infer patterns and to understand complex circumstances. Its engines pre-process fresh data streams from its sources, expedite processing of future data batches, match data against pre-determined patterns and trigger events based on detected patterns.

CEP offers immediate insight and enables fast action taking. In-memory computation allows to run Data movement and processing in parallel, increasing speed. CEP solutions add computing power by processing the data before it is submitted to the data stores or file systems.

In-memory databases (IMDB)

IMDBs are faster than traditional databases, because they use simpler, internal algorithms and executive fewer central processing unit instructions. The database is preloaded from disk to memory. Accessing data in memory eliminates the seek-time involved in querying data on disk storage. The applications communicate through SQL, which receives records in the RAM and triggers the query optimizer.

IMDBs constrain the entire database to a single address space. Any data can be accessed within microseconds. The steadily falling RAM prices favor this solution.

Cache Clusters

They are clusters of servers in which memory is managed by a central software designed to transfer the load from upstream data sources (databases) to applications and users. They are typically maintained in-memory and can offer fast access to frequently accessed data. They sit between the data source and the user.  Traditionally they accommodate simple operations such as reading and writing values. They are populated when a query is sent from a data user to the source. Prepopulating data into a cache cluster of frequently accessed data improves response time. Data grids can take caching a step forward by supporting more complex queries and using massive parallel processing (MPP) computations.

 Appliance

Massive parallel processing sits between data access and data storage. Appliance here is a pre-configured set of hardware and software including servers, memory, storage, input/output channels, operating systems, DBMS, admin software and support services.

It may have a common database for online transactions and analytical processing, which improves the interactivity and speed.  Appliances can perform complex processing on massive amounts of data.

Implementing and maintaining high performance data bases on clusters is challenging and few companies have the necessary expertise to do so themselves.

Custom-silicon circuit boards enable to develop their specific solutions. It enables development on devices for specific use cases and allows for network optimization (integrating embedded logic, memory, networking and process cores). This plug and play functionality offers interesting possibilities.

Continue part 4 out of 5

Big Data Series – Part 2 - Traditional data approaches not enough anymore

Given the varying types, sources and sheer size of data today the traditional approach of collecting data in a staging area, transforming into desired format, loading in mainframe/ data ware house and then delivering requested data to users on a point by point query does not work well any more.

Companies must perform calculations, run simulations models, compare statistics at fast speed to generate insights. Real-time analytical tools able to pre-process streaming data and correlate data from internal and external sources, offer interesting opportunities, but also complex challenges.

Data acceleration enables massive amounts of data to be ingested, processed, stored, queried and accessed much faster. It ensures multiple ways for data to come into the company’s data infrastructure and be referenced fast.

Data acceleration leverages hardware and software power through clustering and helps correlate different data sources, including localization. It improves interactivity by enabling users and applications to connect to the data infrastructure in universally accepted ways and ensuring that user queries are delivered as quickly as required.

Continue part 3 out of 5

Big Data Series - Part 1 Technical challenges

Big Data requires to learn much about data as an asset and analytics. Data is the most precious asset in an organization, the currency of the enterprise.

Companies’ data ecosystems have become complex and littered with silos. A large majority of companies is still not able to make full use of Big Data advantages.

There are many challenges with Big Data: Lack of knowledge, varying definitions & expectations, different views about data sources and use cases, ignorance about valuable data sources, technologies, etc.

Companies must understand data across the entire data supply chain and their individual stages: Identifying & leveraging data sources, importing, enhancement of data value, combination with other data, generation of insight, and taking of specific actions.

This means: companies must mobilize data across the enterprise; deeply understand, analyze and determine value of respective data; understand business use case and data patterns to determine appropriate actions.

It requires companies to commit to continuous discovery, experimentation, testing, learning, adapting and innovation.

There are many approaches, solutions and technologies presently offered in the Big Data domain and quickly evolving. Companies need to be aware of the different options and their pros & cons to combine those to an overall solution.

Continue part 2 out of 5    

Applications for competitive advantage - recommended actions

Applications have become a key driver of strategy, innovation and competitive differentiation. Applications are a crucial gateway to seamless customer experience, new services, and revenue streams. Application and business strategies need to align and merge over time.

Companies need to become software driven businesses. They require a new mind set and way of working, an overall organizational approach to business, IT and applications.

Many opportunities and much to be done

A new IT/ OT operating model needed

IT and Operational Technology must increasingly integrate in the age of IoT. Companies require a new IT operating model and a way of how they design, build, use and manage software. Enabling software in itself is becoming a revenue generating product.

The cloud can help mesh together the formerly inaccessible enterprise and machine generated data. It will help bring different business functions even closer together.

New software development thinking

Traditional coding of applications with complex, lengthy implementation cycles does not meet the business requirements any longer. Companies require modular architectures. They need to use next generation integration techniques, driven by a mobile-first, cloud-first mind set.

Applications need to be quickly assembled out of existing, small, reusable components leveraging modular architectures.

Leverage available technologies

Massive amounts of available data, processing power, natural language learning, cognitive computing and machine learning, rule-based algorithms and other advances in data science call for embedding Software Intelligence directly in the applications or processes.   

Intelligent automation helps achieve major productivity increases, minimize errors and throughput time. It also can effectively support and enhance humans in higher quality work. Integrated Analytics enables Applications to analyze, comprehend and take appropriate actions independently.

Digital Agents, enabled by self-governance, are already serving customers. The artificial intelligence company IPsoft has already deployed an effective digital help desk application that can understand human language in 10 languages, search knowledge and databases and respond to specific customer questions within seconds.

Develop better software

The technological advancement enables also of how software can be developed. Test automation tools can use cognitive computing and robotics to generate test artifacts (scenarios, conditions and results) based on plain text functional requirements.

Post deployment tools for service operations can continuously accelerate problem resolution by curating specialized application knowledge and leveraging descriptive analytics and natural language processing.

Applications can and should now automate routine tasks, improve business processes through integrated analytics und ultimately govern themselves.

Leverage Agile, DevOps and other agile engineering techniques!

The importance of APIs

New Application Programming Interfaces (APIs) must enable flexible, efficient exchange of internal and external software components and services. The Internet of Things will bring together all current IT devices with technology equipment, sensors and other devices.  Products will increasingly turn into product-service hybrids.

Support Big Data & Analytics correctly

Extracting the biggest value from Big data & Analytics, requires the right structures, processes and components across the value chain. Big data architectures need to be embedded within the business processes and applications, not at alongside.

Re-work IT systems

One of the barriers is the often monolithic nature of IT systems. Astonishingly, 70 percent of all business transaction still happen in COBOL. Enterprise architecture must support platform integration capabilities, security, API lifecycle management and monitoring. Virtualization, abstraction, simplification, separation of technical and business logic, modularization, componentization and containerization are effective techniques.  

As part of an ecosystem, the individual player need to work together to mitigate the risks of connected applications. Obsolete and legacy applications carry limited or now security built in and must be tightly managed.

Unfortunately, while key technologies are advancing at massive speed, business processes and applications lag behind.

Use IT across the organization

On a larger scale the formerly stand-alone IT function need to be integrated in and fully used by the individual business functions across the organization and value chain.

Integration, Orchestration and Business Process Management services will help configure applications customized to business needs at an ongoing basis.

+++
To share your own thoughts or other best practices about this topic, please email me directly to alexwsteinberg (@) gmail.com.

Alternatively, you also may connect with me and become part of my professional network of Business, Digital, Technology & Sustainability experts at

https://www.linkedin.com/in/alexwsteinberg   or
Xing at https://www.xing.com/profile/Alex_Steinberg   or
Google+ at  https://plus.google.com/u/0/+AlexWSteinberg/posts

From Big Data to Smart Data – A practical 11 step cycle to implement

There is much excitement, hype and delusion around Big Data. Everybody is aware of opportunities, but high aspirations are often disappointed. A large number of Big Data projects have missed on meeting expectations.
There are many reasons. One key explanation that companies approach Big Data wrong. Overwhelmed by the massive amount of data (available or imagined), companies through ever increasing computing power and data bases at the growing problem.
I suggest a different approach. Moving from Big Data to Smart Data: Thinking first, then collecting and processing data.
Enclosed I offer an 11 step process that works as endless repeating cycle of planning, execution, learning and improvement:

1. Ask the right questions based on your objectives and develop sensible hypothesizes: focus on business problems and explore opportunities across the value chain; identify ways to leverage digital technologies
2. Collect, create and acquire the right, necessary data prioritized by objectives and head maps)
• Generate data oneself
• Transaction data (Turn over and dynamic turnover sales data, usage and service data)
• Behavior data (customer profiles and core customer data)
• Other data: customer contacts with/ without transactions, customer requests, campaign reactions, movement profiles, social media relationships, benchmarking, need tree structures
• Own databases are often spread silos, but contain much valuable data!
• Gather free or cheap market data through surveys, etc.
• Buy from external sources
• Obtain through exchange with partners. Understand additional customer needs.
• Identify cooperation partners: Suppliers, retailers, and other companies across industries; build long-term network to share data
• Conduct competitive analysis (customer turnover data of competitors)
• Do additional research (total customer demand; promoters across the customer journey)
3. Understand the customer. Combine the often different, complementary and conflicting views from strategic marketing, Product marketing, Customer services, Sales and Logistics and develop one picture. Unite all respective parties across the organization that should be involved to create a commonly agreed pool of socio-demographic, psychographic and transactional data. This allows for an integrated segmentation later. Do multi-variant analysis of customer transaction data as starting point.
4. Segment customers in ideally groups of homogeneous consumer behavior. They should have the same decision parameters for consumption & purchase. Groups should be well distinguishable and separate. The naming and context should be clear and agreed across the entire organization. Apply hierarchical, partitioned and fuzzy cluster analysis or density based techniques.
5. Further correlate and refine segments. Use power questions to further segment. Test and refine. Be aware that changes to segments will need to occur as objectives and priorities change.
6. Focus on 20/80 rule – 20 percent of data generates about 80 percent of insight. Companies have hardly ever all data that they need. And it is not necessary. Companies can get started with available data by looking where there are the biggest results for the effort. A good example is lost-order analysis.
7. Refine USP by understanding customer needs, channel preferences, purchase drivers, share of wallet with new and current customers, campaign selection, offer customizations, improvement of algorithms, enable machine learning.
8. Determine and take action according to specific segments. Test different offering options and improve accordingly.
9. Manage channel mix. Use Envelope Analysis to optimize output level in relation to input variables.
10. Understand and manage customer journey:
• Put yourself fully in the shoes of the customers and define all interaction from their perspective, not company perspective.
• Understand all contact points and the individual experiences of customers at each touch point
• Prioritize customer touch points and take actions. Here, often decides success and failure of companies.
• Identify and establish connections between touch points; improve to speed up the movement from one touch point to the other
• Identify purchase promoters
• Align marketing activities accordingly
• Deploy special techniques such as associated analysis and collaborative filtering to suggest customized product and service recommendations to prospective buyers.
• Establish and manage metrics
11. Learn and refine begin a new circle

+++
To share your own thoughts or other best practices about this topic, please email me directly to alexwsteinberg (@) gmail.com.

Alternatively, you also may connect with me and become part of my professional network of Business, Digital, Technology & Sustainability experts at

https://www.linkedin.com/in/alexwsteinberg   or
Xing at https://www.xing.com/profile/Alex_Steinberg   or
Google+ at  https://plus.google.com/u/0/+AlexWSteinberg/posts

Predictive Analytics with Big Data - Challenges & solutions

Business analytics and Big Data help improve Customer Experience (precise customer segmentation , interaction & servicing: increased loyalty & retention, Operational Efficiency (Increase transparency, Resource Optimization, Process Quality and Performance) and developing new business modes (expanding existing or generating new revenue streams).

Data becomes big data when its variety, volume or velocity exceeds the ability of traditional IT systems to ingest, store, analyze and process this data. Big data requires often a both technical and cultural change.

Traditional tools work on enterprise data captured in the data warehouses. Additional statistical analysis, data mining, text mining and predictive analytics takes now usually place on separate, dedicated servers. The process of exporting and creating copies on external servers is time consuming and becomes infeasible when data amounts become too large.

So let us talk about Prescriptive Analytics in relationship with Big Data.

Available tools make predictive analysis increasingly manageable. It is not only domain for data scientist any longer. One click predictive modelling automatically run a series of algorithms on the data and select the one with highest accuracy.

A number of challenges still remain and require respective actions:

-          Explore and discover what data you really have and how these data sets relate to each other.

-          Develop insight through a process of experimentation and iteration you gradually.

-          Mine the data to discover patterns and relationships

-          Determine how such data relates to the traditional enterprise data

-          Simplify the process to implement and automate the necessary actions.

-          Minimize data movement to conserve computing resources (ETL architecture becomes with less efficient with increasing data amounts)

-          Use intuitive discovery, BI tools and In-database analytics; use Hadoop for pre-processing data to identify macro-trends and special information (such as out of range values)

-          Enable decision making and informed action based on predictive modelling, business rules and self-learning.

A systematic step-by-step view on process can help companies:

1.       Identify and gather data relevant to the business goal from a variety of sources across data silos in enterprise applications and external sources (social media, public, licensed). Use visualization tools to ease work.

2.       Prepare the data. Integrate and enrich into an analytical data set: Calculate aggregate fields, merge multiple data sources, fill missing data, strip extraneous characters, etc.

3.       Build predictive model using statistical and machine learning algorithms (depending on type/ completeness of data available and level of prediction desired). Run analysis on training data and use model to predict test data set.

4.       Evaluate and assure predictive model is effective and accurate. It must predict the test data set.

5.       Use model in applications and deliver actionable prescription to business (predict opportunity/ avoid negative event)

6.       Monitor, improve and update model (adjust parameters of algorithms, add new/ more data)

There are proprietary and open source programming tools. The Open Software Community is strongly driving predictive analysis. The open source programming language R is a widely used across the industry. API libraries in Python, Java and Scala are available. Many BI platforms (Accenture, Deloitte, Infosys, etc.) already include some predictive analytics capabilities.

IBM, SAS and increasingly SAP are the clear leaders in predictive analysis tools.

IBM has the most comprehensive set to build models, conduct analysis and deploy predictive applications both on-premises and in the cloud. SAS provides data scientists with an all-in-one visualization and predictive analytics solution, integrated with R, Python and Hadoop. Other providers are RapidMiner, Alteryx, Oracle, Alpine Data Labs, Angoss, Dell, Fico and Knime and Microsoft Azure Machine Learning.

One major problem still remains: Much time and effort needs to be spend in the data preparation (30 to up to 60 percent) when using data from data ware houses. A main reason is that data is often stored without context. The process integrating data from multiple databases become very complex. Modelling the context takes another 20 to 30 percent.  The following chart explains:

Traditional companies such as SAS, IBM, Oracle and Cognos try to solve the problem leveraging their computing resources and throwing “brute force” at it.

Another option that online retailers and credit card companies use is to build applications that store their own transactional context and then process that data in batch after execution. The difficulties are: data volumes become large and logging difficult (storage, overhead for the application, etc.). Difficult to gain value from the data in real time. Still significant post-processing occurs. Often it is not feasible to enable already to existing applications.

Another interesting option, proposed by OpTier, would be to create transaction context through a third party software application and build a single stream of data from multiple sources. This is still an area that requires more research.

+++
To share your own thoughts or other best practices about this topic, please email me directly to alexwsteinberg (@) gmail.com.

Alternatively, you also may connect with me and become part of my professional network of Business, Digital, Technology & Sustainability experts at

https://www.linkedin.com/in/alexwsteinberg   or
Xing at https://www.xing.com/profile/Alex_Steinberg   or
Google+ at  https://plus.google.com/u/0/+AlexWSteinberg/posts