Guest,Welcome to the Finance hub
Business Intelligence: Foretelling a future boom (Part I)
Article » Finance
Posted by : thedesk   May 19 2005
Hot careers: Datawarehousing/Business Intelligence
What is business intelligence?

The business environment today is getting more and more competitive. Customers are more sophisticated and selective, demanding higher levels of service, quality and customization. Because of globalization, competitors are bigger and more powerful. In such an environment, companies that work smarter have a competitive advantage. Rather than reacting to crises and opportunities, these organizations anticipate them. The secret weapon most of these smart organizations use is insightful information (not data) – more specifically, highly integrated information which empowers them with insights about what drives the business and how to optimize business processes to better meet strategic goals and objectives.

What has enabled organizations to derive insights is highly integrated information derived out of operational data. Large organizations have been collecting vast stores of operational data for years through the implementation of ERP, CRM, SCM, eCommerce applications etc. The challenge is not just to collect and store the data efficiently, but to be able to analyse and use it effectively. E.g., In an article on Wal-Mart's massive data collection, The New York Times notes that Wal-Mart has 3,600 stores, 100 mln customers weekly and 460 terabytes of customer data stored on the company mainframes.

Yet very few organizations extract more than a fraction of the information available – making it hard to get the integrated information needed for decision making. Companies have made so much investment into enterprise applications like ERP, CRM and SCM, the next logical step is to extract the maximum value from these deployments. In fact the term ‘Business Intelligence’ stands for the broad range of software and solutions aimed at collection, consolidation, analysis and providing access to information that allows users across the business to make better decisions.

The process of Business Intelligence (BI) is like that of a ‘data refinery’ and is a two stage process. A BI environment
1) first takes raw data from various sources and converts it into information, more specifically a data warehouse extracts data from multiple transaction or operational systems and integrates and stores the date in a dedicated database. E.g., a datawarehouse might merge customer records from multiple operational systems (e.g., orders, sales, service, complaints and loyalty programs) into a single file/datawarehouse or multiple data marts, which is typically a specialized multidimensional database optimized for query processing and report generation. This is the datawarehousing stage. This consumes almost 60-80% of the time of the technical team, because operational data is rarely clean, consistent or easy to integrate. The technical team needs to have a good idea of the underlying business
2) …And then converts the information into knowledge. Users, typically business users, use analytical tools to query, report, analyse, mine, visualize and act on the data in the datawarehouse. Their analysis identifies trends, patterns, and exceptions. Analytical tools enable users to convert information into knowledge. This is the analytical stage. This stage can include predictive Analysis, or Data Mining, the process of examining large amounts of data in search of hidden patterns and predictive information that allows organizations to make proactive decisions that ultimately improve efficiency and effectiveness

How is it useful?

Business intelligence basically allows companies to leverage their information assets as a competitive advantage. Some examples of its use are in:

1) Identifying cost-optimisation ideas, identifying key suppliers, track which products are hot and which are not to reduce inventory costs, etc.
e.g., It has enabled Wal-Mart to rapidly collect and integrate detailed sales information down to the Stock Keeping Unit (SKU) level. The ability to analyse near-real time data, enables the company to stock only those brands that sell within a particular region and allows managers to analyse the profitability of various products through both inventory turnover and gross margin, thereby improving overall profitability.

2) Allowing businesses to better understand the demand side of their businesses (analyzing customer behaviour) and uncover new business opportunities, optimize prices, identify more profitable customer segments, analyse behaviour of the most profitable customers, identify more profitable products, loyalty analysis, campaign analysis, etc. This can be done through Business Intelligence solutions as customer information can be consolidated throughout the company, Sales Information, contact history, customer service information, channel preference, revenue history, product preference, demographic data, etc. all brought together. Analysis then can be done on this consolidated data to produce meaningful customer intelligence.

3) For detailed financial analysis like P&L analysis, budget analysis, cash flow analysis, accounts receivable aging analysis, etc. and help with regulatory compliance with Sarbanes-Oxley Act etc.

Why is it a hot career?

From 1992 to 2000 companies made enormous investments in technology. Through these investments companies have collected huge amounts of transactional data. Now businesses are asking, “How do we leverage these investments?”. With business becoming more and more competitive, those companies which leverage, exploit and maximize their information assets have a strategic advantage over their competitors. In the US, "Since last June, there's been an upswing in full-time placements and temp help relating to business intelligence within Fortune 100 firms," according to Daniel Barber, president of Niche Technologies, which works with 65 of the 100 largest companies. The pickup in hiring demand has been especially acute in finance, marketing forecasting and sales, says Barber, with financial and consumer services companies being particularly aggressive in expanding IT manpower and building out BI deployments involving data warehousing.

In March 2005, Gartner forecast that BI-related employment in the US, will triple at large firms by 2008. A survey from the firm found that 39 percent of North American respondents expect to increase spending on business intelligence this year, while another 34 percent plan to spend the same amount this year as they did last year. Gartner's survey canvassed 917 business and IT respondents from several industries in North America, Europe and Asia. A Forrester Research survey of 1,368 technology decision makers in North America and Europe found they plan to boost IT spending 3.9 percent overall. Regulatory concerns and an increasing quantity of data kept business intelligence in the top spot in planned purchases, at 9%.

Even a look at the financial results of some of the Business Intelligence vendors shows that growth is healthy. Sales at Business Objects, increased by 65% in 2004 as compared to 2003. Sales at SAS, the privately held business intelligence and analytics giant, reached $1.53 billion in 2004, up 15 percent. SAS pointed to healthy demand for its SAS 9 intelligence product, which has shipped to 20,000 new and existing customers, according to the company. The story's similar at MicroStrategy, Hyperion and Informatica, among others. MicroStrategy registered 39 percent sales growth in the fourth quarter of 2004. Hyperion's sales grew 13 percent year-over-year in its most recent quarter. Informatica reported a 7 percent revenue boost in the final period of 2004.

What are the stages in the implementation process and what are the technologies/tools used:

Stage I: Directing & Planning:
The business intelligence process begins when business leaders generate the questions that will help them achieve their objectives. E.g., Who are my most profitable customers? What is the gross margin for each product line? The business needs are used to develop specific requirements and in giving planning and direction to guide information gathering and formulation of answers. The directing and planning phase is an iterative process. It involves drawing up specific requirements and getting answers to specific questions, the answered questions lead to more questions.
E.g., The head of (home loans) mortgage for a financial services firm asks the question, “How many mortgages are sold to existing customers?” The analytic team for the mortgage group is asked to answer this question – starting the Business Intelligence process.

Stage II: Information Gathering:
There are many sources of information within companies today. Transactional data is stored in: Point of Sale, ERP, CRM, SCM, Customer Service applications, etc. The different systems create, process and store different pieces of information each and every day. This process is continuous. The Information Gathering process is where the different sources are examined to determine the necessary data sources to obtain data to answer the questions.
This data is stored in OLTP (Online Transaction Processing) databases. These databases organize data efficiently to serve their main function, which is capturing and updating high volumes of information. These systems are typically not designed to show trends or to perform multidimensional analysis, which is the exploration of summary information across many different key factors at the same time. Also, the language used to access and update data with operational systems is called structured query language (SQL). SQL is designed for operational systems, where data is stored in two-dimensional arrays or tables in an underlying RDBMS or relational database management system. However, there are inherent limitations with SQL when used to view data needed for multidimensional analysis.
E.g., The team begins by understanding what information they need to answer the question. The data they need is being captured throughout the company; they just need to determine where it is and how to get it. At this point, IT may become involved to help identify the appropriate data sources.

Stage III: Design and Data Integration:
The Data Integration phase is the integration of the raw data into a useable format for analysis. This can be the design and creation of a new database, addition of data into an existing database or the consolidation in some other type of analysis system (i.e., utilizing an analytic tool to extract and manipulate data). This data is extracted from operational systems and generally stored in datawarehouses or data marts. Thus this phase involves the design of these datawarehouses as well as the extraction, transformation and loading (ETL) of data into these datawarehouses/datamarts. These are OLAP (online Applications Processing) databases. OLAP applications enable multidimensional views of data, and usually, have better calculation-intensive capabilities and time intelligence. Because OLAP applications are multidimensional, the Business Intelligence industry uses a term called ‘Data Cubes’ or ‘OLAP Cubes’, a cube aggregates the facts in each level of each dimension in a given OLAP schema.

This phase can generally be thought of as the Extract, Transform and Load (ETL) processing that occurs within Business Intelligence environments. This phase is a complicated and time consuming phase because many times the quality of base data is not very good.
E.g., Once the data sources are identified, the team works (likely in partnership with IT) to extract and integrate the information.
Some Technologies/Products/Vendors: Informatica, Ascential datastage, Ab initio, Hummingbird, Hyperion, Cognos, Business Objects (recently acquired ACTA) etc.

Stage IV: Analysis & Production:
The analytic group leverages data mining tools and techniques to sort through the data and create the intelligence. Data mining is the analysis of data with the intent to discover gems of hidden information in the vast quantity of data that has been captured in the normal course of running the business.

The end result of this phase is the production of “intelligent” answers, in proper context, with supporting commentary. In some cases this is as simple as creating a report. In other cases, it is a detailed assessment of predictive indicators for cross-selling campaigns. Additional requirements may also be generated during this phase as analysts identify other pertinent questions to be answered.
E.g., Utilizing the data, the analytic team will look at the mortgages sold and identify those who were sold to existing customers – those individuals who currently own one or more products.
Some Technologies/Products/Vendors: SAS, SPSS, Business Objects, Microstrategy, Siebel Analytics etc.

Stage V: Dissemination:
The dissemination phase is the delivery of intelligence products to the consumers who request them. This typically involves the use of a Business Intelligence tool to publish an scorecard or executive dashboard (provide "at-a-glance" information about business performance across the enterprise), standard reports or the ability to actively review the data. This is used for intelligence that is self-explanatory.
E.g., The answer is then provided back to the consumer (head of mortgage) who, in reviewing the answer, now asks, “How do we sell more mortgages to existing customers?” And the next iteration of the process is born. The analytic team then moves back to understanding the attributes of mortgages holders who have other products and can identify prospects for cross-selling the mortgage product.
Some Technologies/Products/Vendors: SAS, Business Objects (which acquired Crystal Decisions for US$1.2 billion last year), Hyperion(which gobbled up Brio Software for US$142 million also last year), Microstrategy etc.

Which kind of organizations would be hiring people for implementation?
Consulting and implementation companies, Business Intelligence product vendors and enterprises/companies that buy such solutions will hire people for implementation.

Consulting and implementation companies will include software services firms which deal with working on hetrogenous platforms and mix of technologies. These are Firms which have a business consulting model and are not focused on the choice of technology they want to use in a project.

Because of the strategic nature of the information obtained from the Business Intelligence application, organizations would probably like to have internal people playing strategic roles in the implementation process.

Telecom, BFSI and Manufacturing verticals and consumer service companies are the ones which are more likely to go for implementations in BI/DW.

Please look out for Part II of this article on ‘Hot Careers: Datawarehousing/Business Intelligence’, which will cover the kind of manpower which is required in an implementation as well as some typical jobs descriptions for jobs in this area.