No Batteries Required - Home
Business Intelligence and Big Data - Part I
Ray Kahn
FEB 25, 2013 09:02 AM
A+ A A-

The foundation of an effective corporate strategy is a solid understanding of the competitive landscape. Organizations, like humans, evolve; they are never static as the ecosystem they operate in undergoes constant change. And in order to thrive they too must adapt: thus the need for Business Intelligence (BI).

BI is a discipline, and like all disciplines it has a definition: It is a set of business processes, technologies and tools that help corporations analyze and understand their performance by using data, lots and lots of data.  BI encompasses an array of technologies, processes and resources that assist corporate managers and executives with making better decisions quicker. BI uses variety of sources of data – internal and external - data mining, analytics and a rich UI to construct a visual representation of factors affecting an organization’s operating environment including customers, finances, operations and capabilities. This information, theoretically, would then help executives discover new opportunities in product developments, cost containment and process improvement, to name a few.  BI is not a panacea however; it is a tool and like all tools the input matters greatly.

Lots of Data = New Challenges

BI is a consumer of data, and lots of it – AKA Big Data. Big Data refers to amounts of data that can’t be stored in conventional storage systems and RDBMS and which requires distributed parallel computing to process and manage (see Technologies section). The amount of data I am talking here could reach petabytes or even exabytes; really big data.

Big Data of courses creates a whole set of new challenges for IT. For one IT will need to take into consideration strains on the organization’s network, storage infrastructure and computers that would be handling much more data than usual. The sources of this data vary greatly as well - server logs, social sites, blogs and business data (PowerPoint presentations, emails, Word documents, etc) - with hardly a standard format among them. This is data that is unstructured in nature and storing it in relational databases would be problematic at best or impossible at worst. Also remember that RDBMS are not designed to handle agile data sets. Don’t take me wrong Big Data does not come prepared. In fact the vast majority of effort dealing with Big Data is spent cleaning and transforming it from its raw format to a structured one before it can be useful and provide any insights. This puts additional stress on IT as more IT resources must be committed to the clean-up task of Big Data.

The old adage of “garbage in, garbage out” applies here. It’s not just enough to have a lot of data but the quality of data matters greatly as well. In fact a key component in BI’s effectiveness is the quality of input data. And here IT’s role is crucial in making sure that quality data is made available. In fact close cooperation between business and IT is crucial in making a success out of a BI effort.

Another challenge with Big Data is storage. As I mentioned earlier conventional storage systems can’t meet the demands of Big Data. Organizations would have to make huge investments in scalable storage systems which raise the question of how to build an IT infrastructure to handle Big Data. Since there are many solutions to Big Data challenge it is essential to engage corporate executives during the selection process; such a decision should not be made by IT alone. Big Data solutions are software, cloud or appliance based and your choice will depend on factors ranging from available resources, privacy and security of data, organizational culture, and scope of your project and cost of the solution. 

In part II I will write about IT challenges and solutions to BI and Big Data. 

As always document your work, it may save your career one day.


Ray Kahn

Director of Information Technology and Services

IEEE Computer Society


[%= name %]
[%= createDate %]
[%= comment %]
Share this:
Please login to enter a comment: