Case Study

Transforming business insights through data visualization has become essential. Business intelligence goes beyond mere data collection and turns complex data into actions, making data visualization a critical component of decision-making processes. How can Business Intelligence tools, best practices, and AI-driven visualizations streamline decision-making and foster collaboration across teams? Keep reading to find out.

Importance of BI

Business Intelligence is often misunderstood in its scope. While data visualization practices are quite popular, fundamentals are frequently neglected. With the rapid development of new technologies, digitalization has become critical for businesses to stay competitive.

The challenge for traditional data-centered companies has now become not collecting the data but collecting it in a way that can be leveraged for competitive advantage. This is where Business Intelligence stands as a unique tool in every enterprise – a plethora of practices and technologies that help build a company-wide consistent single source of truth.

Data Visualization: The Fun Side of BI

Other than providing efficient data structures and diminishing data silos, the fun part of Business Intelligence – Data visualization, can help with the aggregation and tracking of data or it can be the catalyst for cross-departmental collaboration and better decision-making. While fostering an environment where people get unified access to meaningful data, all this is done with the correct security policies in mind.

Data visualization in BI is the last stage of the reporting cycle that should not be overlooked.

Best Practices in Data visualization

Is Visual Analytics Biased?

When we talk about visualizing data, creating reports, building end-user dashboards, or encouraging self-service, the only thing that should cross our mind is: ‘visual analytics’. A large proportion of the audience would understand if they looked at any report designed by someone else. The point here is that each person uses their discretion, personal biases, and primed preferences to reflect their work in how something would ‘look good’. In simpler terms, every one of us has a different ‘taste’. That does not make it wrong or right. However, visual analytics is a field of research that combines scientific methods, based on human cognition, on how to develop end-user reports that are both appealing and intuitive.

Visualization Requires Efforts

Another key flaw we have identified from our experience with working with ETRM data is that users expect to have all the benefits of visualizations without putting in the effort to unify and structure it. That means, cleaning up or preparing the underlying data that we will be displaying in a report is a more crucial step than building the visuals, selecting the colors, formatting, etc. What visualization is simply aimed at is to give meaning to our data. Humans are naturally born with pattern recognition abilities and data visualization helps us uncover data secrets by using what is natural to humans, thus improving efficiency in the dynamic world we live in today.

Some of the key contributors to visual analytics are authors Stephen Few and Edward Tufte, who, with their research, have revolutionized the way report developers see data and present it.

“There is no such thing as information overload. There is only bad design.” – Edward Tufte

Working with softer colors, considering accessibility, understanding data granularity, and selecting proper data representation are some of the components that go into giving data the appropriate context and enabling users to combine data patterns effortlessly.

Transactional Vs. Aggregated Data

With this in mind, the practice of data amalgamation can be distributed across business cases. As we have previously witnessed, risk reporting is quite different than if we were to run a manufacturing business. Certain data structures distinguish the practices that are applied to the data – whether it is transactional or aggregated.

One may be good with a normalized data model, and another may require denormalization or even a mixture of both, depending on the end goal. With the increasing number of regulations in the energy trading sector, reporting complexity requires a thorough understanding of the value added from the reporting implementation. It is therefore crucial to also acknowledge the collaboration aspect between Business and IT in building Business Intelligence applications in a complex environment.

Tools

Everyone would want to ask: ‘What tools are you using?’

There is no such thing as the perfect tool. And with the various tools on the market, the market does not make it easier. Perhaps only one or two will be useful for each organization, depending on their setup. Picking a tool is not an easy task. To simplify it, tools are either specific and proprietary, or commercial.

Proprietary Tools

The proprietary tools are usually industry/niche-specific with a specific data model and structure in mind. There are benefits to this, as well. For instance, a pre-made data infrastructure will speed up the process of deciding on an optimized data structure. On the other hand, this is a limitation as enterprise data may come in many forms, from unsupported data sources or an organization may eventually need more flexibility.

Commercial Tools

This specific nature of proprietary tools is the reason why most organizations prefer to go with a commercial reporting tool, such as Tableau or MS Power BI, that provides sufficient flexibility to the point where external systems can aid the data optimization and deliver data in a cohesive format.

From there, it is the management’s decision and outlook expertise to decide how it will be used. Is a premium needed? Is it going to be ingesting clean data? What is the update frequency and daily load?

Hybrid

A hybrid approach for smaller enterprises with a substantial domain know-how is to use commercially available reporting software, to design and build an optimized end-to-end reporting solution, with a scope in mind to provide fast time-to-market to qualified organizations.

Such is the case with Numera – a data warehouse solution to standardize the reporting of ETRM business data, utilizing the flexibility of a selected reporting tool like MS Power BI. Although the reporting tool is just the user interface and it can be interchangeable, finalizing the reporting depends on the flexibility of the front-end software and it can sometimes mean options are limited, considering the data model’s complexity and compute demands.

AI in data visualization

Artificial intelligence has gained significant popularity in recent years, although it has been in use for over a decade. That is primarily because of the inputs commercial tools receive from their users. As with all data aggregation, data unravels patterns. AI can automate those patterns, detect specific naming conversions, and generate a pre-selected type of report.

What’s behind the buzzword AI?

AI is a strong word for how it is being used for reporting. We would rather classify it as a pattern recognition capability, given specific inputs. The AI part of reporting is indeed the recognition of data and table structures, including detecting table relationships, and data types, identifying table constraints, optimization techniques, and more.

One of the latest tools that has been the most marketed in the last year has been MS Fabric. In addition to aggregating data from multiple sources, it combines reporting with data engineering and process orchestration. Tools like Azure Synapse Analytics support these functions. They also enable data transformation using Python and other programming languages, allowing users to generate reports from ingested tables.

As someone who has worked on hundreds, if not thousands, of reports, using AI to generate reports has brought up a bit of uncertainty. Each time we automate the creation of something valuable, we become disconnected from its underlying components. Perhaps the engine implements a calculation that is not entirely aligned with the business logic. It does help with the initial preparation of all key metrics but debugging everything done by AI may take as much time as creating it ourselves, given we understand how calculations should work and their dimensionality.

On the bright side, there will often be benefits to knowing when AI works best and using it for efficiency purposes. After all, creating visuals can be a tedious task. Users’ best practices for AI usage would more likely include the understanding of where AI would do a good job, which comes from experimenting and leveraging it to their advantage to discover data patterns, rather than blindly relying on it.

General tips and guidelines

General guidelines can apply to small and large-scale projects, regardless of the industry in which the business operates.

1. Know the end result

It may be common sense, however, when developers start building the data model, the first thing they need to know is the desired result. This can be achieved not only throughout the requirement-gathering process but also through building swift PoCs (report prototypes) of what the report would look like, given a specific data model, considering the users’ report usage.

This means that for smaller-scale projects – even in energy trading, developers would build a table schema that facilitates specific report usage. For larger-scale projects, the data model may be different – so it can cover a variety of business cases. However, specific sections of it can be split into the so-called ‘data marts’ or views/subsets of data that correspond to the desired data format.

As a simple example, a data mart can correspond to a collection of records in a more granular format, represented in a single report to track data changes or validate records within a lower timeframe.

2. Keep only relevant data

The second point worth mentioning is that only relevant data should be kept. Indeed, users always need all of the data, for many reasons, and there could be a compromise in how historical data is accessed, given reporting or database limitations. Identifying a win-win scenario in this case is essential for timely and consistent reporting.

Even with all the technology surrounding us, on-prem or managed services still have compute limitations that may cause timeouts, server unavailability, and queued queries. Aligned with the increasing complexity of ETRM business data, we would all be happy to display an unlimited amount of history in our reporting, without any delays. However, no tool has been developed yet that offers unlimited resources.

3. Transform early and understand visual analytics

Tying all this up into the last two points, transformations should be done as early as possible. Having built a PoC for users to test the reporting functionality, we can then comply with all the visual analytics practices and build a reader-focused data consolidation page that will enforce proper decision-making.

Author: Petar Nikolov

Editor: Hristina Tankovska