Analytics can help an organization in many ways. Using data effectively can help an organization maximize its potential and avoid problems. All these benefits rely on an effective analytics solution, which allows the organization to use its data effectively.
There are a myriad of ways to create an analytics solution. An organization could apply many processes and technologies based on their needs. However, most solutions have some similarities and follow the same basic pattern.ย
The Analytics Process
The image above shows a high-level view of the analytics process and the types of people involved. It is colour-coded to show which persona is responsible for which parts. Starting at the top and working downward:
1. Data Sources: Data comes from somewhere. These could be financial/accounting systems, APIs (such as those provided by social media platforms or weather services), sensors and meters on equipment, files, manual entry, and more. The first step in the process is to collect this data and bring it into the Analytics Environment.ย
2. Analytics Backend: Usually, only the developers see the backend of an analytics system, but it does a lot. The backend stores and transforms the data, preparing it for viewing in a process known as ETL or ELT. Raw data from the source systems is often difficult to understand and may contain errors, but these are fixed here. Data flows through several stages or layers, sometimes denoted by medals (Medallion architecture), as the data progresses from its raw form to report-ready form.ย
3. Machine Learning: Only some organizations will have this, but part of the backend processes can be machine learning, where predictions and advanced analysis are applied to the data.
4. Reports and Dashboards: End users consume the data through reports and dashboards once the data is ready.ย
Most of this process will exist multiple times for every solution. These are different environments; usually, one is called production and is the process that feeds the reports business users see. A development environment supports production where fixes, updates, and new parts are created. Sometimes, a test environment exists, allowing changes from development to run and test before becoming available in production.
A solution can centralize the data so one database is holding it or break it down by various parts of an organization.
Development Team
Behind every analytics solution, there’s a dedicated development team that creates, maintains, updates, and upgrades it. A small organization may only have one person wearing all the hats, whereas a larger one will have teams for each role. Here are a few of the more common types of developers.
Data Engineer: These people manage the backend of the analytics system. Responsible for data extraction and processing using a variety of tools.ย
Data Analyst: The report creators. They build the reports and dashboards which the end users see.
Data Scientist: Responsible for running machine learning and advanced analytics on the data.
Business Analyst: It is a less technical role, but they are business-process experts who work with developers to translate business requirements into technical solutions.
Governance
With the complexity of an analytics solution, a sense of order is essential. Governance, a set of standards and processes, is the key to ensuring the smooth operation of a solution. It encompasses various elements, each playing a vital role in maintaining order.
Access Controls: Who gets access to the data? How do they get access? For instance, a data engineer will likely have access to almost everything in a development environment but have limited access in a test or production environment. A data analyst may only have read access to most backend items but full access to the reports and dashboards.
Data Quality: A set of standards the data must meet, including conventions to use to various other quality standards.ย
Compliance: Abiding by applicable laws and standards.ย
Master Data Management: Ensuring that the same terms and values are used across an organization so that different datasets don’t use different terms for the same thing.ย
Oversight and Auditing: A team that is responsible for making sure Governance is being done and done correctlyย
Conclusion
Whatever the specifics of an analytics solution, they all have a typical pattern. Data flows from a source, is transformed and prepared, and then presented in reports and dashboards. This process requires several different people to build and maintain. When it is all done, the results can be revolutionary for an organization.