Example Data Architecture Framework
Below is a suggested framework that will provide a basic footprint from which to build your Qlik data model from your data fabric, to serve both the needs of your organisation, Archive to Report, but also address many data access risk and compliance issues. Including for ESG and GDPR.
Qlik’s n tier architecture in itself does not provide any form of data compliance. Qlik provides the tools and the Qlik Management Console (QMC) is advanced in being able to apply many and various data access and security models, however you first need to understand the governed data access framework that you wish to adopt. Coupled with your application life-cycle management strategy. Differentia Consulting created this document (revised 2022) as a guide from which you can build your data model incorporating other technologies as appropriate. What is important is to recognise the key significance of each stage. You may need more stages and possibly fewer however you should build your data model to plan to incorporate into your design the need to meet the intended use case of each, as identified by various BI strategists since BI and data modelling began.
We suggest that the chief data officer (CDO) of your organisation gains a full understanding of this model and applies the framework to the data first. It is not uncommon for a large organisation with multi-site deployment to join models together and have various levels within each stage. Complexity and need will determine requirement. We recommend clients keep work instructions for each data source as it is added to the data model as various stakeholders will need to approve the use of data at each stage.
Whilst the framework may appear complex to some, it is actually simplified by purpose, for those adopting Differentia Consulting’s #SmarterBI methodology it has been how the design of many QlikView documents have been built over many years. What is different with Qlik Sense, especially SaaS, is the ability to separate data provision by IT from consumption by the business (Business Analysts). In a way to deliver true segregation of duty. Ability to more easily demonstrate compliance and most certainly enjoy Qlik more by everyone.
Qlik Governed Data Access – Example Framework
Qlik Governed Data Access, example framework…
To simplify where Qlik can add value to your enterprise a series of distinct stages, listed in reverse, have been defined to create a framework for data access:
S6 Public
Data is desensitised, summarised and aggregated making it low risk and suitable for public consumption to meet the need of external stakeholders.
S5 Consolidated
Serving C-level executives the purpose of consolidation is to provide Global KPIs. Given the type of device or document that this is usually consumed on there is not much need for full drill down. Although to add more detail is not an issue.
S4 Self Service (Derived/Trusted/Production)
Semantic “UI” II – Medium Risk: Platform for business discovery in detailed but desensitised pseudonymised data adhering to business rules and logic of the organisation. The UI data stage that Qlik Sense self service is designed for.
S3 Guided Analytics (Conform)
Semantic “UI” I – High Risk: Addition of additional dimension tables and application of business rules and logic. Corporate guided analytics, Operational Dashboards can be built with this data where permissions allow it.
The application of group rules and section access ensures that only the groups that need to see sensitive field names can see them..
If personal and sensitive data exists but not needed for their role, then S4 data may need to be adopted for compliance purposes.
S2 Transform (Standardised)
Given the aim is to create a data model with Dimensions that are consistent, initial transformation is simply for the purpose of ensuring that all column names both comply and data at the stage can be used for validation and Quality purposes. Many clients run integrity/quality checks with Qlik and send alerts with NPrinting. This can be an automated process. Access should be strictly controlled at this stage and processes automated.
S1 Extract (Landed/Raw)
Creation of a time stamped record of a file/table at a point in time, “Raw”. (Daily weekly monthly etc.)
Note: The endpoint of this process can be either can be contained in Qlik or a data store, such as Snowflake eg for Archiving purposes,
[Archiving with Qlik is becoming more popular now with Change Data Capture (CDC) capability, and this is where we can begin the process. Qlik is very capable as an archiving/retirement tool both for ongoing archiving, but also for sun-setting/retiring end of life applications as part of enterprise transformation. Retiring systems and archiving data with Qlik for the purposes of consolidation of ERP or CRM systems for example. Differentia Consulting has particular expertise and delivered a lot of value in this area.]
Each of the stages above may be individually expanded to include multiple levels and parallel test/sanbox stages. For an international organisation deploying Qlik locally, and consolidating financials by region with Qlik, then globally there can easily be 6 to 20 levels, the stages will remain the same.
#SmarterBI with Qlik
Qlik platform-enabled visual analytics is delivered by products that support some of the world’s largest and most innovative companies and organisations. In addition to our Smarter.BI methodology Differentia Consulting has built many Smarter.BI applications with Qlik, forming a client specific Smarter.BI platform which delivers secure and GDPR compliant operational intelligence solutions. TCO, ROI and simplicity are the keys to driving adoption of a Smarter.BI platform powered by Qlik’s platform-enabled visual analytics offerings.
