May 10, 2021

How to build your data & analytics platform in Azure

 Weelin Lim
Weelin Lim Chief Product Officer

Business challenge:

Data and Analytics Roadmap
Microsoft Azure Cloud

Business solutions powered by the cloud have redefined the way we run businesses around the world.

Microsoft Azure has been the innovative model that has changed the face of cloud computing services, as a platform that aligned perfectly with a rapidly-changing market in its formation, and continues to adapt and thrive.

Offering secure and scalable applications for all business data requirements, Azure is used globally, notably by over 95% of Fortune 500 companies.1 It supports a wide range of programming languages, frameworks, databases, operational systems and devices, allowing businesses to integrate with or migrate from existing technologies.

Azure is a broad landscape of data services with hybrid capabilities, fueling innovative and adaptable business strategy. The diverse range of services allows it to offer impactful tools to a range of businesses, regardless of size. Azure features over 200 cloud programmes and services, offering flexibility and specificity, and saving businesses time as data management can be modernised through proven architecture patterns. Maximising benefits relies on developing a consistent strategy with a well-developed roadmap to make more effective use of valuable data. 

Start at the beginning

The first step in establishing successful service patterns using Azure is to understand the scope and capability of the platform, but doing this in line with the requirements of your organisation. With an ever-expanding list of software solutions, from simple web services for hosting a business portal in the cloud, to running on-demand cloud-based chat-bots, the capabilities of Azure are extremely broad. Making use of cloud-based services like remote storage, centralised management, AI and IoT, its services offer a secure platform for building, managing and scaling the analytics capability to suit your business needs.

The majority of organisations will never make use of all the services the platform offers. However, the businesses that benefit most are those that have a clear understanding of their data and analytics needs, allowing them to concentrate on the areas that will bring the most value. Being clear on how your data will be used to further your business objectives is key in delivering value from data and analytics, as this will impact your scale and timeframe. Getting to this clear understanding can be time-consuming, as it will involve managing stakeholders and obtaining clear requirements of how data needs to be used, but is hugely important as it ensures end-user buy-in and increases the likelihood of a successful delivery. Facilitating clear communication on the scope and objectives of your data and analytics will be invaluable as you progress, and your architecture evolves throughout development.

Building blocks to success

Azure provides flexibility to instantiate services as and when they are needed, so you can change your platform capabilities as your progress. You can set up services incrementally, but you should have a roadmap and strategy so that you know which area to start with to get the most business value.

Data can be incredibly varied and there are different patterns for delivering specific analytics requirements, depending on whether you are doing data warehousing for BI/MI, streaming real-time analytics, IoT analytics, AI/MI etc.

Knowing that the Azure platform will support these other data types and patterns gives you the confidence that your platform can be extended to support these when the time is right. This is why having a strategy and roadmap in mind is so important up front.

Developing a roadmap

The Azure data landscape is a busy scenario that offers different services and capabilities all focused on helping you get the most value out of your data. They can be roughly grouped into five sections: data ingestion, data storage, analytics, visualisation and intelligence. Although there are many more Azure services on offer, the ones shown below are the ones that are likely to be most useful for the majority of organisations building an initial data and analytics platform in Azure.

Developing a roadmap involves looking through the services offered in each of the above sections and honing in on which will be most suitable for your data and desired outcomes.

Initially getting an understanding of your current overall data variety will help you shape the architecture of your data platform needed to support analytics. 

Getting an architecture design in place is important to ensure your platform will support all your needs. Coming up with a prioritised list of analytics that bring the most value early will help you determine which data to tackle first and will then inform you of the components that will be initially needed.

The first stage, data ingestion, is all about getting your data into Azure. Depending on your data sources, there are a number of typical ways of ingesting that data. Data Factory is used to schedule data-driven workflows (pipelines), ingesting data from disparate data stores. Event Hub and IoT Hub brings in data from sensors and processes. The type of data you have, and how it is generated will determine which service you choose.

Once ingested, your data needs to be stored in a way that’s most useful for your needs; here you have services such as Data Lake, Blob Storage, SQL Database and Cosmos DB. Depending on the data sources, you may choose various storage options to suit them. If you have a data lake approach, you may choose to land all data in there initially before taking some of that data into other storage options for further processing. If you only have structured data that will only be stored in a data warehouse, you may decide to load your data directly into SQL DB.

In the end, you may end up using a variety of storage options to deliver on your architecture.

The analytics engine, as the next stage in the workflow, provides compute power to perform calculations, drive ML models, transform and prepare the data for analysis.

Your requirement may be for simple data transformation compute power that could be driven simply using Azure SQL DB, or larger-scale MPP power using Synapse or Data Bricks.  Stream Analytics provides the ability to generate analytic information from streaming data sources. Again, depending on your requirement, this will shape the services you will use.

Azure provides a set of advanced and specialist intelligence services to deliver capabilities such as natural language processing, image recognition, Chat bot builders and more. Not all of these will be relevant to all organisations, especially in the early stages of their data and analytics journey, but these may become more relevant as they mature. Having this capability on-hand to be integrated easily is a huge benefit for organisations with those requirements on their roadmap.

Data visualisation is one of the most fundamental ways of providing analytics to your organisation and Power BI is the leading BI/analytics tool in this area. It provides a rich and highly interactive capability for delivering impactful data analytics to a wide audience.

Conclusion

In its simplest form, creating a successful data and analytics platform in Azure could be regarded as establishing and maintaining the following framework: data, intelligence, action. The cohesive product suite can adapt to requirements of any complexity, and you can construct your specific architecture which will serve a range of purposes, building out from the above simplified stricture.

As you build out your architecture through each of the stages of the funnel: data ingestion, data storage, analytics, visualisation and intelligence, it’s important to consider your requirements of this data, as this will influence decisions. Considering your ‘why’ will lie at the heart of your roadmap planning.

The final stage in building your data and analytics in Azure is to experience how these stages work in practice, experimenting and learning through the use of the services. Azure’s excellent tools and technology will provide a reliable framework for your landscape but using these tools in the right way, and making them work for your data requirements are where the real value in the platform lies. With Azure, you can be confident that the framework you are building your architecture around is flexible, secure, scalable and ever-improving, enabling you to bring your data and analytics vision to life now and in the future.

This article is part of our Data-Driven SMB series. For more information, advice and resources on how to accelerate your organisation’s data and analytics maturity, click here or contact us today.

Reference

1 The only consistent, comprehensive hybrid cloud. Microsoft Azure, 2018

Business challenge:

Data and Analytics Roadmap
Microsoft Azure Cloud
 Weelin Lim
Written by Weelin Lim Chief Product Officer

Weelin is the creator of the TrueCue Platform, and Chief Product Officer, having worked in the data and analytics industry for the last 20 years. In that time, he has delivered numerous large and small-scale BI and data and analytics platforms across a variety of industries, including healthcare, finance, media and energy.

eBook: Cloud Data Warehousing Playbook for SMBs

If you are an small or medium business looking for a way to give your business the edge, then this playbook is for you. It introduces the concept of modern data warehousing and explains how it serves as the foundation for good analytics. It covers options. It gives tips. It warns of the pitfalls.

Download eBook