Implementing Microsoft Fabric: Lessons from 2024

Contact us for more information

2024 was a whirlwind year for Microsoft Fabric, marked by hundreds of updates that transformed its ecosystem! 

From AI integration to enhanced data management, Fabric has significantly evolved. Curious about what these changes mean for your organisation? Explore our Microsoft Fabric white paper to gain deeper insights. In the meantime, let’s take a quick look at last year’s most transformative changes: 

  • AI takes center stage: from Copilot assisting with everything from creation to explanation, to the integration of AutoML and AI skills, AI is deeply embedded in the Fabric ecosystem. 2025 will be the year for AI to prove its value with tangible results on various use cases. Out of our data team at BDO Digital has spawned an AI specialty to guide organisations through the AI roadmap from AI strategy to implementation of AI use cases.  
  • OneLake is making leaps: with its single source of truth intact, OneLake now provides more ways than ever to access and manage your data. You can connect through APIs, leverage OneLake shortcuts straight in the file explorer on your device, or query and manipulate data directly in Fabric. 
  • Real-time analytics is getting serious: the ability to process and analyse live data has moved from concept to reality. Key enablers include enhanced support for streaming data pipelines, event-driven processing with triggers, and faster query execution powered by Fabric's optimised compute engine. 
  • Fabric’s identity is shifting: is it still just an analytics tool, or has it grown into something much bigger? This shift is driven by its integration with numerous operational systems and the addition of SQL Server capabilities within Fabric, making it a versatile platform for both analytics and operations. 
At BDO Digital, we took an active role in the Fabric community, working with Microsoft to share our feedback on private preview features and sharing our ideas for interesting new features. We also migrated BDO’s internal data platform fully to Fabric. As recognition of our contributions, we were invited to speak at the European Microsoft Fabric Community Conference in Stockholm to share our experiences. We are also thrilled to share that our entire data team has achieved the Azure Data Engineer Associate certification (DP-600). But just as we were getting comfortable, Microsoft decided to raise the bar with the more advanced Microsoft Certification: Fabric Data Engineer Associate (DP-700). We are committed to staying ahead of the curve, and to help you navigate these rapid changes, we have summarised the key lessons we learned this year. 

Lesson 1: Be Ready to Adapt

The only constant with Fabric is change. Whether it’s new tools, features, or best practices, staying flexible is essential. Build your solutions with adaptability in mind, ensuring components can be swapped out as needed. Keeping an eye on resources like the Fabric Update Blog is crucial for staying informed. 

Fabric works with a private preview, public preview, and general availability for changes. This means that you have some preparation time. You could, for example, use your development and test environments to experiment with public preview features. Additionally, with access to private preview features, we can evaluate and provide informed guidance to organisations on the readiness of new functionalities for production environments.

Lesson 2: Strategic Decisions  

Before diving into implementation, invest time in strategic decisions around storage, transformation tools, and architecture.

Storage

When it comes to storage, Fabric offers three key options: 

  • Lakehouse: best for unstructured or semi-structured data (e.g., JSON, CSV, Parquet). Ideal for data engineers and machine learning scenarios requiring large-scale data processing. 
  • Warehouse: geared towards structured data with robust querying capabilities. Perfect for analytics teams working with large datasets that require SQL-based analysis. 
  • KQL Warehouse: optimised for log and telemetry data. Ideal for operational monitoring and scenarios requiring high-speed queries on time-series data. 

Choose based on your workload and team expertise:
Lakehouse for flexibility, Warehouse for analytics, and KQL Warehouse for real-time operational insights.

Transformation tools

Fabric offers several transformation tools to suit different needs: 

  • Pipeline: ideal for orchestrating complex workflows across Fabric’s ecosystem. 
  • Dataflow: perfect for low-code transformation scenarios, enabling rapid development of reusable data preparation tasks. 
  • Notebook: the best choice for data scientists and engineers who need custom scripting capabilities for advanced transformations. 
  • Eventstream: tailored for real-time data processing and analytics, providing support for streaming workloads. 

For optimal results, it’s important to match the right tool to the persona of the user. 

Architecture

The medallion architecture - a framework for organising data into Bronze, Silver, and Gold layers - remains a robust foundation for most use cases. Each layer serves a specific purpose, and it aligns closely with Fabric’s storage options and transformation tools: 

  • Bronze: stores raw data. 
  • Silver: refines data into clean, queryable formats. 
  • Gold: represents business-ready data for analytics and reporting. 

This layered approach ensures data quality, scalability, and efficient processing. 

Lesson 3: Leverage Git Integration and CI/CD Pipelines

Fabric now supports integration with version control systems like Git and has native CI/CD pipelines, making it easier to manage deployments. Git integration is particularly useful for aligning with existing version control systems, ensuring smooth collaboration across teams. Moreover, nearly all Fabric artifacts are accessible via the API, opening up an array of automation possibilities. These tools help streamline collaboration, reduce errors, and make deployments faster and more reliable. If you’re not already using them, now is the time to start. 

Lesson 4: Understand and Optimise Capacity

Fabric’s capacity management is structured around a tiered system designed to handle varying workload levels. Each tier offers a predefined amount of resources - measured in capacity units - to support specific use cases. These units govern how much data can be processed, stored, or queried within a given tier. Scalability is achieved through these tiers by selecting the most appropriate level for your current needs and scaling up as demand grows. This approach ensures cost-efficiency by tailoring resources to workload requirements. 

Fabric's capacity use is self-managing, thanks to features like bursting, smoothing and throttling. Bursting enables jobs to run at peak performance by allowing temporary usage beyond the allocated capacity. Smoothing, on the other hand, reduces the impact of spikes in compute demand by borrowing from future capacity. Throttling serves as a protective measure, activating only when sustained overuse occurs, ensuring that monthly capacity limits are not exceeded. This dynamic system ensures consistent performance while preventing resource over-allocation. 

The Metrics App provides real-time and historical insights into capacity usage, helping administrators monitor and optimise resource allocation. Additionally, the upcoming Capacity Calculator assists in forecasting and right-sizing your capacity needs based on anticipated workloads. 

Tip: for those who want to take capacity management in their own hands, pausing your capacity can be a strategic way to "force smoothing". This technique allows you to balance future compute demand and reduce the likelihood of throttling, maintaining smoother overall performance. Furthermore, reserving capacity can lead to significant cost savings—up to roughly 40%. Combining reserved capacity with a smart, automated pay-as-you-go (paygo) policy enables you to pause and restart instances dynamically, optimising both costs and performance. For those who want to dive even deeper, Fabric offers controls to fine-tune your Spark pool settings. Making smart adjustments here can go a long way in preventing over-allocation and throttling altogether.  

Capacity management is critical because the overall user experience and advanced features like Copilot depend heavily on optimised capacity. By effectively leveraging tools such as mirroring and shortcuts, you can maximise resource utilisation, minimise bottlenecks, and maintain seamless operations within your Fabric environment. These capabilities collectively ensure that your system not only meets current demands but also scales efficiently for future needs.

Lesson 5: A Platform for Everyone

Microsoft Fabric caters to three key personas: data analysts, administrators, and data engineers/scientists. Fabric’s development roadmap focuses on delivering features that genuinely improve the experience for each persona. A few examples of 2024's updates include: 

For data engineers/scientists: 
  • Enhanced Git integration, enabling better collaboration with existing version control systems. 
  • APIs that make nearly the entire platform programmatically accessible, unlocking more automation possibilities. 
  • Improvements to develop artifacts as code such as building models using TMDL code, simplifying the development process. 
  • AutoML and AI skills to get started with AI right away.  
For data analysts:
  • A more intuitive interface so if you are somewhat familiar with Power BI you can find your way easily around Fabric.  
  • Data explorer for instant insights into your datasets.  
  • Folders and visual enhancements in workspaces, streamlining content organisation. 
For administrators:
  • Advanced governance controls, including private links and endpoints, providing better data security and management. 
  • Metrics App for real-time monitoring and management of capacity. 

This is just a small sample of new features. However, a list of features would be incomplete without highlighting a fan favorite for each persona... Power BI now supports dark mode!

Lesson 6: Experiment and Play

Fabric’s low barrier to entry makes it perfect for experimentation. With so many features and options, there’s always something new to learn. Encourage your team to explore, test, and innovate. 

Lesson 7: Work with Experts 

The sheer breadth of Fabric’s capabilities can be overwhelming. Partnering with experts ensures you are leveraging your data platform effectively. From optimising costs and building production-ready environments to implementing AI use cases on your data, having the right guidance can make all the difference. 

Conclusion

2024 showed us that Microsoft Fabric is a platform in motion, evolving rapidly to meet the demands of modern data environments. By staying informed, adaptable, and open to innovation, you can unlock its full potential. As your trusted partner, BDO Digital is here to help you navigate the journey. Whether you are just getting started, looking to optimise your existing setup or diving into AI use cases; our team has the expertise to make it work for you. 

Contact us today to explore how we can transform your data strategy.