Data Engineering in Microsoft Fabric

Products

Dynamics 365 ERP

Dynamics 365 Finance
Connect your operations with your finance

Dynamics 365 Supply Chain Management
Take control of your supply chain

Dynamics 365 Project Operations
Unify all your project management

Dynamics 365 Human Resources
Optimize your HR processes

Dynamics 365 Commerce
End-to-end retail management solution

Dynamics 365 Business Central
Unified, adaptive business operations platform

Dynamics 365 CRM

Dynamics 365 Sales
Turn your opportunities into sales

Dynamics 365 Customer Service
Build customer loyalty

Dynamics 365 Customer Insights
Connect with your audience

Dynamics 365 Customer Voice
Know your customers

Dynamics 365 Field Service
Optimize your field service

Power Platform

Power Pages
Connect with your customers and suppliers

Power BI
Base your decisions on data

Power Automate
Automate your processes

Power Apps
Turn your ideas into applications

Power Virtual Agents
AI-driven, conversational support bots

Modern Work

Microsoft 365
Your office in the cloud

Microsoft Fabric

Data Factory
Automated, flexible, real-time data orchestration

Data Engineering
Efficient and scalable data processing

Data Science
Unified, automated, end-to-end modeling

Data Warehousing
Scalable, integrated, real-time warehousing

Real-Time Intelligence
Intelligent data insights in real time

OneLake
Unified data lake platform

Data Activator
Real-time automated data triggers
Blog
Contact us

Data Engineering in Microsoft Fabric

Optimizing Data Workloads & Scalable Pipelines

Microsoft Fabric’s Data Engineering platform enables organizations to efficiently design, build, and maintain data infrastructures for collecting, storing, processing, and analyzing large datasets. With tools like lakehouses, data pipelines, Spark job definitions, and notebooks, users can streamline data workflows, ensuring accessible, organized, and high-quality data throughout its lifecycle.

Unified storage solution

Combines data lake and data warehouse capabilities to manage both structured and unstructured data in a single location.

Flexible analytics

Supports SQL-based queries, machine learning, and advanced analytics tools.

Seamless data integration

Allows integration of various data types for comprehensive analysis and reporting.

Notebook creation

Similar to other standard item creation processes in Microsoft Fabric, you can easily create a new notebook by navigating to the Fabric Data Engineering homepage. These options provide a simple and efficient way to get started with creating notebooks in your environment.

Connect lakehouses and notebooks

Fabric notebooks now offer seamless integration with lakehouses, allowing you to easily add new or existing lakehouses directly from the Lakehouse Explorer.

Support multi-languages

The file editor in Fabric notebooks allows you to view and edit files directly within both the notebook’s resource folder and the environment’s resource folder. Supported file types include CSV, TXT, HTML, YML, PY, SQL, and more. With the file editor, you can easily access and modify these files, benefiting from features like keyword highlighting and language support.

Overview

Apache Spark, an open-source distributed computing system, plays a key role in processing large datasets for data engineering and science tasks. Within the Microsoft Fabric platform, the integration of Apache Spark provides robust capabilities for scaling and managing data tasks. There are different versions of Apache Spark available within Microsoft Fabric, including Runtime 1.1, which uses Apache Spark 3.3, Java 1.8, Python 3.10, Scala 2.12.15, Delta Lake 2.2, and R 4.2.2. However, Runtime 1.1 will reach its end of support in mid-2024, so users are advised to upgrade to newer versions like Runtime 1.3 for the most advanced features and support.

Multiple runtimes support

Microsoft Fabric provides users with the flexibility to switch between multiple runtimes, making it easier to manage workloads and minimize disruptions due to potential incompatibilities. This versatility ensures that users can choose the runtime best suited for their specific use cases while maintaining a smooth and efficient workflow.