LS Insight Azure Data Factory

The LS Insight setup process creates the LS Insight - Azure Data Factory (ADF). The ADF contains everything needed to populate the star schema dimensions and facts in the LS Insight - Data Warehouse. This topic describes all the different factory resources and how they are used.

Overview

Data Factory

Author

Monitor

Manage

Note: This is a detailed description of the ADF components to give you a better understanding of what happens where. Everything about how to use the ADF when setting up LS Insight is explained in the onboarding process.

Overview

The components are:

  • Integration runtime gateway
  • Linked services
  • Pipelines

To view these components in the Azure portal you need to open the LS Insight ADF from the resource list and then select Author & Monitor from the overview page.

When you open the Author & Monitor view, you will see a new tab in your browser and get access to the data factory menu that will be referenced many times in this ADF description. The menu consists of icons and labels that lead to the different sections.

Data Factory

This is just a generic section where you can access videos and knowledge about how to get started with ADF. Since the LS Insight ADF is created by the deployement script, you can just use this information if you want to explore the options of ADFs further.

Author

The Author section is the heart of the ADF. In this section you have access to the pipelines that are an essential part of the LS Insight ETL.

Pipelines

The LS Insight pipelines are arranged in a folder structure by where in the process they are used. Some pipelines query the LS Central source database, while others move data between tables in the Data Warehouse by running stored procedures from the DW database. Other pipelines are created to control the order the pipelines are run in. Their only purpose is to execute other pipelines.

Datasets

There are three data sets in the LS Insight Datasets folder and they are used in different lookup activities in the pipelines.

LSCentralColumnMetadata
LSInsightDW
SourceDataset

Monitor

A special page is dedicated to Monitoring pipelines while they run. This goes both for scheduled pipelines and monitoring of manually triggered pipelines and pipeline chains.

Manage

The Manage section of the Azure Data Factory contains the Integration runtimes gateway and the Linked services. Both play a part in the connection from Azure to the LS Central source database and to the LS Insight DW.

Integration runtime gateway

The manual setup of the LSInsight-IntegrationRuntime gateway is explained in the LS Insight onboarding process. It is only needed when LS Central is on-premises, otherwise the AutoResolvedIntegrationRuntime is used.

Tip: Microsoft has some extensive documentation about the integration runtime.

Linked services

The LS Insight onboarding process creates the ADF with two linked services.

LS Central source

This linked service stores the connection to the LS Central source database and utilizes the LSInsight-IntegrationRuntime for that connection.

LSInsightDW

This linked service stores the connection to the LS Insight DW database.

If the LS Insight database is located in Azure, it uses the AutoResolvedIntegrationRuntime to connect to the Azure SQL database, but if LS Insight is located on-premise within the same domain as the LS Central database, it can utilize the LSInsight-IntegrationRuntime that was created during the LS Insight setup.