Parallelize Copy Actions in Azure Information Manufacturing unit | by René Bremer | Oct, 2024

Azure Information Manufacturing unit (ADF) is a well-liked software for shifting information at scale, notably in Enterprise Information Lakes. It’s generally used to ingest and remodel information, typically beginning by copying information from on-premises to Azure Storage. From there, information is moved by way of totally different zones following a medallion structure. ADF can be important for creating and restoring backups in case of disasters like information corruption, malware, or account deletion.

This means that ADF is used to maneuver giant quantities of information, TBs and typically even PBs. It’s thus necessary to optimize copy efficiency and so to restrict throughput time. A standard manner to enhance ADF efficiency is to parallelize copy actions. Nevertheless, the parallelization shall occur the place a lot of the information is and this may be difficult when the info lake is skewed.

On this weblog put up, totally different ADF parallelization methods are mentioned for information lakes and a venture is deployed. The ADF resolution venture may be discovered on this hyperlink: https://github.com/rebremer/data-factory-copy-skewed-data-lake.

Information Lakes are available all sizes and manners. It is very important perceive the info distribution inside an information lake to enhance copy efficiency. Take into account the next state of affairs:

• An Azure Storage account has N containers.
• Every container incorporates M folders and m ranges of sub folders.
• Information is evenly distributed in folders N/M/..

See additionally picture beneath:

On this state of affairs, copy actions may be parallelized on every container N. For bigger information volumes, efficiency may be additional enhanced by parallelizing on folders M inside container N. Subsequently, per copy exercise it may be configured how a lot Information Integration Models (DIU) and copy parallelization inside a replica exercise is used.

Now contemplate the next excessive state of affairs that the final folder Nk and Mk has 99% of information, see picture beneath:

This means that parallelization shall be performed on the sub folders in Nk/Mk the place the info is. Extra superior logic is then wanted to pinpoint the precise information places. An Azure Operate, built-in inside ADF, can be utilized to realize this. Within the subsequent chapter a venture is deployed and are the parallelization choices mentioned in additional element.

On this half, the venture is deployed and a replica take a look at is run and mentioned. All the venture may be present in venture: https://github.com/rebremer/data-factory-copy-skewed-data-lake.

3.1 Deploy venture

Run the script deploy_adf.ps1. In case ADF is efficiently deployed, there are two pipelines deployed:

Subsequently, run the script deploy_azurefunction.ps1. In case the Azure Operate is efficiently deployed, the next code is deployed.

To lastly run the venture, make it possible for the system assigned managed identification of the Azure Operate and Information Manufacturing unit can entry the storage account the place the info is copied from and to.

3.2 Parallelization utilized in venture

After the venture is deployed, it may be seen that the next tooling is deployed to enhance the efficiency utilizing parallelization.

• Root pipeline: Root pipeline that lists containers N on storage account and triggers youngster pipeline for every container.
• Baby pipeline: Baby pipeline that lists folders M in a container and triggers recursive copy exercise for every folder.
• Change: Baby pipeline makes use of a swap to resolve how listing folders shall be decided. For case “default” (even), Get Metadata is used, for case “uneven” an Azure Operate is used.
• Get Metadata: Record all root folders M in a given container N.
• Azure Operate: Record all folders and sub folders that include not more than X GB of information and shall be copied as a complete.
• Copy exercise: Recursively copy for all information from a given folder.
• DIU: Variety of Information Integration Models per copy exercise.
• Copy parallelization: Inside a replica exercise, variety of parallel copy threads that may be began. Every thread can copy a file, most of fifty threads.

Within the uniformly distributed information lake, information is evenly distributed over N containers and M folders. On this state of affairs, copy actions can simply be parallelized on every folder M. This may be performed utilizing a Get Meta Information to listing folders M, For Every to iterate over folders and duplicate exercise per folder. See additionally picture beneath.

Utilizing this technique, this is able to indicate that every copy exercise goes to repeat an equal quantity of information. A complete of N*M copy actions might be run.

Within the skewed distributed information lake, information will not be evenly distributed over N containers and M folders. On this state of affairs, copy actions shall be dynamically decided. This may be performed utilizing an Azure Operate to listing the info heavy folders, then a For Every to iterate over folders and duplicate exercise per folder. See additionally picture beneath.

Utilizing this technique, copy actions are dynamically scaled in information lake the place information may be discovered and parallelization is thus wanted most. Though this resolution is extra complicated than the earlier resolution because it requires an Azure Operate, it permits for copying skewed distributed information.

3.3: Parallelization efficiency take a look at

To check the efficiency of various parallelization choices, a easy take a look at is ready up as follows:

• Two storage accounts and 1 ADF occasion utilizing an Azure IR in area westeurope. Information is copied from supply to focus on storage account.
• Supply storage account incorporates three containers with 0.72 TB of information every unfold over a number of folders and sub folders.
• Information is evenly distributed over containers, no skewed information.

Take a look at A: Copy 1 container with 1 copy exercise utilizing 32 DIU and 16 threads in copy exercise (each set to auto) => 0.72 TB of information is copied, 12m27s copy time, common throughput is 0.99 GB/s

Take a look at B: Copy 1 container with 1 copy exercise utilizing 128 DIU and 32 threads in copy exercise => 0.72 TB of information is copied, 06m19s copy time, common throughput is 1.95 GB/s.

Take a look at C: Copy 1 container with 1 copy exercise utilizing 200 DIU and 50 threads (max) => take a look at aborted because of throttling, no efficiency acquire in comparison with take a look at B.

Take a look at D: Copy 2 containers with 2 copy actions in parallel utilizing 128 DIU and 32 threads for every copy exercise => 1.44 TB of information is copied, 07m00s copy time, common throughput is 3.53 GB/s.

Take a look at E: Copy 3 containers with 3 copy actions in parallel utilizing 128 DIU and 32 threads for every copy exercise => 2.17 TB of information is copied, 08m07s copy time, common throughput is 4.56 GB/s. See additionally screenshot beneath.

On this, it shall be seen that ADF doesn’t instantly begin copying since there’s a startup time. For an Azure IR that is ~10 seconds. This startup time is fastened and its impression on throughput may be uncared for for giant copies. Additionally, most ingress of a storage account is 60 Gbps (=7.5 GB/s). There can’t be scaled above this quantity, until further capability is requested on the storage account.

The next takeaways may be drawn from the take a look at:

• Vital efficiency can already be gained by growing DIU and parallel settings inside copy exercise.
• By working copy pipelines in parallel, efficiency may be additional elevated.
• On this take a look at, information was uniformly distributed throughout two containers. If the info had been skewed, with all information from container 1 situated in a sub folder of container 2, each copy actions would want to focus on container 2. This ensures comparable efficiency to Take a look at D.
• If the info location is unknown beforehand or deeply nested, an Azure Operate could be wanted to establish the info pockets to verify the copy actions run in the best place.

Azure Information Manufacturing unit (ADF) is a well-liked software to maneuver information at scale. It’s extensively used for ingesting, remodeling, backing up, and restoring information in Enterprise Information Lakes. Given its function in shifting giant volumes of information, optimizing copy efficiency is essential to attenuate throughput time.

On this weblog put up, we mentioned the next parallelization methods to reinforce the efficiency of information copying to and from Azure Storage.

• Inside a replica exercise, make the most of commonplace Information Integration Models (DIU) and parallelization threads inside a replica exercise.
• Run copy actions in parallel. If information is understood to be evenly distributed, commonplace performance in ADF can be utilized to parallelize copy actions throughout every container (N) and root folder (M).
• Run copy actions the place the info is. In case this isn’t recognized on beforehand or deeply nested, an Azure Operate may be leveraged to find the info. Nevertheless, incorporating an Azure Operate inside an ADF pipeline provides complexity and needs to be averted when not wanted.

Sadly, there isn’t any silver bullet resolution and it all the time requires analyses and testing to seek out one of the best technique to enhance copy efficiency for Enterprise Information Lakes. This text aimed to present steering in selecting one of the best technique.