Deploy Azure Batch Apps Service Sample through Azure Batch Apps Portal

Microsoft Azure Batch Apps is an Azure service which provides the ability to run compute-intensive and massively parallel workload on demand. These workloads could be managed either via Azure Batch Apps APIs or the Batch Apps portal. This article uses a sample from GitHub (https://github.com/spsarkar/AzureBatchDependenciesStorage ) to demonstrate the usage of Azure Batch Apps through Azure Batch Apps management portal.

Preparing Azure Batch Service

You need to have Visual studio 2013 or above to build this project. This project uses nugget packages Microsoft Azure Batch Apps Cloud SDK (PM> Install-Package Microsoft.Azure.Batch.Apps.Cloud –Pre) and Windows Azure Storage (PM> Install-Package WindowsAzure.Storage –Pre). You need to create Azure batch Service from Azure Management portal if you haven’t done it yet.

Workflow to publish and run Azure Batch applications

This article (http://azure.microsoft.com/en-us/documentation/articles/batch-technical-overview/ ) describes the basic concepts required for Azure Batch Service and ‘Workflow to publish and run an application with Batch Apps’ section of this article demonstrates workflow to publish and run Azure Batch Apps. Briefly Azure Batch apps consist of two major component (Application image and cloud assembly). Diagram source: http://azure.microsoft.com/en-us/documentation/articles/batch-technical-overview/

Application Image

An application image is a zip file containing application executables and related necessary support files. A dummy test application image (mri-processing-dummy.zip) is included in the GitHub project. This zip contains two Windows Batch File ( niftiInit.bat and skullStrip.bat ) which copies text content of input file to output files.

Cloud Assembly

A Cloud Assembly is a zip file containing cloud assembly that will invoke and dispatch workload to Azure Batch Service. It contains a Job Splitter and a Task processor. Cloud assembly is not included in the GitHub project, you make it by yourself by compiling the project itself and zipping the content of the output folder.

Entry Point to Azure Batch Apps

‘ApplicationDefinition’ represents the main entry point in the cloud assembly. This definition includes the following:

• Name of the Job

• Name of the Application

• Job Splitter – This step gives us the possibility to split a job in multiple tasks

• Task Processor – which invokes the application executable for a given task.

  public class ApplicationDefinition { public static readonly CloudApplication Application = new ParallelCloudApplication { ApplicationName = “AzureBatchNiftiProcessing”, JobType = “AzureBatchNiftiProcessing”, JobSplitterType = typeof(AzureBatchNiftiProcessingJobSplitter), TaskProcessorType = typeof(AzureBatchNiftiProcessingTaskProcessor) }; }

Job Splitter

Job Splitter does the splitting of your job into multiple tasks. This enables you to run multiple parallel and dependent tasks. For More detailed information, please read this article (http://azure.microsoft.com/en-us/documentation/articles/batch-dotnet-get-started/ ).

         protected override IEnumerable Split(IJob job, JobSplitSettings settings)
        {
            var reorientTask = new TaskSpecifier
            {
                TaskId = TaskIds.Reslice,
                RequiredFiles = job.Files.Take(1).ToList(),
                Parameters = job.Parameters,                
            };
            var reorientTask2 = new TaskSpecifier
            {
                TaskId = TaskIds.Reslice1,
                RequiredFiles = job.Files.Take(2).ToList(),
                Parameters = job.Parameters,
            };
            var skullStripTask = new TaskSpecifier
            {
                TaskId = TaskIds.SkullStrip,
                Parameters = job.Parameters,
                DependsOn = TaskDependency.OnId(TaskIds.Reslice)
            }.RequiringAllJobFiles(job);
            var skullStripTask2 = new TaskSpecifier
            {
                TaskId = TaskIds.SkullStrip1,
                Parameters = job.Parameters,
                DependsOn = TaskDependency.OnId(TaskIds.Reslice1)
            }.RequiringAllJobFiles(job);
            return new List { reorientTask, reorientTask2, skullStripTask, skullStripTask2 };
        }

Defining & Running a Tasks (Implementing ParallelTaskProcessor.RunExternalTaskProcess)

At this stage you define each of the non merge task specified and returned from the job splitter. It invokes application with the appropriate arguments, and return a collection of outputs that need to be kept for later use. The following could be specified for ExternalProcess :

• CommandPath – The path of the executable file

• Arguments – The command line argument for the executable

• WorkingDirectory – The working directory for the external program process

• CancellationToken – Gets or sets a cancellation token which can be used to cancel the external.

            var originalInputFileName = task.RequiredFiles[0].Name;
            var inputFile = LocalPath(originalInputFileName);
            string commandPathStr =  ExecutablePath(@"mri-processing\niftiInit.bat");
            string strExecutableLocation = Path.GetDirectoryName(ExecutablesPath);
            var outputFile = Path.Combine(strExecutableLocation, strOutputFileName);
            string externalProcessArgs = string.Format("\"{0}\" \"{1}\"", inputFile.Replace(".nii", string.Empty), outputFile.Replace(".nii", string.Empty));
            var process = new ExternalProcess
            {
                CommandPath = commandPathStr,
                Arguments = externalProcessArgs,
                WorkingDirectory = LocalStoragePath
            };
  

Building Cloud Assembly

You need to specify the access details of Azure storage account associated with your azure batch app service. You will find the details on how to find out this azure storage details here (http://sarkar.azurewebsites.net/2015/03/16/uploading-large-executable-on-azure-batch-service-app-management-portal). Follow the instructions below to build the cloud assembly:

1. Add Azure Storage access details to DownloadFile method in TaskProcessor.cs file in the AzureBatchDependenciesStorage project.

2. Build the AzureBatchDependenciesStorage project.

3. Open the output folder of the AzureBatchDependenciesStorage project.

4. Select all the DLLs (and optionally PDB files) in the output folder.

5. Right-click and choose Send To > Compressed Folder.

Uploading the Application to Batch Apps Service

1. Open the Azure management portal (manage.windowsazure.com).
2. Select Batch Services in the left-hand menu.
3. Select your service in the list and click “Manage Batch Apps.” This opens the Batch Apps management portal.
4. Select Services in the left-hand menu.
5. Select your service in the list and click View Details.
6. Choose the Manage Applications tab.
7. Click New Application.
8. Under “Select and upload a cloud assembly,” choose your cloud assembly zip file and click Upload.
9. Under “Select and upload an application image,” choose your application image zip file and click Upload. (Be sure to leave the version as “default”.)
10. Click Done.

Running Jobs from Azure Batch Apps portal

1. Open the Azure management portal (manage.windowsazure.com).
2. Select Batch Services in the left-hand menu.
3. Select your service in the list and click “Manage Batch Apps.” This opens the Batch Apps management portal.
4. Select Services in the left-hand menu.
5. Select your service in the list and click View Details.
6. Choose the Manage Applications tab.
7. Click Run Jobs.
8. Enter Job Name, select Job Type and enter any parameters your job.
9. Under “Select the input files for your job” choose your input file. For simplicity, this sample project requires that these two files have these names (brain.nii and tissue.nii) . The GitHub project contains a zip file ( TestInputFiles.zip ) containing two input files and those files could be used here.
10. Then you could start the job.

Running Jobs using Azure Batch App Client APIs

Azure Batch Apps Could also be managed through Azure Batch App Client APIs. The client example ( ImageMagick.Console.Client ) project from “Microsoft Azure Batch Apps Samples” (https://code.msdn.microsoft.com/Azure-Batch-Apps-Samples-dd781172 ) demonstrates how this could be achieved. The source code and documentation from this sample project could be re-used easily to start submitting & monitoring the jobs for this sample project also.