Welcome to my personal blog! I’m excited to start sharing my experiences, learnings, and insights from my 17+ years journey in data, analytics, and AI/ML engineering.

What to Expect

On this blog, you’ll find content about:

• Data Engineering: Building scalable data pipelines, ETL best practices, and modern data stack
• Machine Learning: ML model development, deployment, and MLOps strategies
• AI/LLMs: Working with Large Language Models, prompt engineering, and AI applications
• Cloud Architecture: Designing and implementing cloud-native solutions on AWS, Azure, and GCP
• Leadership: Team management, technical mentoring, and agile methodologies
• Best Practices: Code quality, testing, DevOps, and production-ready solutions

My Background

I’ve spent over 17 years specializing in building and managing the entire lifecycle of data, analytics, and AI/Machine Learning systems. My passion lies in:

• Designing scalable AI solutions that solve real-world problems
• Leading high-performing teams in agile environments
• Implementing DevOps practices for smooth operation
• Delivering impactful, user-centric solutions

Why This Blog?

I believe in giving back to the community that has helped me grow throughout my career. Through this blog, I aim to:

1. Share Knowledge: Document solutions to complex problems I’ve encountered
2. Learn in Public: Deepen my understanding by explaining concepts
3. Connect: Build relationships with fellow engineers and technologists
4. Contribute: Help others accelerate their learning journey

Stay Connected

I’ll be posting regularly about technical deep-dives, tutorials, case studies, and reflections on technology trends. Feel free to reach out if you have questions or want to discuss any topics!

You can find me on:

• GitHub
• LinkedIn
• Twitter

Let’s learn and build amazing things together! 🚀

Have a topic you’d like me to write about? Drop me an email or connect on social media!

Preparing Azure Batch Service

You need to have Visual studio 2013 or above to build this project. This project uses nugget packages:

• Microsoft Azure Batch Apps Cloud SDK (PM> Install-Package Microsoft.Azure.Batch.Apps.Cloud –Pre)
• Windows Azure Storage (PM> Install-Package WindowsAzure.Storage –Pre)

You need to create Azure batch Service from Azure Management portal if you haven’t done it yet.

Workflow to Publish and Run Azure Batch Applications

This article (http://azure.microsoft.com/en-us/documentation/articles/batch-technical-overview/) describes the basic concepts required for Azure Batch Service and ‘Workflow to publish and run an application with Batch Apps’ section of this article demonstrates workflow to publish and run Azure Batch Apps. Briefly Azure Batch apps consist of two major components (Application image and cloud assembly).

Diagram source: http://azure.microsoft.com/en-us/documentation/articles/batch-technical-overview/

Application Image

An application image is a zip file containing application executables and related necessary support files. A dummy test application image (mri-processing-dummy.zip) is included in the GitHub project. This zip contains two Windows Batch Files (niftiInit.bat and skullStrip.bat) which copies text content of input file to output files.

Cloud Assembly

A Cloud Assembly is a zip file containing cloud assembly that will invoke and dispatch workload to Azure Batch Service. It contains a Job Splitter and a Task processor. Cloud assembly is not included in the GitHub project, you make it by yourself by compiling the project itself and zipping the content of the output folder.

Entry Point to Azure Batch Apps

ApplicationDefinition represents the main entry point in the cloud assembly. This definition includes the following:

• Name of the Job
• Name of the Application
• Job Splitter – This step gives us the possibility to split a job in multiple tasks
• Task Processor – which invokes the application executable for a given task

public class ApplicationDefinition 
{ 
    public static readonly CloudApplication Application = new ParallelCloudApplication 
    { 
        ApplicationName = "AzureBatchNiftiProcessing", 
        JobType = "AzureBatchNiftiProcessing", 
        JobSplitterType = typeof(AzureBatchNiftiProcessingJobSplitter), 
        TaskProcessorType = typeof(AzureBatchNiftiProcessingTaskProcessor) 
    }; 
}

Job Splitter

Job Splitter does the splitting of your job into multiple tasks. This enables you to run multiple parallel and dependent tasks. For More detailed information, please read this article (http://azure.microsoft.com/en-us/documentation/articles/batch-dotnet-get-started/).

protected override IEnumerable<TaskSpecifier> Split(IJob job, JobSplitSettings settings)
{
    var reorientTask = new TaskSpecifier
    {
        TaskId = TaskIds.Reslice,
        RequiredFiles = job.Files.Take(1).ToList(),
        Parameters = job.Parameters,                
    };
    
    var reorientTask2 = new TaskSpecifier
    {
        TaskId = TaskIds.Reslice1,
        RequiredFiles = job.Files.Take(2).ToList(),
        Parameters = job.Parameters,
    };
    
    var skullStripTask = new TaskSpecifier
    {
        TaskId = TaskIds.SkullStrip,
        Parameters = job.Parameters,
        DependsOn = TaskDependency.OnId(TaskIds.Reslice)
    }.RequiringAllJobFiles(job);
    
    var skullStripTask2 = new TaskSpecifier
    {
        TaskId = TaskIds.SkullStrip1,
        Parameters = job.Parameters,
        DependsOn = TaskDependency.OnId(TaskIds.Reslice1)
    }.RequiringAllJobFiles(job);
    
    return new List<TaskSpecifier> { reorientTask, reorientTask2, skullStripTask, skullStripTask2 };
}

Defining & Running Tasks

At this stage you define each of the non merge task specified and returned from the job splitter. It invokes application with the appropriate arguments, and return a collection of outputs that need to be kept for later use. The following could be specified for ExternalProcess:

• CommandPath – The path of the executable file
• Arguments – The command line argument for the executable
• WorkingDirectory – The working directory for the external program process
• CancellationToken – Gets or sets a cancellation token which can be used to cancel the external

var originalInputFileName = task.RequiredFiles[0].Name;
var inputFile = LocalPath(originalInputFileName);
string commandPathStr = ExecutablePath(@"mri-processing\niftiInit.bat");
string strExecutableLocation = Path.GetDirectoryName(ExecutablesPath);
var outputFile = Path.Combine(strExecutableLocation, strOutputFileName);

string externalProcessArgs = string.Format("\"{0}\" \"{1}\"", 
    inputFile.Replace(".nii", string.Empty), 
    outputFile.Replace(".nii", string.Empty));
    
var process = new ExternalProcess
{
    CommandPath = commandPathStr,
    Arguments = externalProcessArgs,
    WorkingDirectory = LocalStoragePath
};

Building Cloud Assembly

You need to specify the access details of Azure storage account associated with your azure batch app service. You will find the details on how to find out this azure storage details here (http://sarkar.azurewebsites.net/2015/03/16/uploading-large-executable-on-azure-batch-service-app-management-portal).

Follow the instructions below to build the cloud assembly:

1. Add Azure Storage access details to DownloadFile method in TaskProcessor.cs file in the AzureBatchDependenciesStorage project.
2. Build the AzureBatchDependenciesStorage project.
3. Open the output folder of the AzureBatchDependenciesStorage project.
4. Select all the DLLs (and optionally PDB files) in the output folder.
5. Right-click and choose Send To > Compressed Folder.

Uploading the Application to Batch Apps Service

1. Open the Azure management portal (manage.windowsazure.com).
2. Select Batch Services in the left-hand menu.
3. Select your service in the list and click “Manage Batch Apps.” This opens the Batch Apps management portal.
4. Select Services in the left-hand menu.
5. Select your service in the list and click View Details.
6. Choose the Manage Applications tab.
7. Click New Application.
8. Under “Select and upload a cloud assembly,” choose your cloud assembly zip file and click Upload.
9. Under “Select and upload an application image,” choose your application image zip file and click Upload. (Be sure to leave the version as “default”.)
10. Click Done.

Running Jobs from Azure Batch Apps Portal

1. Open the Azure management portal (manage.windowsazure.com).
2. Select Batch Services in the left-hand menu.
3. Select your service in the list and click “Manage Batch Apps.” This opens the Batch Apps management portal.
4. Select Services in the left-hand menu.
5. Select your service in the list and click View Details.
6. Choose the Manage Applications tab.
7. Click Run Jobs.
8. Enter Job Name, select Job Type and enter any parameters your job.
9. Under “Select the input files for your job” choose your input file. For simplicity, this sample project requires that these two files have these names (brain.nii and tissue.nii). The GitHub project contains a zip file (TestInputFiles.zip) containing two input files and those files could be used here.
10. Then you could start the job.

Running Jobs using Azure Batch App Client APIs

Azure Batch Apps Could also be managed through Azure Batch App Client APIs. The client example (ImageMagick.Console.Client) project from “Microsoft Azure Batch Apps Samples” (https://code.msdn.microsoft.com/Azure-Batch-Apps-Samples-dd781172) demonstrates how this could be achieved. The source code and documentation from this sample project could be re-used easily to start submitting & monitoring the jobs for this sample project also.

Over 2GB seems like a very large application image, some programs could become very large, but it is rare to see programs above 200-300 MB. This was probably the reason why this 2GB limit in batch management portal makes sense. But in some cases some people might need to upload some application package which exceeds this size limit. Naturally it is not the executables themselves that are so large. For example to run image processing algorithm on volumetric (3D) brain images might require to use some template files that makes this application package big.

The Workaround

Even if Batch service App management portal doesn’t allow users to upload application images which exceeds this limit, we could still be able to do it by using AzCopy (http://azure.microsoft.com/en-us/documentation/articles/storage-use-azcopy/) or similar Azure storage application to upload those. To be able to do it, we need to understand how Azure Batch Service App uses Azure Storage.

Understanding Azure Batch Storage

A storage account is automatically created when Azure batch App Service is created. This is created in the same region where Batch app service is created and the storage name starts with ‘batchapps’. The exact naming convention of the storage account is “batchapps” + (random padding up to 24 chars). The storage account for a particular service can be found via Batch Apps management portal: Select service, “View details” -> “Sync access Keys”.

Getting Storage Access Keys

Once you get Azure storage account from Batch portal, you could proceed to Management Portal to get storage access key which will be required to upload these executables. For detailed instructions, please visit ‘View, copy, and regenerate storage access keys’ section of this article (http://azure.microsoft.com/en-us/documentation/articles/storage-create-storage-account/).

Storage Container Naming Convention

The storage account where this executable ends up is created in your subscription so you can actually locate this and upload your zip to a container called:

• “clouddrives” (for cloud assemblies)
• “greenbutton-cloud-assemblies” (for application image)

Create it if it doesn’t exist. The naming convention of the storage account but is a not-so-attractive-looking generated string and should stand out! The blob should be named ApplicationName.zip where ApplicationName is that defined in your Cloud Assembly.

Uploading Large Files

Following the above mentioned naming convention, now you can use AzCopy or any other tools of your choice to upload those to storage so that Azure Batch App service finds those apps correctly.

Additional Resources

• To learn more about Azure Batch, please visit http://azure.microsoft.com/en-us/documentation/services/batch/
• You could find Azure storage related information here (http://azure.microsoft.com/en-us/documentation/services/storage/)

• Using Azure batch API
• Using Azure Batch Apps API

If you would like to learn more about how to get started with Azure Batch Apps please read tutorials from this article (http://azure.microsoft.com/en-us/documentation/articles/batch-dotnet-get-started/).

Azure Batch Apps is a feature of Azure Batch that provides application-centric way of managing and executing Batch workloads and this article applies to Azure Batch Apps only. It includes a management Portal where you can manage jobs, view logs and download outputs without having to write your own client code.

Task Dependencies in Batch Apps

Job splitter will split job in subtask. Using Azure Batch Apps API some developer might be interested to use Azure Batch single job to run sequence of consecutive tasks where some phases are embarrassingly parallelizable and some are not. These tasks have several interdependencies, where some tasks can be run in parallel but others must wait for the previous tasks to provide their outputs. They still would like to run the whole pipeline as a single Azure Batch Apps job for simplicity and leverage the Azure Batch Management portal to submitting jobs and downloading outputs.

As of today there is no way in the API to specify intermediate task output files (which doesn’t exist during the job submission) as one of the required files. I tried few ways to make this work within the API, but it didn’t work. RequiredFiles takes IFileSpecifier instead of just a string (e.g. “intermediate.jpg”) and there is no implementation available for IFileSpecifier. I implemented the interface on my own specifying file name as “intermediate.jpg”.

public class MyFileSpecifier : IFileSpecifier
{
  public string Name { get; set; }
  public DateTime Timestamp { get; set; }
  public string OriginalPath { get; set; }
  public string Hash { get; set; }
} 

But this didn’t work (An exception occurred processing task 2: One or more errors occurred. The remote server returned an error: (404) Not Found).

The batch apps runtime doesn’t seem to download intermediate file from the job’s container and hence ‘Task.RequiredFiles’ doesn’t contain outputs from previous completed tasks, only the original input file is available. It make sense for batch apps runtime not to download all the intermediate output files because you don’t always need these files in the Embarrassingly Parallel compute. Besides it could slow down the task execution time significantly and could be waste of disk space downloading these intermediate files unnecessarily.

Solution Approaches

As the runtime doesn’t seem to download intermediate files, the batch application cloud assembly can download only the required files from the job’s container to the target virtual machines. Job’s container is located inside the azure storage account associated with batch service. There are several ways to achieve this and some of these includes the following:

• Using Azure Batch Apps REST API (https://msdn.microsoft.com/en-us/library/azure/dn820126.aspx), but this requires OAuth 2.0 authentication of your application with Azure Active Directory (Requests must be authenticated using an OAuth 2 bearer token issued by Azure Active Directory) which involves several more steps in Azure portal and bit more coding required to make that works. I would write another blog post showing this in future.

• Using Azcopy or Azure Storage library (Microsoft.WindowsAzure.Storage nuget) is the simplest approach to get this working along with Azure Batch Apps .NET API in the cloud assembly.

Using Azure Storage Library to Get Required Files in TVM

A storage account is automatically created when Azure batch App Service is created. This is created in the same region where Batch app service is created and the storage name starts with ‘batchapps’. We need to get the Azure Storage account Name and access key in our cloud assembly in order to get those file downloaded in the task virtual machines.

We need to add the code to ParallelTaskProcessor implementation of our code.

protected override TaskProcessResult RunExternalTaskProcess(ITask task, TaskExecutionSettings settings)
{
    string blobName = "resliced_brain.nii";
    string blobContainerName = "job-" + task.JobId.ToString();
    string targetFolder = LocalStoragePath;
    string strFileDownloaded = string.Empty;
    string storageAccountName = "[YOUR BATCH STORAGE ACCOUNT NAME]";
    string storageAccountKey = "[YOUR BATCH STORAGE ACCOUNT KEY]";
    
    string connectionString = string.Format(@"DefaultEndpointsProtocol=https;AccountName={0};AccountKey={1}",
       storageAccountName, storageAccountKey);

    //get a reference to the container where you want to put the files
    CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(connectionString);
    CloudBlobClient cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient();
    CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference(blobContainerName);
    CloudBlockBlob blobSource = cloudBlobContainer.GetBlockBlobReference(blobName);

    if (blobSource.Exists())
    {
        //blob storage uses forward slashes, windows uses backward slashes; do a replace
        // so localPath will be right
        string localDestination = Path.Combine(targetFolder, blobSource.Name.Replace(@"/", @"\"));
        //if the directory path matching the "folders" in the blob name don't exist, create them
        string dirPath = Path.GetDirectoryName(localDestination);
        
        if (!Directory.Exists(localDestination))
        {
            Directory.CreateDirectory(dirPath);
        }
        blobSource.DownloadToFile(localDestination, FileMode.Create);
        strFileDownloaded = localDestination;
    }
    
    string strInputFileName = "resliced_brain.nii";
    string blobContainerName = "job-" + task.JobId.ToString();
    string inputFile = DownloadFile(blobContainerName, strInputFileName, LocalStoragePath);

    // code to specify this input File as a parameter to run the external processs task
}

Source code for the cloud assembly demonstrating this could be found at (https://github.com/spsarkar/AzureBatchDependenciesStorage)

Additional Resources

• Sign up for the preview here (https://account.windowsazure.com/PreviewFeatures)
• Learn about Batch (http://azure.microsoft.com/en-us/documentation/services/batch/)