Project 6: Hands-On with Microsoft Azure — One-Page Summary

Student: Saloua Daouki
Project: Cloud Solution Deployment on Microsoft Azure
Goal: Deploy a real-world cloud solution that includes persistent storage, compute resources, and network security.

1 Persistent Storage

• Service Used: Azure Blob Storage (movielensstorage123)
• Description:
  • Created a Storage Account with Blob Containers to securely store large datasets.
  • Organized data for easy access, scalable storage, and long-term persistence.
  • Integrated storage directly with compute resources for smooth data pipelines.

2 Compute Resources

• Services Used:
  • Azure Databricks (MovieLens-DB):
    • Managed Spark cluster for big data processing and machine learning.
    • Runs data cleaning, transformation, model training, and output generation.
  • Azure OpenAI (finalproject-openai):
    • Supports advanced AI tasks and model experimentation.
  • Azure AI Foundry (salou-md1zl8vr-eastus2):
    • Used to deploy and test containerized models and scale AI services.

3 Network Security

• Setup:

  • All resources grouped in Resource Group (MovieLens-FinalProject) for organized management.

  • Network Security: Default Azure security settings and role-based access controls protect my storage and compute resources in this student deployment.

  • Managed secrets securely using Spark variables instead of storing or hard coding keys in my notebook.

4 Workflow Overview

• Spark jobs on Databricks read raw datasets from Blob Storage → process and analyze data → save results back or call Azure AI services.
• Demonstrates a secure, efficient, end-to-end pipeline for big data and AI in the cloud.

5 Key Benefits

• Uses Azure Free Account resources efficiently.

• Follows best practices for storage, compute, and security.

• Fully integrated with the final project for a real-world ML solution.

Fully deployed, secured, and managed in Microsoft Azure.

Notebook Link: My Notebook