Manage Azure Blobs with F#

Hello World!

So you are going to notice a slight shift in this blog to start incorporating not only video game development, but hardcore data analytics.  As part of that shift, I am going to start incorporating F# into my standard set of languages as it is the language of hardcore data analytics if you roll with the .NET stack.

This particular article is about building a console based blob manager in F# instead of C#.  The very first thing I noticed about using F# to manage my blobs as opposed to C# is just the sheer reduction in lines of code.  The code presented here is a port of the C# article located here.  This code will eventually make its way into a production system which is part of a big data solution I am building.  New data sets that we acquire will be uploaded into blob storage, an entry stored into a queue, with a link to the data set.  Once a job is prepared to run, the data will be moved to Hadoop to do the processing and then stored in its final location.  So step 1 is…Store data in Blob storage.

So lets take a look at some code.

Blob Provider

 

module BlobProvider

open Microsoft.WindowsAzure.Storage
open Microsoft.WindowsAzure.Storage.Auth
open Microsoft.WindowsAzure.Storage.Blob
open Microsoft.WindowsAzure
open System.Configuration
open System.IO

let GetStorageContainer containerName = 
    try
        let conString = Microsoft.WindowsAzure.CloudConfigurationManager.GetSetting("StorageConnectionString")
        let storageAccount = CloudStorageAccount.Parse conString
        let blobClient = storageAccount.CreateCloudBlobClient()
        let blobContainer = blobClient.GetContainerReference(containerName)
        blobContainer.CreateIfNotExists() |> ignore
        let bcPermissions = new BlobContainerPermissions()
        bcPermissions.PublicAccess = BlobContainerPublicAccessType.Off |> ignore
        blobContainer.SetPermissions(bcPermissions) |> ignore
        blobContainer
    with
        | ex -> printfn "Error: %s" ex.Message; null

let UploadToBlockBlob containerName blobName filePath =
    try
        let blobContainer = GetStorageContainer containerName
        let blockBlob = blobContainer.GetBlockBlobReference(blobName)
        use fileStream = System.IO.File.OpenRead filePath
        blockBlob.UploadFromStream fileStream |> ignore
    with
        | ex -> printfn "Error: %s" ex.Message; 

let ViewBlockBlobs containerName =
    try
        let storageContainer = GetStorageContainer containerName
        storageContainer.ListBlobs() 
        |> Seq.iter (fun (blob : IListBlobItem) -> 
            if blob 😕 CloudBlobDirectory then
                let blob = blob :?> CloudBlobDirectory
                blob.ListBlobs() 
                |> Seq.iter (fun (blob2 : IListBlobItem) ->
                    if blob2 😕 CloudBlockBlob then
                        let blob2 = blob2 :?> CloudBlockBlob
                        printfn "Blob Name: %s" blob2.Name
                    else
                        let blob2 = blob2 :?> CloudPageBlob //there are only 2 options, so it must be page.
                        printfn "Blob Name: %s" blob2.Name
                )
            else
                printfn "Returned incorrect directory type: %s" (blob.GetType().ToString())
        )
    with
        | ex -> printfn "Error: %s" ex.Message

let DownloadBlockBlob containerName blobName filePath =
    try
        let storageContainer = GetStorageContainer containerName
        let blob = storageContainer.GetBlockBlobReference blobName
        use fileStream = System.IO.File.OpenWrite filePath
        blob.DownloadToStream fileStream |> ignore
    with
        | ex -> printfn "Error: %s" ex.Message


let DeleteBlockBlob containerName blobName =
    try
        let storageContainer = GetStorageContainer containerName
        let blob = storageContainer.GetBlockBlobReference blobName
        blob.Delete() |> ignore
    with
        | ex -> printfn "Error: %s" ex.Message

Program Execution

module AnalyticsProgram

open BlobProvider
open System

let UploadBlob () = 
    let containerName = "datastaging"
    printfn "Please type the blob name followed by enter."
    let blobName = Console.ReadLine()
    let blobName = "data/" + blobName
    printfn "Please type the file path of the data file followed by enter."
    let filePath = Console.ReadLine()
    printfn "Please wait while the file is uploaded."
    BlobProvider.UploadToBlockBlob containerName blobName filePath
    printfn "Completed.  Press any key to continue."
    Console.ReadKey

let ViewBlob () =
    let containerName = "datastaging"
    BlobProvider.ViewBlockBlobs containerName |> ignore
    printfn "Completed.  Press any key to continue."
    Console.ReadKey

let DownloadBlob() =
    let containerName = "datastaging"
    printfn "Please type the blob name followed by enter."
    let blobName = Console.ReadLine()
    let blobName = blobName
    printfn "Please type the desired download location."
    let filePath = Console.ReadLine()
    printfn "Please wait while the file is downloaded."
    BlobProvider.DownloadBlockBlob containerName blobName filePath |> ignore
    printfn "Completed.  Press any key to continue."
    Console.ReadKey

let DeleteBlob() =
    let containerName = "datastaging"
    printfn "Please type the blob name followed by enter."
    let blobName = Console.ReadLine()
    printfn "Please wait while the blob is deleted."
    BlobProvider.DeleteBlockBlob containerName blobName |> ignore
    printfn "Completed.  Press any key to continue."
    Console.ReadKey

[<EntryPoint>]
let main argv = 
    let mutable command = "string"
    while (command.CompareTo("exit") <> 0) do
        printfn "*********************"
        printfn "Cmds: exit upload view download delete"
        printfn "Please type a command followed by enter."
        command <- Console.ReadLine()
        if command.CompareTo("upload") = 0 then 
            UploadBlob() |> ignore
        elif command.CompareTo("view") = 0 then
            ViewBlob() |> ignore
        elif command.CompareTo("download") = 0 then
            DownloadBlob() |> ignore
        elif command.CompareTo("delete") = 0 then
            DeleteBlob() |> ignore
        elif command.CompareTo("exit") <> 0 then
            printfn "Improper Command.  Cmds are case sensetive."
            printfn "Commands are: exit upload view download delete"
    0 // return an integer exit code

Summary

So when I set about doing this task, I was expecting F# to be more similar to JavaScript than C#.  After going through this exercise, I learned quite a lot about the F# language.  F# to me appears to be more C# like than JavaScript like, but provide better pipelining of results of functions into new functions and streamlining data transformation tasks.  There are some obvious difficulties I ran into, such as trying to figure out how to do type checking in F#, but that’s just like when I had to discover the typeof keyword in C# for the first time.  Overall my code size is down significantly and I have started getting he hang of this thing.  You can download the full project from here.  Note that you need to go into the App.config and in the connection string, replace the Account Name and Account Key with your own.

 

One thought on “Manage Azure Blobs with F#

  1. Pingback: F# Weekly #47, 2014 | Sergey Tihon's Blog

Leave a Reply

Your email address will not be published. Required fields are marked *