Exploratory IoT Analysis with R

Hello World!

These days I need to make videos instead of written articles, so I am going to post a few of those here.

In this video we will do an initial exploratory analysis on a water flow data set that came from a prototype that I built. The prototype consists of a water pump, a valve and a flow meter. The data set exists in SQL Azure. We will use R and R Studio to perform the analysis from an Azure virtual machine.

The code is:

Continue reading

Powering AzureML with Hadoop HBase

Hello World!

Today is a freaking cool day.  Why do you ask?  Because today I am writing an article on how to use two of the coolest freaking big data/data science tools out there together to do epic shit!  Lets start with HBase.  HBase is a way to have a big data solution with query performance at an interactive level.  So many folks are starting to just dump data into HBase.  In the project teddy solution, we are dumping tweets, dialogue and dialogue annotations to power our open domain conversational api.  There really is no other way that is easy to use for us to do this.

The second part of project teddy is to predict based on an incoming conversational component, what sort of response the speaker is attempting to illicit from the teddy bear.  If we power our teddy bear with predictive analytics and big data, this would be perfect.  What better platform to do this quickly and easily than AzureML?

This is a follow up article to this one: http://indiedevspot.com/2015/06/30/writing-tweets-to-hbase-simply/

Continue reading

Writing Tweets to HBase – Simply

Hello World!

Many folks may know that the South Florida Evangelism team is undertaking a task that many think is impossible.  Well, in that statement all I hear is “there is still a chance!”  The end goal is to create a teddy bear that can have a conversation about anything.  So step one is to collect as much dialogue as possible from as many sources as possible and annotate them.  What better place to power an association engine for word and phrase relevance than something that forces you down to 140 characters to get your message across.

So as any normal developer I decided to start by looking for samples already out there.  MSDN has a great starter for writing tweets and doing sentiment analysis with HBase and C#.  The only issue with the sample is, that it is very poorly written and difficult to understand with no separation of concerns.  So I want to go through simplifying the solution and separating a few concerns out.

Continue reading

Using Census Data to Help Pick your Child’s Name

Hello World!

So I had a life changing event this past Sunday at 8:55am 5/24/2015.  My first child was born!  Both child and wife are healthy and happy.  Everything is good in life.  Like many couples though, my wife and I struggled to find the right name for our child.  We didn’t want something too common, or was an old person name, or so rare and funky that nobody could spell it.  We also realized we just had a general lack in knowing what names were out there.  So after much debate and discussion over what to name her, I started doing a bit of an analysis using some census data.  I want to thank Jamie Dixon for providing the data that he found for use in his Dinner Nerds article.  The data itself can be found here.  This article will discuss the code used to go through all of the data and provide insights into child names.

Continue reading

How to Datamine Zillow

Hello World,

As many of you may know at this point, I am relocating to South Florida.  Final location to be determined, but will probably be renting around Pompano Beach or Fort Lauderdale while working out of Venture Hive and the Microsoft Fort Lauderdale Offices.  So what does this have to do with Zillow?  Well, It has EVERYTHING to do with Zillow.  What I’ve found while searching for homes is that between Realtors, Zillow and Trulia, they really just don’t have a predictive analytics solution that works for me.  So I decided to give a shot at AzureML to mash together a few datasets to send me notifications more to my liking than is currently being sent.  So step 1 in this plan is to data mine Zillow.  Luckily, Zillow has an api for that.  Or if you are feeling particularly frisky, Zillow gets their data from ArcGIS (example for Raleigh).  So lets get cracking…

Continue reading

So you want to be an Analytics Developer

Hello World!

I get a series of questions all the time.

  1. How do I switch careers to be a developer?
  2. How do I become a data scientist?
  3. How do I add intelligence to my code?
  4. How do I get a job in distributed computing?
  5. How do I code more analytically?

The answer to these questions are pretty much all the same.  Step 1, learn about it and build one piece of software focused on that goal.  Step 2, go for it, just do it.  So that said, Microsoft has a fantastic resource, Microsoft Virtual Academy, which provides free training around various topics from entry level to advanced.  This article focuses on a learning plan with MVA to attain the goal of becoming an Analytics Developer.

Continue reading

Intro to C# – The Analyst’s Way

Hello World!

I have recently been informed that many of my articles may be a bit advanced for folks, so I am going to kick off a series of C# articles dedicated to the Beginner to programming.  I have no idea how long this series is going to be, I’ll just keep adding to it as requests come in for various topics.  This series is meant to take the absolute beginner to a level in which they can possibly derive value from my other articles.  Those of you who do Jiu Jitsu with me, know you have to shrimp before you can roll, so this is sort of like shrimping.

Continue reading

Mocking IoT Telemetry Data with Azure

Hello World!

This should prove to be an interesting series of posts coming up, as I am working on a new project that is very unique and interesting. The idea is to use incoming data from Arduinos, Raspberry Pis, Gallileos, Edisons and other assortments of IoT type devices connected to oil and gas pipelines to determine if a leak is currently in progress and also predict if a leak is likely to occur in the future based on current and trending conditions.

My part in the project is all back end analytics, and I have very little to do with the actual telemetry and hardware. The telemetry will be posted using Azure Event Hubs, and thus my portion of the project begins with mocking that real time data at a large enough geo dispersed scale that I can develop a system that can handle it, and then switch my configurations to consume from the production event hubs. Since I am no longer a consultant working on projects with trade secrets and everything these days is about the elevation of skills in the community, I have posted everything on github that you can download and peruse at your leisure. Please note that this is in progress and well the github source may not necessarily work when you look. I’ll try to enforce a standard to comment “working – comment” on pushes to the repository. The git repository is located here: https://github.com/drcrook1/OilGas

Continue reading

Deploying F# Web Job to Azure

Hi All,

This article is one of those that is going to help remind me how to do this deployment, as it can be a bit tricky.  If you are working with F# for web jobs, like I have started doing, there are a few steps.

  1. Create a new console application
  2. Add proper nuget packages
  3. Manually add a .dll reference and copy said .dll to output
  4. Zip up and deploy entire output folder.

Continue reading

Manage Azure Blobs with F#

Hello World!

So you are going to notice a slight shift in this blog to start incorporating not only video game development, but hardcore data analytics.  As part of that shift, I am going to start incorporating F# into my standard set of languages as it is the language of hardcore data analytics if you roll with the .NET stack.

This particular article is about building a console based blob manager in F# instead of C#.  The very first thing I noticed about using F# to manage my blobs as opposed to C# is just the sheer reduction in lines of code.  The code presented here is a port of the C# article located here.  This code will eventually make its way into a production system which is part of a big data solution I am building.  New data sets that we acquire will be uploaded into blob storage, an entry stored into a queue, with a link to the data set.  Once a job is prepared to run, the data will be moved to Hadoop to do the processing and then stored in its final location.  So step 1 is…Store data in Blob storage.

Continue reading