Matt Seto, MSc

August 13, 2019

Using Azure Tools to Scale a Community Mapping Project

The Action Mapping Project (AMP) collects hand-drawn maps from middle-school and high-school aged youth around the Tacoma region. These maps include details about areas and routes that are used or avoided by youth in the region. Data gleaned from maps are being used to inform decision-making about community infrastructure and are helping to record the collective memory of young residents in a community. Now entering its 4th year, the project has grown to include dozens of schools in Tacoma and has collected thousands of hand-drawn maps ready to be processed.

A student hand-drawing maps in response to AMP questions

Previously data processing was a manual operation requiring a GIS operator to convert drawings to digital images, and then process them through a logic model to derive common responses and identify specific areas of interest common among student responses. This process is not scalable, and as the project anticipates increasing growth in upcoming years we needed a solution to automate and eliminate the manual processing of these maps so that we could adapt to the upcoming increase in responses.

Thanks to Azure space supplied by Microsoft, we were able to rapidly prototype automated solutions to improve scalability of the project. Firstly, Azure Blob Storage represented a fast and easy way to implement a centralized data repository for our mapping teams. Using the readily available blob documentation we quickly set up a web portal to handle uploading and sorting into blob containers for pre-processing.

To process our data we set up an Azure virtual machine running ESRI geospatial software. Our VM uses a bash job to kick-off a python script to process all of our data. Simplified, the model pulls any new maps and stages them for processing by further sorting and categorizing the inbound maps and then running them through the logic model to determine areas of interest. Once complete the results are posted back to a blob container that houses processed datasets. Prior to adopting the Azure workflow, the data processing procedure took roughly a day and required someone to push data through logic steps sequentially. After our initial run with the Azure VM we were able to process nearly 5000 maps in under 3 hours. Saving both time and effort.

The Azure toolset has allowed us to successfully and rapidly prototype an automated solution to allow scaling as our student map returns is set to increase. We have begun to identify additional Azure features that would suit our project as we grow. First is the addition of PostgreSQL as our DB solution for categorizing and organizing inbound map responses. Initial results show the Azure/PostgreSQL connection to be almost plug-and-play and very quick to set up. We are also excited by the automatic scaling and high-availability provided by Azure. Secondly, we are preparing to set up and deploy Azure-logic apps to automatically notify, log, and kick-off the VM bash script to process new inbound data. The drag and drop style coding of logic apps again allows for quick development and prototyping without the requirement of learning a new framework.

The Azure tools provided to us for this project have given us insight into how we should go about scaling up our research. We were able to prototype quickly and effectively with only a small development team, thus allowing more of our researchers the time to collect more student data. As the project continues to roll forward, we anticipate deeper integration with Azure and the associated toolset it provides.

 

For additional details for Action Mapping Project, contact Dr. Matt Kelley at the University of Washington Tacoma.

 

 

Matt Seto is the Technical Lead for the Action Mapping Project, Technical Project Manager for UWT’s Office of Community Partnerships, and UWT Alumni.