In the first of our Project Focus blogs, Geospatial Data Engineer Tom Wiseman gives us a glimpse into some of the pioneering work undertaken at thinkWhere, part of the Idox group.
Can you give us an overview of a project you have been working on recently?
I’ve been spending time on a project to automate our third-party data uploads to theMapCloud, our innovative geospatial data platform.
The reason for this piece of work won’t be alien to anyone who’s had to source, process, load and maintain data from third-party sources – it simply takes time! We serve over 300 datasets via theMapCloud, which have been managed through semi-manual and manual processes. This has been taking up several days a month. Now we are checking for new data daily and applying the update.
What has the work entailed? What skills have you used?
I’ve mostly used pre-existing skills such as FME Desktop and FME Cloud with bits of SQL. There has been an element of learning our own applications and how they hang together, and ensuring the loading process we have put in place does all the things required of our systems.
What have you learned throughout the course of this piece of work?
Automating our data pipelines will create efficiency and improve the service we offer to our customers. I still find it surprising that despite the well-established data and metadata standards we have in the industry, there are still many idiosyncrasies to overcome.
It has also further cemented in my mind that there are significant gaps in data coverage that could be plugged by the Geospatial Commission and Central Government by harmonising many ‘local-government-owned’ datasets through the adaptation of standards and central point of publication. This would make it easier for the geospatial community to consume national data, innovate and provide solutions to real-world problems.
How will the project benefit thinkWhere and its clients?
Historically, the updating of third-party datasets was taking around 2.5 days a month, over the course of the year this adds up to 30 days. When this work is completed, we should have freed up most of those 30 days’ capacity, which can be used to work on other things for the benefit of our customers.
The new processes also ensure that we are keeping data up to date for our customers who rely on this data for making their own internal business decisions.
Ultimately these more efficient processes should see us able to onboard and serve more data to our customers and make theMapCloud a more appealing product.
Are there any wider benefits?
There are potential wide-ranging benefits for some of the more traditional Idox customer base. Many of these datasets are statutory planning constraints and as such I believe theMapCloud services dovetail nicely with many of the local government functions within Idox and can remove administrative overheads for our customers. Customers can use the rich catalogues of geospatial data from theMapCloud in any GIS application which is capable of consuming standard web services too.
There are also direct opportunities for theMapCloud geospatial data services to be consumed by other parts of Idox Geospatial or other parts of the Idox group to be included in their applications. Having trusted and reliable sources for our key geospatial data will generate new opportunities.
What’s next on your to-do list?
The next focus will be starting to identify other datasets of value which we can onboard into theMapCloud to provide even more value for our customers. Watch this space for more details!
