Exciting Times in MarkLogic Geospatial

by Jennifer Tsau

Data, Data, Everywhere

One of the most exciting results of the mobile revolution is that users (both consumers and enterprises) everywhere are connected and empowered through technology.

That's right -- everywhere. Mobile technologies generate a plethora of geospatial data, and Internet of Things devices will spawn even more of it. According to a Pew Research Center report1, more than 70% of adult smartphone users are already sharing their location information to interact with their mobile apps. Location-based services in mobile apps using global positioning satellites (GPS) data is just one part of the "location intelligence" story. Websites provide targeted advertisements and services based on a visitor's internet protocol (IP) address location. Retail stores track the movement of goods within a store with radio-frequency identification (RFID) tags. Blue-tooth beacons are enabling cities to provide navigation services within public facilities for the visually impaired. Companies are utilizing geofencing as security solutions to prevent access to services or data based on a user's location.

Without a doubt, location intelligence and location-based services are becoming more prevalent. Enterprises understand this and are increasingly integrating geospatial data into their ecosystems to build richer application functionality and "smarter", contextualized operational processes.

In MarkLogic 8.0-4, we've made it even easier to develop applications with powerful geospatial functionality on top of MarkLogic.

New Geospatial Features in MarkLogic 8.0-4

In MarkLogic 8.0-4, we introduced a number of geospatial search enhancements. This includes 13 new conversion/validation functions for common geospatial data serializations (e.g. GeoJSON, KML, GML, etc.), 13 new geospatial utility functions, and support for newer versions of KML and GML. All of the new APIs are supported in both JavaScript and XQuery.

Since visualizing geospatial concepts is always fun, I decided to build a little app to showcase some of the new APIs. Head over to this GitHub repository, clone it to your desktop, and setup the app to follow along with my descriptions below.

Here's a screenshot of the app:

How I Built My Little App

The goal of my application was to provide a visualization of a handful of the new APIs, all of which are server-side APIs. Hence, I decided to make MarkLogic serve as both the database and application server.2 Most of my application's functionality (e.g. computing the results when you select a region) comes from MarkLogic evaluating the server-side JavaScript files (result.sjs, region-approx.sjs, convert.sjs).

For the map visualization, I chose to use the Google Maps JavaScript API, which is a popular and friendly choice for digital maps (as long as you have Internet access). Most of the client-side JavaScript (script.js) in my app is concerned with interacting with the Google Maps API (e.g. what happens when you click on a shape, adding markers on the map when you click a button) and updating the HTML to display results.

Finally, for the sake of comprehensiveness, I'll also mention that I used jQuery and Pure.css for the UI.

Playing with the App and the "Take-home Message"

Conversion Functions

Play With the App: Select a region on the map. You'll see the rest of the page populate with various geospatial data serializations for that same region. I wanted to illustrate how the same region can be represented in a variety of formats3, and the heterogeneous type of data you'll likely come across when gathering geospatial data to use in your application.

"Take-home Message": There's a lot of data munging that goes on in application development. Now, anyone familiar with geospatial data will tell you that the diversity in geospatial data formats often leads to a lot of "munging" before one can do anything useful with that data. What we've done in 8.0-4 is introduce a number of APIs for converting common geospatial data serializations (e.g. GeoJSON, KML, GML, etc.) to and from MarkLogic's internal representation (the cts:region base type, which you can read more about here).

That means we've saved you from spending a bunch of development time doing a lot of parsing / converting work. At MarkLogic, we want you to save your brain power to do bigger and better things (than writing code to do such things such as replacing parentheses with brackets). These APIs make it very easy to feed geospatial data from a variety of sources into MarkLogic, have MarkLogic interact with the data, and then feed the data back out to whatever format the various client applications may expect, should that be GeoJSON, KML, and so forth.

The new 8.0-4 APIs I've used here include geojson.toGeojson(), geokml.toKml(), geogml.toGml(), georss.toGeorss(), geo.toWkb(), and geojson.parseGeojson().

(See the release notes for a full list of the new conversion functions.)

Utility Functions

Play With the App: After you've selected a region on the map, click one of the buttons on the right side of the GeoJSON Format box. (For "Approximate my region!", you'll want to first input a threshold value. I'd recommend a value under 1 since my regions are spanning fairly small distances.) Based on what you've clicked, you will see a marker or shape added to the map.

"Take-home Message": Continuing the story of MarkLogic capabilities that make life easier, many of the APIs we provide make it easier and faster for developers to create their applications. These new 8.0-4 APIs fit that profile, and the ones I've used in my application include:

  • geo.countVertices() returns the number of vertices in a region. This may be useful for understanding the complexity of a given geospatial region, so that you can use geo.regionApproximate() to simplify the region (and by consequence, simplify the computation process for geospatial operations performed on that region thereafter). Or, you may want to build logic within your application to validate/categorize any user input region (e.g. add an attribute to any input linestring with more than X vertices, etc.)
  • geo.interiorPoint() returns a point that is guaranteed to be within the bounds of the given region. You might want to use this API when you have documents in your database that already contain regions such as polygons or linestrings, but you'd prefer to get results back in which those regions are represented as points. You might build an app where users can search for gas stations in a neighborhood -- although the gas stations are represented as polygons in your existing document data, you use geo.interiorPoint() because you'd rather show the gas stations as points on the map instead. This API can also come in handy when you want to find the distance between two regions using geo.distance(), and you only want a rough estimate instead of exactly specifying two given points (from, say, vertices on the boundaries of those regions) to calculate the distances.
  • geo.polygonToLinestring() constructs a linestring from the vertices of a polygon. Say, you have a lake represented as a polygon in a document, but you want to work with the lake's shoreline instead and represent it as a linestring. This could also come in handy should you want to concatenate the border of a polygon with a series of other linestrings by using geo.linestringConcat().
  • geo.regionApproximate() returns a simplified approximation of the region, using the Douglas-Peucker algorithm. I might have many large regions with a large number of vertices (e.g. detailed polygons for states west of the Mississippi), and I want to check for linestrings (e.g. interstate highways) in my database that intersect all those regions using another new 8.0-4 API cts.matchRegions. geo.regionApproximate could come in handy here to simplify the polygons and speed up the operation.

(See the release notes for a full list of the new utility functions.)

Now It's Your Turn

Now that you've had a brief glance of our new MarkLogic geospatial 8.0-4 functionality in action, I'd encourage you to explore further and check out the rest of what's new in 8.0-4!

The next time you're thinking about building an app with location-based services, let MarkLogic be at the top of your list for the bottom of your stack!

-----------

1"Seventy-four percent of adult smartphone owners ages 18 and older say they use their phone to get directions or other information based on their current location." Source: "Mobile Technology Fact Sheet," Pew Research Center. (link ).

2 As best practice in production environments, we recommend a 3-tier architecture. Check out our Reference Application Architecture Guide for more information. In this case, we're just focusing on exploring and getting familiar with MarkLogic's server-side geospatial capabilities, and there is no business logic to separate from the database so I haven't implemented a 3-tier architecture here.

3Thankfully, the new APIs allowed me to convert a cts:region to these formats with a single line of code each. Look at result.sjs to see where these APIs are used.

Comments

  • Hi, great tutorial but it doesn't seem to be working on my local ML server (8.0-5.4). I'm getting a 500 error from the server when clicking on one of the shapes. The error message is "XDMP-BADPOLYGON: cts:polygon(()) -- Illegal polygon". The request payload has all of the points so I'm guessing it might just be a parsing problem. ML Log file: 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: XDMP-BADPOLYGON: cts:polygon(()) -- Illegal polygon 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: in /MarkLogic/geospatial/geojson.xqy, at 128:6, 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: in polygon(object-node{"type": text{"Polygon"}, "coordinates": array-node{array-node{}}}) [1.0-ml] 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: $polygon-or-points = object-node{"type": text{"Polygon"}, "coordinates": array-node{array-node{}}} 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: $vertices = () 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: in /MarkLogic/geospatial/geojson.xqy, at 159:18, 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: in complex-polygon(object-node{"type": text{"Polygon"}, "coordinates": array-node{array-node{}}}) [1.0-ml] 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: $complex-polygon = object-node{"type": text{"Polygon"}, "coordinates": array-node{array-node{}}} 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: in /MarkLogic/geospatial/geojson.xqy, at 204:26, 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: in parse-geojson#1({"type":"Polygon", "coordinates":[[]]}) [1.0-ml] 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: $regions = object-node{"type": text{"Polygon"}, "coordinates": array-node{array-node{}}} 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: $r = object-node{"type": text{"Polygon"}, "coordinates": array-node{array-node{}}} 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: in /result.sjs, in convertGoogObjToCtsRegion() [javascript] 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: in /result.sjs, at 14:42 [javascript] 2016-07-10 23:33:08.671 Notice: GeoMarklogicApp: in /result.sjs, at 81:17, in convertGoogObjToCtsRegion() [javascript] Keen to hear if you have any ideas. Thanks
    • Hey James, Thanks for commenting! Ah -- good catch -- looks like the updates to the Google Maps JavaScript API caused some incompatibility. If you edit your default.html file to replace: <script src="https://maps.googleapis.com/maps/api/js?callback=initMap" async defer></script> with: <script src="https://maps.googleapis.com/maps/api/js?v=3.23&callback=initMap" async defer></script> ....that should fix the issue. I've updated the repository with that edit as well. Let me know if that doesn't work. Thanks again for catching that, Jennifer
      • Perfect, that did the trick - thanks! One thing it might be worth noting in the Readme is that I think you need to have a google maps key for it to work, it wasn't letting me load the map otherwise. Pretty easy to spot from the javascript console mind you https://developers.google.com/maps/documentation/javascript/get-api-key
        • Thanks, James. Looks like requiring an API key may also be due to updates in the Google Maps JavaScript API pricing plan on June 22, 2016. (https://developers.google.com/maps/documentation/javascript/usage) Will make a quick note in the ReadMe now!