About Dotspotting


Dotspotting is the first project Stamen is releasing as part of Citytracking, a project funded by the Knight News Challenge. [1] We're making tools to help people gather data about cities and make that data more legible. Our hope is to do this in a way that's simple enough for regular people to get involved, but robust enough for real research to happen along the way.

There's currently a whole chain of elements involved in building digital civic infrastructure for the public, and these are represented by various Stamen projects and those of others. At the moment, the current hodgepodge of bits—including APIs [2] and official sources, scraped websites, sometimes-reusable data formats and datasets, visualizations, embeddable widgets etc.—is fractured, overly technical and obscure, held in the knowledge base of a relatively small number of people, and requires considerable expertise to harness. That is, unless you're willing to use generic tools like Google Maps. We want to change this. Visualizing city data shouldn't be this hard, or this generic.

So the first part of this project is to start from scratch, in a 'clean room' environment. We've started from a baseline that's really straightforward, tackling the simplest part: getting dots on maps, without legacy code or any baggage. Just that, to start. Dots on maps.

But dots on maps implies a few other things: getting the locations, putting them on there, working with them, and—crucially—getting them out in a format that people can work with.

We've had several interactions with different city agencies so far, and while the situation has changed alot in the last few years, we've realized that, for the foreseeable future, people aren't going to stop using Word and Excel and Pages and Numbers to work with their data, or even stop using paper. It's made us think that if this stuff is really going to work out in the long run, we need to focus our thinking on projects that can consume as well as export things that cities and people actually use and use now, and not stick with projects that have to rely on fancy APIs or the latest database flavor.

It's great that San Francisco and New York are releasing structured XML data, but Oakland is still uploading Excel spreadsheets (it's actually awesome that they do), and the Tenderloin police lieutenants are printing out paper maps and hand-placing colored stickers on them. At some point, if this really is the way things are going, we're going to need to meet the needs of actual functioning city agencies—and while APIs are great and necessary, for now that means Excel spreadsheets and Word docs. It also means being able to easily read in data that people have uploaded to google maps, interface with SMS systems like those that Ushahidi are pioneering. And it means being able to export to things like PowerPoint and Keynote, scary as that may seem.

What we've launched with is the baseline work that's being done to make this stuff internet-native. There's a login and permissions system that pretty much works. Uploading .csv files full of dots works. Each dot has an HTML page of its own, for example, like they do on Crimespotting. Collections of dots (we're calling them sheets) work, and you can export them. And there are dots on maps.

What's up with the funny map, above, you ask? That's an undocumented Easter egg that allows you to change the default base map for Dotspotting on the fly using something called a templated URL. A templated URL is just like any other URL on the web except that certain characters are replaced (filled in) dynamically by the software that reads them. For maps, a templated URL will consist of a host and a path followed by a string that looks like this: /{Z}/{X}/{Y} which tells the code that displays the map how to ask a server for a tile at a given zoom level ({Z}) a given column ({X}) and a given row ({Y}).

For example the templated URL for prettymaps map tiles would be: http://prettymaps.stamen.com/201008/tiles/isola/{Z}/{X}/{Y}.png.

You can ask Dotspotting to swap out the default toner map tiles with any templated URL by pass along a template argument with a sheet URL containing a URL, like this: http://www.dotspotting.org/u/12/sheets/191/?template=http://prettymaps.stamen.com/201008/tiles/isola/{Z}/{X}/{Y}.png&static=1.

Which is long and a bit squirrely and a bit scary so we've also the ability to pass along a named map tile provider instead. If Dotspotting sees a known provider it will take care of making sure the map tiles are set correctly. Instead of having to keep track of all that extra gobbledygook at the end of a Dotspotting URL (above) you can just write: http://www.dotspotting.org/u/12/sheets/191/?provider=prettymaps instead. Hooray!

There are currently seven providers that you can specify: lynch, midnightcommander and paledawn, all from the nice folks at CloudMade; mapquest-aerial and mapquest-road from MapQuest; openstreetmap and, of course, prettymaps. We'll add more soon and if there's a particular map provider you'd like to see be part of Dotspotting, let us know.

Speaking of which, the code for Dotspotting is available for download on Github, and licensed for use under the GNU General Public License. We're planning on releasing the code as we work on the project, in the hope that working in this kind of transparent manner from the beginning will both benefit the project and serve as an example of the way we'd like to work with foundations on this kind of work.

[1] You can read more about Citytracking & the Knight Foundation here and here.

[2] An Application Programming Interface (or API) is a structured way to talk to machines about data.