OpenStreetMap old timers know about the infamous 2009 TIGER import of road data in the US that continues giving to this day. Our story has none of the, maybe deliberate, shenanigans (see the TIGER improvement project) that went on back then in the US, but there are clearly some similarities in the lessons that should be learnt.

Back in July of 2011 the Swiss community undertook a big effort to import municipality boundaries from swisstopo (the marketing name of the federal Swiss GIS department) (see osm.wiki/Switzerland/swissBOUNDARIES3D). Being an OSM n00b at the time with just a bit over a year editing experience I didn’t really do anything useful for the import proper, but I did organise the explicit permission needed from swisstopo as this was many years before their data would become available for use on open terms for us in September 2021. With a couple of technical hiccups along the way that are not really documented, we finally managed to complete the work by early August.

Fast forward to today: I’ve been going on for a few years now that we really need a quality assurance process so that we can discover and track differences between swisstopos data and what is in OSM. We knew and fully expected that there would be differences, because:

• at the time of the import we simplified the boundaries quite significantly because of the resource constraints of the computer hardware available to us,
• over the last 14 years we independently followed the mergers and other changes of the municipalities (see osm.wiki/Switzerland/2026_Municipality_Mergers), and didn’t expect this to improve the accuracy of the boundaries, and we knew that now and then there would be associated minor geometry changes that we wouldn’t be able to track,
• and then just general decay due to glueing and accidental modifications.

Thanks to work by our community member habi inspired by earlier work by Branko Kokanović from Serbia, we have now have daily QA data and boy, we were wrong.

… See full entry

Some of the readers may have experienced the sinking feeling when they just attempted to upload dozens if not 100s of changes and a network glitch resulted in the editor app failing the upload.

What’s the problem when that happens?

It is literally that your editing app doesn’t know if any new objects it created locally have been successfully created on the server or not. If they have been created, retrying the upload will create duplicates, and if you don’t retry you potentially loose your work.

This post is related to work that I’m doing for version 21 of Vespucci that automates all of this, but the interesting thing is that you can easily handle many of the cases manually without editor support.

As you may know OSM changesets are not atomic, but individual uploads are (yes, you can upload multiple times in to the same changeset). With other words if your upload succeeded, but your editing app lost the response from the server, all your changes will have been successfully made, and, the other way around, if your upload fails it will fail completely.

The other point to note is that changesets are opened in a separate operation and are only automatically closed after a timeout of an hour if they are not explicitly closed after the upload by your editing app.

A normal upload operation will roughly contain the following steps

• open the changeset.
• upload the data and process the response. Depending on your editing app it will update the downloaded data in place, or throw it away and re-download from the server.
• close the changeset.

This means that there are simple cases that you can handle manually, assuming you are not uploading more than once per changeset (more on that later):

… See full entry

Way back in 2015 I wrote this diary post Is OSM business unfriendly? on how core OSM limiting itself to data collection had fostered a bustling ecosystem.

Organisations from small to large, commercial and non-commercial have built their businesses on building OSM know-how, normalizing the data, aggregating it with other sources, and providing services on top of it. It is one of the major factors that has made OSM not just a hobby project, but a notable player on the global stage.

Yes, the Linux Foundations OMF is disruptive, but the disruption is mainly in that it will remove a major part of the raison d’être for these organisation in the wider OSMspace.

The surprising bit at SOTM-EU over the weekend was just how hilariously unaware both Linux Foundation members and representatives were that their main effect will be stomping out a whole raft of SMEs, and the sudden realisation by the victims that US big tech and their non-profit front are not their friends.

What does this mean for core OSM?

I’ve argued that we should be moving the boundary of what we do outwards so that we can at least provide more of what was provided by the layer of service providers. Not all of the market for geo data and services outside of the US tech bubble is going to enthusiastically embrace that their choice of suppliers has been reduced to a duopoly. But I will concede that doing nothing and letting the market forces play out is the more OSMish reaction.

PS: I would point out that all of the above has already chewed though multiple times in public. Maybe if the Linux Foundation doesn’t want to be put on the spot and look very very out of touch they should read what other people are saying about them.

I haven’t done a contributor statistics diary post in a long time, but I suspect that the latest update of the graphs on the wiki is likely to lead to some questions.

As we can see in

the number of new contributors to OSM has dropped quite a bit in 2022. As the overall number of contributors has always been dominated by the new contributors it isn’t a surprise that the overall number is down too.

The interesting question is if we can pinpoint any “source” of this reduction. As a 1st investigative step I gave the numbers for editing apps used by new editors a look. I’ve been producing the numbers for a long time, but have never published them previously, if we look at 2021 and 2022 we see:

… See full entry

Four years ago on May the 25th 2018 the, then new, EU-wide General Data Protection Regulations (GDPR) went in to force. It is therefore quite fitting that nearly to the day 4 years later, the OSMF has given up any pretense of ever fulfilling the legal requirements arising from the regulation and, perhaps more important, undertaking the ethically and morally indicated steps to protect the privacy of our contributors.

It isn’t as if the OSMF was caught by surprise in May 2018. The topic was by far the best researched, documented and planned change in its history. And it isn’t as if the quality of the work or the conclusions were in doubt, completely independent and unaware of what had been prepared by the LWG three years earlier, at last years SOTM there was a talk by Robert Riemann that came essentially to the same conclusions.

But, four years later, with the sole exception of new users having to accept the OSMF terms of service on sign up (added by yours truly), none of the required changes to the APIs and data access have even been started on OSMF properties.

To make the situation worse, OSM isn’t simply static, new services, third party and OSMF operated are being added, existing operations are changed and rarely are the data protection impacts and consequences considered.

Ironically some organizations in OSM-space have taken required steps, for example Geofabrik, OSMcha and Pascal Neis’s HDYC, that the OSMF has not.

Why have we ended in this fraught with legal and financial danger place? On the one hand there are no brownie points to be won with championing the changes to the OSM website, the API and data distribution. Since Frederik has left the board no director has been willing to show any support for the matter.

… See full entry

As every quarter I’ve updated the contributor statistics in the OSM wiki. The most notable effect is the drop off in new contributors in the 2nd half of 2021, which can at least partly be attributed to the demise of maps.me as a source of new mappers.

This is more obvious when you consider just the new users from this source

Note that the numbers include new users using organic maps, but even that doesn’t change the trend.

See my diary posts from 2020, 2018 and 2017 for more on the topic.

Just realized that I didn’t notice that on Tuesday (the 5th) I sent out the 10’000ths welcome message to a contributor.

I’ve been doing this for SOSM since December 2013, nearly 8 years ago, averaging a bit over 100 messages per month.

I’ve discussed our expectations (or rather the absence of them) before see osm.org/user/SimonPoole/diary/40071

PS: The happy “winner” was osm.org/user/JFischli

As a rule of thumb, the primary name would be the most obvious name of the feature, the one that end users expect data consumers to expose in a label or other interface element.

From the OSM wiki Key:name

I’m currently tidying up a couple of loose ends in preparation for the release of version 16 of Vespucci. One of those dangling bits was supporting the new format of the name-suggestion-index (NSI).

The NSI was conceived back in 2013 by Aaron Lidman as a list of canonical name spellings for chains of stores, restaurants and other similar facilities, most notably for use in iD. Vespucci has supported use of the NSI nearly since day one, see a diary post from 2014 (this works quite differently in current Vespucci versions). Since then it has morphed to a, supposedly, authoritative source of tagging for a very wide range of objects. Some would say that the current version has expanded its reach far beyond what is actually useful (flagpoles ffs), but that is not the topic of this post.

Prior to the release of iD 2.20 the NSI had been in a long period of stasis with updates being made to the data, but these were not actually being deployed and given the larger format changes I had all but forgotten about it and the data in Vespucci was really old. However over the last couple of months some colleagues pointed out some weird behavior in Vespucci and iD when using NSI generated presets for tagging. In particular name tags are being added to things like excrement bag dispensers and automated postal package depots.

… See full entry

Today we’ve started rolling out the 1st beta release of 16.0. If you intend to install this, or at a later date the release version, please read the release notes http://vespucci.io/help/en/16.0.0%20Release%20notes/ before installing.

Due to the preparation for targeting Android 11 there have been some changes in where files are stored that you need to know about.

The first beta release of Vespucci 15.2 has just become available on googles play store, alternatively you can download it from releases in Vespucci repository.

While it would always be nice if as many users as possible could test this as fast as possible, this year due to events outside of our control (bintray demise, mapillary sunsetting its current API at extremely short notice, not to mention the usual multiple man weeks of work due to googles requirements), we need to get this release out of the way asap, any help with that would be appreciated.

Dedicated address mode and address handling improvements

When we introduced the “simple mode” in Vespucci 12 in 2019 there was no way to directly add address nodes without switching to the old element creation mode via long press (you could however add a node and then predict the address in the property editor). To make this a bit easier 15.2 introduces a dedicated “Address” mode that switches the “plus”-button to allow adding

• an address node (including automatically adding an entrance tag if the node is on a building outline)
• an address interpolation
• a map note

Further improvements include support for addr:block, addr:conscriptionnumber and addr:block_number in the address tags preference, and the use of information from geocontext project for region specific default address tags.

Support mode specific launcher shortcuts

After adding a short cut for the “help” system in 15.1, we’ve added further short cuts for specific modes: Indoor, Tag-only and Address.

In launchers that support the functionality short cuts can be accessed by a long press on the app icon, and dragged to create short cut links for direct access.

Address mode demo

Support merging closed ways/polygons

Basic support for merging two closed ways/polygons has been added to the multi-select mode. If their are no common nodes, or if the function dedects inner rings this will create a multi-polygon, otherwise a single closed way.

… See full entry

I’ve just updated the contributor statistics for the whole year 2020.

As maps.me has hit the news again, this time in a more negative way, I thought it might interesting to give the influence of the app on new contributors another look.

… See full entry

Vespucci 15.1 has been available for a couple of days now from googles play store marking the third larger release (14.1, 15.0 and 15.1) this year and our traditional release for the holiday season.

Details on what has changed can be found further down, but first let me point out a few important points from the past year and things to be aware about that will be happening in 2021.

The OSMF and iD

2020 marked the OSMF becoming a participant in the OSM editor market place, you might say what does that concern me? Well, even if not all the developers are competing for funding (like iD, Rapid, StreetComplete and many others), all are competing for relevance. Or put differently: mapper market share. How exactly this will play out going forward isn’t clear yet, but it already has consequences in development, for example I’ve put all tablet related improvements on hold. The required effort just doesn’t makes sense with the OSMF expanding iDs use outside of the classic desktop on to tablets..

More https/SSL woes

A couple of weeks back Let’s Encrypt dropped a bomb. Let’s Encrypt is by far the most popular provider of free encryption keys for https use, and are widely used throughout open source and data, and the OpenStreetMap ecosystems. For example Let’s Encrypt certificates are used on all OSM infrastructure that has so fully migrated to https that use without is no longer possible.

Android 7.1 as the oldest version that will no longer work with Let’s Encyrpt is fairly young, and lots of users are “still”* using it. An estimated 33% of the 2.5 billion Android devices will be effected. So we can’t simply ignore this happening.

… See full entry

I pointed out some of my concerns with the change to the OSMF articles of association and changes to who is to become eligible to a member on the the OSMF mailing list a couple of weeks back, see https://lists.openstreetmap.org/pipermail/osmf-talk/2020-October/007231.html

Those comments were just on the general concepts that were being proposed not the actual text, which we have now received as part of the formal general meeting announcement.

Unluckily my concerns not only persist, they have increased with respect to the proposed change to the OSMFs articles that will allow the board to create committees that include non-board members.

Currently a large part of OSMFs business is carried out by working groups, in particular the OWG running the infrastructure, the CWG communicating on behalf of the OSMF and the LWG handling licence and other intellectual property matters.

The working groups go back to a OSMF board meeting in November 2008, https://docs.google.com/document/d/1OcoOqas1NnOLyhiyFZk12K_BpjhWeXvVU7Nl4MDSGeU/edit?usp=sharing and it is clear that these were established to have topic specific groups in which the board could work on OSMF business with additional participants without having to have the full board present. With other words, working groups have always been board committees with additional members, that this practice wasn’t quite inline with the foundations articles was and has been just a bit of fuzziness among a lot of other issues that the OSMF has had.

… See full entry

After a large number of delays, mainly to due to late breaking issues with undocumented behaviour changes and other issues due to the androidx migration, I’ve tagged the 15.0.0 release. Installable APKs should be available from google play and the Vespucci github repository in the next couple of days (not from f-droid which is lagging a quarter of a year currently and is unlikely to fix its build problems any time soon).

More information on the future of the app will be announced later.

End of support for pre Android 4.0 devices

Version 15 no longer supports devices with Android versions older than 4.0 / API 14. We’ve now migrated the Vespucci code base to use the “androidx” libraries to provide backwards compatibility over multiple Android versions and there is no feasible way to continue to use the old support libraries that would be necessary for older Android versions.

Import note: even though the app will work on Android 4.0 devices, devices with Android version prior to 4.1 do not have TLS 1.2 support that is required for accessing the OpenStreetMap API since August 2nd 2020. If you are experiencing authorization issues on Android versions between 4.1 and 4.4 see Can’t (re-)authenticate - TLS 1.0 / 1.1 issues.

Support for array like semantics for multi-selects

In some circumstances, tags with multiple objects have array-like semantics (that is a fixed number of elements) contrary to being variable length lists. A typical example are lane related tags, which contain the number of elements in the lanes (or lanes:forward and lanes:backward) tag. There is no practical difference for tags that need to have a value for every element, but in the case of tags that have free form, potentially empty values, for example destination:lanes, there was previously no way of indicating this in the preset.

… See full entry

Those that follow the Vespucci twitter account @vespucci_editor will already know, on Sunday the main had TLS 1. and 1.1 support OSM API turned off. This is of a bit wider relevance as older Android devices do not support TLS 1.2 out of the box or at all, and obviously this may effect other apps as well and even such simple things as retrieving map tiles may no longer work when tile caches are updated.

What is older in this context? Well there is no support in any form for devices older than 4.1, introduced in 2012 and no out of the box support, before 4.4.something (likely 4.4.3). 4.4 was released just 6 years ago, so is not that old. Vespucci will enable TLS 1.2 on devices in the range 4.1 - 4.4.? and will work just fine with the OSM API, with two caveats:

• it is possible that there is no support for TLS 1.2 even though there should be, sorry nothing we can do about that.
• while everything for which the network connections are directly initiated by the app should work, components that open their own connections may fail.

Off the top of my head there are only two situations in which the later applies, one not so important is opening issues on github, the other more relevant is authorizing the app via OAuth to use the OSM API. If that happens on your device you should follow the instructions on the Vespucci documentation site and switch to using login and password. Yes, the irony of this is not lost on me.

The deployment of upgraded operating system versions without TLS 1.0 and 1.1 can be followed on Dropping TLSv1 and TLSv1.1 support. If you have any questions or issues not covered above please don’t hesitate to ask.

Seems as if I never wrote a diary entry on 14.1 that has been available for a couple of months now. The release notes can be found as always on the Vespucci documentation site.

Obviously version 15 is quite far along at this point and major parts of the work have already been completed. In particular the migration to the AndroidX libraries and re-targeting for Android API 29 have been finished and, as a consequence, support for Android versions before 4.0 has now been completely dropped.

I’m not planning on adding major new functionality in this release, after major internal changes a shake-down period tends to be required due to device specific issues that are only exposed when the app gets actual use. Changing too much just compounds that. There are however a number of small changes (for example indicating the number of pending changes for upload on the main menu bar) and performance improvements, that will make the user experience more pleasant.

Work on the release can be followed in the Vespucci github repository.

… See full entry

While keeping a quarterly minor or major release schedule is a bit of an uphill battle, I believe the work on the next major version of Vespucci has progressed far enough that the first beta will be made available later this month (snapshot development builds are available now).

The fall releases always tend to suffer from porting to the next mandatory Android API/SDK version and this one is not different. I assume that this typically consumes a man-month of development work, this time around it doesn’t seem to be quite so bad however there are still a few issues I haven’t resolved that need to be fixed before release.

The bad news is that this will be the last release that will support an Android 2.3-3.0 compatible build. The reason is that to be able to continue to follow googles minimum target API/SDK rules we will have to switch to a new packaging of the Android libraries that provide backward support for older versions of Android (this implies that the estimated one man month for following google changes for next year is very optimistic). As all the package names have changed it is simply not feasible to use an old support library as we have done up to now for the legacy build with minimal addition version specific code. Undoubtedly it is a side effect of the packaging change that google is quite happy with.

My intent is to revive the legacy build for 4.0-4.3 devices when google drops support for them, which is likely not far away.

… See full entry

This is a minor release mainly containing UI improvements. You can install the beta from the google play store (after enabling the “beta” releases setting), or obtain it from the Vespucci repository.

Updated opening hours editor

The opening hours editor (a separate project found here) has received a number of updates, including

• tapping the time bars will start a large time picker, this was previously only available via the menu.
• ability to load and save templates to on device files.
• support for region and object type specific templates.

Use of SAF file picker for Android KitKat and later

Previously, to give a consistent look and feel over as many different Android version as possible, we were using a third party file picker/selector for those operations that require a file to be selected for input or output. Unluckily in recent Android versions removable “SD card” storage has become unavailable to the third party library which is rather annoying as this is the location where you typically want to save larger files.

To work around this we are now using the system SAF (Storage Access Framework) file picker for Android KitKat (19) and later, in older Android version the third party file picker should work fine. Using the SAF file picker does have the additional advantage that cloud-based storage location are usable for operations that don’t need direct file access, so you can save and load files from your favourite cloud storage provider if you want to.

Download offline data directly with automatic configuration

… See full entry

If you remember in Going completely offline with Vespucci last December I gave some insight in to the improved offline support in Vespucci 13.0. Now that version 13 is available, I need to loose a couple of words on how to generate the files.

The original mapsplit application was created back in 2011 by well known OSMer Peter Barth, at the time it was used to prepare input for OSM2World. The slight rework that I started late last year adds support for producing output in MBTile format (instead of copying a couple of 100’000 or so files to your mobile :-)), higher and variable maximum zoom level for the tiles and an optimisation pass.

… See full entry

Some of you may have heard of the upset around the introduction of “scoped storage” in the upcoming Android Q (10) release, see for example androidcentral

So why should you care about this change, after all it promises to enhance users privacy, and it isn’t exactly new that the goog doesn’t really care about burning developers time?*

Well there is a special angle to this new feature for “geospatial” apps, it essentially inhibits direct file access and assumes content can be accessed as a stream of bytes. That is all fine and dandy if you are writing something that just access pictures, videos and other media files, but it breaks down as soon as we are using larger structured files.

Classic example a MBTiles format file containing images or vector tiles or anything else, or a geopackage file and so on. Now there are workarounds of sorts, but they essentially all require copying around potentially multiple GBs of data, potentially duplicating it on device and are not really practical or user friendly.

A further twist is that a user will typically want to persist larger downloads over app de- and reinstalls, data in app private directories doesn’t survive this. For example vespucci creates a separate directory exactly to circumvent this issue.

See https://commonsware.com/blog/2019/03/26/death-external-storage-can-haz-file.html for some more information and https://issuetracker.google.com/issues/128591846#comment50 is a comment from a well known Mapbox developer (in case you are wondering why the later doesn’t turn up when googling, yes, the goog doesn’t actually index its own issue tracker).

Now, after protests, google has essentially pushed back enforcement of this by a year to Android R, but that is just kicking the can down the road a bit.

*In the case of paid devs, well they are paid, the others are simply expected to spend a couple of $1000 per year equivalent amount of time to accommodate whatever google has decided to break this time around,