How to import an existing list of initiatives (POIs) into OpenStreetMap

openstreetmap
Tags: #<Tag:0x00007fc0018ae008>

(Michael Maier) #1

Most of our partners already have an existing mapping system. The (currently) only possible way to ‘switch’ to TransforMap as a mapping platform is to switch to OpenStreetMap (OSM), if the dataset is suitable for OSM. Data therefore has to be added to OSM. When adding large datasets in bulk to OSM, we call it an import.

OpenStreetMap has very strict guidelines on imports. But don’t be afraid and read on, when done the right way imports can be a huge benefit for all.

There are prerequisites that must be fulfilled:

  • The owner of the data must hold ALL rights on it and be able to release it as Public Domain (CC-0)
  • → they MUST NOT be geolocated with Google!
  • The import must be guided/carried out by an experienced TransforMapper/OpenStreetMapper.
  • Data must be suitable for OSM (only Points of Interests visible on the ground for a longer time (e.g. no events)
  • Be aware that data in OSM is
  • Worldwide accessible, and analyzeable in bulk, also commercial use explicitely allowed.
  • Subject to change (anyone can edit it) with no guarantee for availability.
  • Will be kept up to date not only by you, but also by 2 million OpenStreetMappers.

To start an import, you can do it yourself if you are an experienced OpenStreetMapper, or contact @species.

How to prepare your data for an import:

  • check the licence of EVERY field, especially coordinates
  • data format: csv, xls or any geo-format (except kml, where we charge extra due to its complexity :wink: )
    • Charset: UTF-8!
    • max length of field: 254 chars.
    • no formatting, no line breaks, HTML, etc allowed!
  • its columns should contain:
    • address splitted in city, postal code, street/hamlet, housenumber columns
    • coordinates (WGS84) (if already available as open data)
    • name of POI
    • own classification (or better OSM tag)

We would appreciate the following additional information columns:

  • contact data (if available, each single entry an own column):
  • webseite email, social network sites
  • tel, fax (int. Format +49 xxxx xxxx-xx)
  • contact name
  • a free text field “description” can be used for additional, unstructured information
  • opening hours (see osm spec here )
  • wheelchair accessibility
  • founding year
  • smoking allowed?

You can help us by preparing you POI with our TransforMap Taxonomy tags:

  • needs: “provides”/“topic”
  • interaction
  • identity
  • OSM “main tag”

Step-by-step How-to for Import

in a spreadsheet software:

  • Check charset, convert to utf-8
  • check and convert tel-nrs to int. format
  • check alignment of columns if using csv
  • create a mapping of import columns to osm key/values
    • convert column heads to keys and cell contents to values
  • check and correct address fields for Country, Postcode, City, Street, House-Nr.
  • delete unused columns

If coordinates are missing, use script (TODO link here) to geolocate via Nominatim (OSM geocoder). For addresses not present in OSM, check open data portals and local ordinance survey. Use of single addresses is often permitted.

(
if you assume that a lot of your points are already present in OSM, it maybe adviseable to do a spatial comparison in QGIS:

Load file in QGIS for further processing and matching to exiting OSM POIs.
Download matching existing OSM-POIs for area (amenity,shop,leisure, … maybe use geofabrik.de shapefile), open in QGIS

create buffer on OSM-objects:

  • vector → geoprocessing → buffer, distance 0.00018
  • vector → analysis → points in polygon: select all new POIS inside buffers
  • rightclick on layer → save selection as “both” (shapefile, or better geojson)
  • rightclick on layer → Attribute Table: Invert Selection
  • rightclick on layer → save selection as “not-yet-in-osm” (shapefile, or better geojson)
  • maybe (if points of your dataset have the same coordinates, use Plugin “Shift Points” with minimal distance (0.00001), save dataset.

)

For import and converting to OSM format, use JOSM:

  • open file to import (“not-yet-in-osm”) in JOSM
  • select all (CTRL-A)
  • add tag “TODO=YES”
  • correct tag keys that where malformed because of shape or geojson conversion
  • and now, for each point:
  • check if coordinates match a feature on aerial and /or osm data - e.g. houses. POIs cannot lie on:
    • in the middle of roads
    • in lakes
    • … find the most reasonable place and move the POI here - EVERY point has to be moved 1-5m, to avoid overlapping
  • if done, remove the TODO=YES tag
  • move on to the next, e.g. search for TODO=YES and pick the first one
  • for points where a location cannot be found, move it to another layer and add tag fixme=yes
    • at first you create a new layer in the “layers” panel
    • C-X, switch to other layer (set the green check to it), C-V
    • switch back to working layer
    • note: do not download any existing OSM data into these layers, use a new one for comparision!
  • when done:
    • save the done file as ready-to-import.osm
    • save the file with the “fixme” POIs. for these POIs, ask the data source about the POIs
      • are the coordinates correct?
      • is the aerial too old? (this can be spottet if a start_date on the POI is supplied)

consult the local(!) OSM mailinglist - (and imports@ if O.K. locally ) for the import, present the ready-to-import.osm, following the import guidelines.

  • if no objections (~1 Week):
    • create a OSM user for the specific import
      • add description text and contact info (email, link to TM proposal site)
      • for bigger imports, add the created import description page to user info
    • add this user to “users currently mapping” on the proposal site
    • open the file to import in JOSM, make a good changeset comment involing links to wiki pages, upload
    • for the “both”-file, use the ~same process, but merge tag data into existing features.

If you want to create a map out of it, use the code on github, where our demo maps are based on.

For sure you will have questions, post them as reply in this topic! Our answers will then used to improve this howto too :smile:


GERMANY / Saxonia / Dresden + Leipzig
FINLAND/commons.fi
How to visualize an existing map under demo.transformap.co
Adding multiple POIs AND creating a selective map
📐 Which tools are available for mapping?
Portugal
Should we use OpenStreetMap as central POI repository?
Portugal
(Adrien Labaeye) #2

Preparing data from http://sharingberlin.de to import.
We have the following question/requirement: is it possible to display one category of filters that is specific to one map under demo.transformap.co?
For the moment sharingberlin.de has organized its POIs under 9 thematic categories that are similar to the needs, but they are themes not needs: e.g. an energy consulting company is ranked under the theme “energy”, but it doesn’t provide “energy”.
So would it be possible, for the map “Sharing Berlin”, to display under “Filters” the category “Theme”?
Under which key should we input those themes? Topic?

Other question: does OSM allows general text description? Is there guidelines (length…)?
cc @Thomas_D @stefan @andreas


Description Texts in OpenStreetMap
(Stefan Sayer) #3

starting point for geolocating via Nominatim, run
$ pip install nominatim
before:


General communities' requirements for TransforMap
(Andreas Arnold) #4

Underlining Adriens question on sharingberlins 9 categories that are mandatory to navigate the map, I’d like to raise awareness for some more practical issues or requirements.

User perspective (people who want to be informed about the Sharing Scene in Berlin):

  • Map must be as easy to navigate as possible! They are no techies and won’t take an effort to understand the taxonomy of TransforMap. That’s to difficult for the few seconds of decision making. Easy-to-digest-categories as we tried to implement are needed, so please make individual filters possible.
  • Maybe it is possible the have a second layer of filters, e.g. for the mode of interaction.
  • A legend is required too.
  • A search function for different project would be nice.
  • When the map is implemented in a frame of another website users can be shown a link to TransforMap, but it would be too disturbing if all other TransforMap maps would be shown as well.

Host perspective (people who have the editing rights of the map):

  • I guess we need to curate the map in order to not have anybody dropping their projects. So far we excluded Uber, and are curious about some consultancy that has once compiled a report on sharing. On the other hand we need to manage the map for the city of Berlin as well and we might need possibilities to moderate.
  • An interface that makes importing/changing data possible. I receive a lot of request to update projects or include new once. The best would be wether it can be done by uploading cvs files or having a database behind.

As Adrien and Jon gave me the impression that sharingberlin is a good practical use case for TransforMap, I hope my wishes for the will become reality. :wink: Thank you guys.


General communities' requirements for TransforMap
(Michael Maier) #5

Hi Andreas,

thanks for your detailed interest, these are exactly the questions I need to improve the map further :slight_smile:

The current maps are “demo” ones to show what is possible. The real life implemented maps will be much simpler, focusing very narrow on one category, e.g. a map only for Solidarity Economy initiatives. The filters will also be less, only the ones that are needed for this type of map. E.g. a map dedicated to mountain biking does probably not need filters for wheelchair users.
And it is not needed to display all our taxonomy filters - only if the map operator wishes so!

Please specify - implementing more filters is not complicated, I need only an exact specification!

This exist for some maps, e.g. the Map of Identities - click “Map Key” on the sidebar on the right.

I haven’t found time to implement the legend for each of the maps, as giving good texts for the different items is also the job of the map operator^^

For a ‘manual’ search you can use the “List of POIs” in the sidebar.

And of course you can use the “search” function (top left, click the magnifying glass). This at least zooms to the POI if found. I have to implement opening the popup, I guess… I’ve created a ticket for this issue.

Of course, the content of the sidebar is open for customizing on on each website implemented - if it is wished to be shown at all.

As this map is currently only fed from OpenStreetMap data, every project can update their data themselves. We’re well aware that editing OSM data is not easy for complete beginners, so we will provide an extensive How-To and a support channel.

As Uber does not operate on fixed places on the ground, they are anyway not welcome in OSM.

Moderation is a big wish from a lot of partners.
As anyone can edit OSM, we cannot moderate here. But we may provide an external database where projects can “whitelist” their and “blacklist” other projects. This may be realized as an “external tag” database for OSM objects. The map would first pull all from OSM, and then filter out items that are not wished.

There exist no automated import tools in OSM, for a good reason.
Each single import has to be authorized by the local OSM community, as imports can severely degrade the quality of the map if done wrongly. A full automation is not possible, because of duplicates. If an item exists in both the source and OSM, the POIs have to be manually merged.

What we can do is to provide a database of all items that have been imported into OSM, to check if an item has been vanished in OSM (deleted because it no longer exists or by error).

I would like to see the sharingberlin data imported as soon as possible :smile:

Thanks for your interest - and let me know if you have further questions!


(Michael Maier) #6

A post was split to a new topic: Description Texts in OpenStreetMap