HeiGIT has been working with deep learning models and street-level imagery for some time, with the goal to contribute to „filling in the world map“ by gaining high-resolution information about regions that are often scarcely mapped. This objective is especially important for humanitarian aid efforts, but also extends to urban planning and civil protection logistics.
While humans are still crucial to coordinate and validate mapping efforts, deep learning approaches can help gain insights in a fast, cost-effective, scalable way. A great information source for this is street-view imagery, which offers information about the world at human eye-level, complementing satellite and drone imagery with a different, highly detailed perspective.
We have already been combining street-level imagery with deep learning methods in a variety of settings, from road surface classification and waste detection to sidewalks width measurement and weather-adaptive routing, as presented in a previous blog post.
HeiGIT´s latest work and advances in street-level imagery analysis now include also:
- an easy way to help volunteers map needed information with a new street-level imagery project type in the MapSwipe app;
- a private Panoramax instance with a dedicated pipeline to automatically analyze crowdsourced street-view imagery.
MapSwipe: Easier Volunteered Geographic Information with Street-Level Imagery
MapSwipe is an open-source app that enables volunteers to map missing geodata, making global mapping efforts more coordinated and efficient.
The latest version of MapSwipe allows not only for satellite images, but also integrates street-level imagery and panoramic, 360 degrees images, which offer much greater detail and significantly expand the mapping possibilities, as announced in this blog post. MapSwipe currently uses Mapillary street-view images, with Panoramax images soon to be supported, too.
With this new feature, volunteers can contribute to identifying new objects with high details, and can also contribute to enriching existing objects with valuable attributes (e.g. road conditions, building use) or to collecting perceptions, e.g. on the greenness or the safety of an area.
The new street-view imagery project type has already been used in a research project aiming at detecting waste in the streets of Dar es Salaam, Tanzania, in collaboration with OpenMap Development Tanzania. Reliably detecting waste on the streets can help optimizing cleaning efforts and reducing health hazards linked to water contamination.
For this purpose, street-view imagery was first manually classified via MapSwipe, with users marking the images showing waste piles on the street. This classified imagery was then used to train a classification-based model for image recognition, which was able to successfully recognize trash bags and waste piles with precision scores between 0.87- 0.98. Even if with some occasional misclassifications, the model allowed to successfully automatize and scale up the waste detection.
Hug-a-Panorama: an Automated Pipeline for Street-View Imagery Analysis in Panoramax
HeiGIT has set up a private Panoramax instance to support collaborative field data collection with selected humanitarian and research partners, enabling partners to contribute and manage imagery within a controlled environment, where access can be restricted to authenticated users. This setup allows to collect high-quality field imagery from areas that are typically missing in global datasets, such as sensitive geotagged imagery from vulnerable regions or infrastructure, while maintaining data governance requirements.
During last year´s creative sprint “Innovation Weeks”, we explored how computer vision and vision-language models can be applied directly to Panoramax imagery at scale. The result was a proof of concept of a crowdsourced street-view imagery analytical pipeline, codename: Hug-a-Panorama.
The Hug-a-Panorama system processes entire image sequences or spatial areas of interest and supports a range of tasks, including image classification, object detection, instance segmentation, and queries for features such as road infrastructure or building elements. It also estimates the geographical coordinates of detected objects, enabling further automatized steps such as road detection, surface classification, and attribute extraction. All outputs are then written back into Panoramax as structured annotations, ready to use and fully compatible with OpenStreetMap.
In this blog article, you can find a more detailed overview of the architecture of Hug-a-Panorama and its use in Madagascar within the AILAS project for weather-adaptive, AI-supported routing system. Hug-a-Panorama has now been open-sourced and the code can be found on GitLab.
Next Steps: More Data Products for Humanitarian Logistics
As the demand for data from under-represented regions grows, we keep working on refining and expanding our pipelines and data products derived from street-level imagery.
We are currently improving our models by integrating newer architectures such as EfficientNet and by expanding the training dataset with new street-level imagery. At the same time, we are building on one of our most requested data products, a deep-learning based, global paved/unpaved road dataset, to include more complex road surface classification at a global scale, such as recognizing different types of dirt roads.
Driven by the need of humanitarian and civil protection organizations, we are also scaling up model performance and data production for the Humanitarian Data Exchange (HDX), with datasets on building material, roof material and power poles. We look forward to releasing new data products to support humanitarian logistics and local services in regions that need more up-to-date, comprehensive maps.
