Street-level imagery is distributed unevenly, which complicates systematic analysis based on the data. Some regions contain a few isolated sequences or no imagery at all. In a busy city center, the same boulevard has been driven, cycled, and walked dozens of times by different people, in different seasons, with various cameras taking photos every few steps.
Empty areas need people with cameras on the ground, as described in our post on collecting street-level imagery in Mozambique. Crowded areas in turn need a good way to choose among the images that already exist.
The problem of plentiness
Open platforms such as Mapillary, Panoramax and KartaView, together with the many volunteers who drive, cycle and walk with cameras, have made street-level imagery freely available to anyone who wants to work with it. Our work also stands on the shoulders of these giants.
This abundance also brings a new kind of challenge, and it becomes visible when the imagery is fed into a data processing pipeline built around resource-hungry deep learning models or vision-language models.
Images taken by a camera on a car or a bike are a few meters apart. That is how sequences work across all platforms, and it is what makes virtual walks so smooth. For analysis, it means that a single street is represented by a large collection of locally near-identical images.
Each image also carries only a little context. We know where it was taken, but rarely what it shows. As with any outdoor photography, not every frame suits every purpose. A wet lens, a night drive, blur, or a passer-by’s face can all make an image unusable.
A vision model can eventually sort through all of this, but it takes time and computing power. Setting aside what is not needed before the model ever sees it is the cheaper option. That’s how the HeiSmile was born.
How does it work?
HeiSlime is a Python toolbox that selects street-level imagery before it enters a computer vision pipeline. The geographic context orchestrates the process before the pixels do, and configurable filters discard redundant, unsuitable, or irrelevant photos along the way. The result is a smaller and better targeted image set, which saves processing time.
The toolbox
Two kinds of data sources feed the pipeline. Street-level imagery providers (currently Mapillary and Panoramax) supply the photos, and an OpenStreetMap provider built on HeiGIT’s ohsome API supplies the map context, either from the latest data or from any historical snapshot. Adding a new imagery provider takes only two methods: one that lists images in an area, and one that downloads a single image.
On top of those sources sit three families of filters:
- Distribution filters for less repetition
- OSM filters for what is around the camera, and what it can see.
- SLI filters: looking at the pixels
Here is what is available today:
How to use it
A full pipeline takes very little code, because users describe what you want rather than how to obtain it. Every filter does one job and passes its result to the next, so users can mix, reorder and reuse filters like building blocks. Filters are also validated and serializable, which means that a pipeline can be saved, shared, and rerun months later with the same result. That matters for reproducible research.
Under the hood, images are cached on disk, so nothing is downloaded twice, and they are processed in batches on the GPU. The work also splits naturally into map tiles, ready to be spread across cloud machines using Dask when a laptop is no longer enough.
pipeline = Pipeline(
sli_providers=[SLIProviderSpec(
provider=MapillaryProvider(token),
params=MapillaryParams(zoom=14, image_type="flat"))],
osm_provider=OhsomeProvider(token),
)
image_paths = pipeline.execute(
aoi=box(coords),
distribution_filters=[SpatialThinning(distance=10)],
osm_filters=[
ProximityFilter(distance=5, query="footway=sidewalk"),
ExcludeFilter(query="tunnel=*", distance=10)],
sli_filters=[ChainedImageFilter(filters=[
SceneType(keep=["urban street"]),
ContainsObject(labels=["a tree", "a bench"])])],
materializer=ImageMaterializer(output_dir=Path('out/images'), extension='jpg'),
)
This pipeline takes Mapillary images in a given area of interest, keep one every 10 metres, only near sidewalks and away from tunnels, then keep the urban street scenes that show trees and benches.
HeiSlime materializes its results in an open, flexible way, so it is easy to combine with existing tools for walkability and cycling research, road surface mapping, humanitarian work, crowdsourced validation, or routing. Users can write the result to disk, or build their own materialisation step to connect it with libraries such as ZenSVI or our HeiMeterStick.
Explore the project: gitlab.heigit.org/giscience/heimeterstick/heislime
