I've noticed that the following query gets slower over time, obviously accumulating ways over time. This is most obvious in Way_Geometry_Store, where an increasing numbers of ways is being prepared.
When running the query for a single FIPS value (e.g. area["nist:fips_code"="16011"];), only a single index and two ways need to be prepared in Way_Geometry_Store, supporting the idea that foreach doesn't clean up some data here.
I believe this is caused by a missing clear() in Area_Constraint::get_ranges which can be fixed by mmd-osm@debc357
This brings the query runtime down from >2h (query below times out) to 50s.
Testing based on commit 021dfdb
Query
[timeout:7200]
[out:csv("nist:fips_code", fips, name, total)];
// All counties with FIPS 6-4 county codes
area["nist:fips_code"~"^[0-9]{5}$"];
// Count the pharmacies in each county
foreach->.county(
.county out tags;
// Collect all matching features in the current county
way["amenity"="fuel"](area.county);
// Sum nodes, ways, and relations together and group by county
make count fips = county.set(t["nist:fips_code"]),
name = county.set(t["name"]),
total = count(nodes) + count(ways) + count(relations);
// out;
)Logging
Adding small instrumentation to show the number of indices and ways whenn calling the Way_Geometry_Store constructor:
diff --git a/src/overpass_api/data/way_geometry_store.cc b/src/overpass_api/data/way_geometry_store.cc
index 75d6de37..b8f1a499 100644
--- a/src/overpass_api/data/way_geometry_store.cc
+++ b/src/overpass_api/data/way_geometry_store.cc
@@ -92,6 +92,9 @@ std::map< Uint32_Index, std::vector< Node_Skeleton > > small_way_members
Way_Geometry_Store::Way_Geometry_Store
(const std::map< Uint31_Index, std::vector< Way_Skeleton > >& ways, const Statement& query, Resource_Manager& rman)
{
+long total = 0;
+for (const auto &r: ways) total+=r.second.size();
+std::cout << "Constructor called with #ways: " << total << " in " << ways.size() <<" indices\n";
// Retrieve all nodes referred by the ways.
std::map< Uint32_Index, std::vector< Node_Skeleton > > way_members_ = small_way_members(&query, rman, ways);Output
Number of ways and indices keeps increasing over time:
nist:fips_code fips name total
02016 Aleutians West Census Area
Constructor called with #ways: 2 in 2 indices
Constructor called with #ways: 0 in 0 indices
60010 Eastern District
Constructor called with #ways: 4 in 3 indices
Constructor called with #ways: 0 in 0 indices
60050 Western District
Constructor called with #ways: 4 in 3 indices
Constructor called with #ways: 0 in 0 indices
60020 Manu'a District
Constructor called with #ways: 4 in 3 indices
Constructor called with #ways: 0 in 0 indices
60030 Rose Atoll
Constructor called with #ways: 4 in 3 indices
Constructor called with #ways: 0 in 0 indices
02185 North Slope
Constructor called with #ways: 6 in 5 indices
Constructor called with #ways: 0 in 0 indices
15003 Honolulu County
Constructor called with #ways: 51 in 41 indices
Constructor called with #ways: 0 in 0 indices
02013 Aleutians East
Constructor called with #ways: 56 in 44 indices
Constructor called with #ways: 0 in 0 indices
02050 Bethel
[...]
Constructor called with #ways: 12689 in 10100 indices
Constructor called with #ways: 0 in 0 indices
47105 Loudon County
Constructor called with #ways: 12689 in 10100 indices
Constructor called with #ways: 0 in 0 indices
47123 Monroe County
Constructor called with #ways: 12689 in 10100 indices
Constructor called with #ways: 0 in 0 indices
37039 Cherokee County
Constructor called with #ways: 12692 in 10103 indices
Constructor called with #ways: 0 in 0 indices
37043 Clay County
Constructor called with #ways: 12692 in 10103 indices
Constructor called with #ways: 0 in 0 indices
37075 Graham County
Constructor called with #ways: 12692 in 10103 indices
Constructor called with #ways: 0 in 0 indices
37173 Swain County
Constructor called with #ways: 12693 in 10104 indices
Constructor called with #ways: 0 in 0 indices
47133 Overton County
Constructor called with #ways: 12693 in 10104 indices
Constructor called with #ways: 0
[...]
I've noticed that the following query gets slower over time, obviously accumulating ways over time. This is most obvious in
Way_Geometry_Store, where an increasing numbers of ways is being prepared.When running the query for a single FIPS value (e.g.
area["nist:fips_code"="16011"];), only a single index and two ways need to be prepared inWay_Geometry_Store, supporting the idea that foreach doesn't clean up some data here.I believe this is caused by a missing clear() in Area_Constraint::get_ranges which can be fixed by mmd-osm@debc357
This brings the query runtime down from >2h (query below times out) to 50s.
Testing based on commit 021dfdb
Query
Logging
Adding small instrumentation to show the number of indices and ways whenn calling the Way_Geometry_Store constructor:
Output
Number of ways and indices keeps increasing over time: