Prune / Clean points to save space

[MrColumbo]

The space used by dawarich increased quite a lot recently and I was wondering if there is a way to automatically remove points where they do not provide any value. For example when I spend 2 hours in a restaurant I like to keep the point when I got there and the point when I left. Depending on the settings on the client dawarich might not even receive any points when I do not move around but it might be nice to be able to bulk delete points for example when there are several points with the same coordinates but different time stamps. In such cases I like to keep the first point and the last point with the same (or very close distance) coordinates.

[Freika]

What takes space is reverse geocoding data being gathered for each of your points (assuming you have it enabled). Points themselves take up quite a little space. Your solution might be to disable reverse geocoding if you're trying to save some space. For now, I have no plans to introduce automatic points pruning.

[MrColumbo]

thanks for the feedback. In my case it was a json log file which kept growing a lot due to of error messages

[Freika]

Have a look at the docker compose file in the repo, there are options to limit amount of logs via log rotation Sent from [Proton Mail](https://proton.me/mail/home) for iOS

…

On Sun, Oct 20, 2024 at 10:32, MrColumbo ***@***.***(mailto:On Sun, Oct 20, 2024 at 10:32, MrColumbo <<a href=)> wrote: thanks for the feedback. In my case it was a json log file which kept growing a lot due to of error messages — Reply to this email directly, [view it on GitHub](#340 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AA4QYDUMS34VKZWONEITUATZ4NTCPAVCNFSM6AAAAABQF2A6B2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAOJZGQ4TQMI). You are receiving this because you commented.Message ID: ***@***.***>

[willemsFEB]

I know this is an ancient discussion, but for anyone arriving at this page who want to do something similar, you can do this yourself with a script.

I made a script that prunes points within my areas (as this is usually where I spend most time). It's a basic greedy algorithm that removes points as long as the point before and after are within a 1 hour period. It runs on all points from yesterday 23h until today 23h, and then I run a cronjob that runs this daily at 23h30 (so before the reverse geocoding job runs at midnight).

user = User.find_by(email: "your@domain.com")

# Define time range: yesterday 23:00 to today 23:00
start_time = Time.current.yesterday.change(hour: 23, min: 0, sec: 0)
end_time = Time.current.change(hour: 23, min: 0, sec: 0)

puts "Processing all areas from #{start_time} to #{end_time}"
puts "=" * 50

# Iterate through all user's areas
user.areas.find_each do |area|
  puts "\nArea: #{area.name} (center: #{area.latitude}, #{area.longitude}, radius: #{area.radius}m)"
  
  # Get points in this area within the time window
  points = user.points.where(
    timestamp: start_time..end_time
  ).where(
    "ST_DWithin(lonlat, :center, :radius)",
    center: "POINT(#{area.longitude} #{area.latitude})",
    radius: area.radius
  ).order(:timestamp)
  
  if points.empty?
    puts "  No points found in this area."
    next
  end
  
  # Prune points to 1-hour intervals
  sorted = points.to_a
  kept = [sorted.first]
  
  sorted.each_cons(2) do |prev, curr|
    kept << prev if curr.timestamp - kept.last.timestamp > 1.hour
  end
  kept << sorted.last unless kept.include?(sorted.last)
  
  to_delete = sorted - kept
  to_delete.each(&:destroy)
  
  max_gap = kept.each_cons(2).map { |a,b| b.timestamp - a.timestamp }.max || 0
  
  puts "  Total: #{sorted.count}, Kept: #{kept.count}, Removed: #{to_delete.count}"
end

puts "\n" + "=" * 50
puts "Done processing all areas."

you then mount this script to the dawarich_app image, (eg at /scripts/prune_data.rb), and then run a cronjob like:

30 23 * * * /usr/bin/docker exec -i dawarich_app bin/rails runner /scripts/prune_data.rb >> ~/logs/dawarich_prune.log 2>&1

[Freika]

Returning to an old topic, in the next releases there will be db changes and also instructions on how to offload raw data stored in the db to external archives