9e68703841
This commit changes the logic for managing the expiration of objects in the database. Before: There was a server-wide hashmap that stored the collection key, id, and expiration timestamp for all objects that had a TTL. The hashmap was occasionally probed at 20 random positions, looking for objects that have expired. Those expired objects were immediately deleted, and if there was 5 or more objects deleted, then the probe happened again, with no delay. If the number of objects was less than 5 then the there was a 1/10th of a second delay before the next probe. Now: Rather than a server-wide hashmap, each collection has its own ordered priority queue that stores objects with TTLs. Rather than probing, there is a background routine that executes every 1/10th of a second, which pops the expired objects from the collection queues, and deletes them. The collection/queue method is a more stable approach than the hashmap/probing method. With probing, we can run into major cache misses for some cases where there is wide TTL duration, such as in the hours or days. This may cause the system to occasionally fall behind, leaving should-be expired objects in memory. Using a queue, there is no cache misses, all objects that should be expired will be right away, regardless of the TTL durations. Fixes #616
38 lines
1.2 KiB
Modula-2
38 lines
1.2 KiB
Modula-2
module github.com/tidwall/tile38
|
|
|
|
go 1.16
|
|
|
|
require (
|
|
cloud.google.com/go/pubsub v1.3.1
|
|
github.com/Shopify/sarama v1.27.2
|
|
github.com/aws/aws-sdk-go v1.37.3
|
|
github.com/eclipse/paho.mqtt.golang v1.3.1
|
|
github.com/golang/protobuf v1.4.3
|
|
github.com/gomodule/redigo v1.8.3
|
|
github.com/mmcloughlin/geohash v0.10.0
|
|
github.com/nats-io/nats-server/v2 v2.1.9 // indirect
|
|
github.com/nats-io/nats.go v1.10.0
|
|
github.com/peterh/liner v1.2.1
|
|
github.com/prometheus/client_golang v1.10.0
|
|
github.com/streadway/amqp v1.0.0
|
|
github.com/tidwall/btree v0.5.0
|
|
github.com/tidwall/buntdb v1.2.3
|
|
github.com/tidwall/geoindex v1.4.3
|
|
github.com/tidwall/geojson v1.2.7
|
|
github.com/tidwall/gjson v1.7.4
|
|
github.com/tidwall/match v1.0.3
|
|
github.com/tidwall/pretty v1.1.0
|
|
github.com/tidwall/redbench v0.1.0
|
|
github.com/tidwall/redcon v1.4.1
|
|
github.com/tidwall/resp v0.1.0
|
|
github.com/tidwall/rtree v1.2.7
|
|
github.com/tidwall/sjson v1.1.6
|
|
github.com/xdg/scram v1.0.3
|
|
github.com/yuin/gopher-lua v0.0.0-20200816102855-ee81675732da
|
|
golang.org/x/net v0.0.0-20210119194325-5f4716e94777
|
|
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1
|
|
google.golang.org/api v0.20.0
|
|
google.golang.org/grpc v1.35.0
|
|
layeh.com/gopher-json v0.0.0-20201124131017-552bb3c4c3bf
|
|
)
|