Conversation
LogBucket cached its table as mcrit/cache/logbuckets.json whatever max_value and bucket_width it was built for, and loaded that file whenever it existed. The package ships it, so every installation hashed with the 100,000/1 default table and SHINGLER_LOGBUCKETS and SHINGLER_LOGBUCKET_RANGE had no effect (familiary#202, familiary#215). The shipped file is now logbuckets_100000_1.json, byte for byte the same table, so default deployments hash exactly as before. Any other table is built in memory once per process and never written, since the package directory may be read-only and several workers start at once. Building it takes 0.2 s at 100,000 entries instead of 1.3 s, with a set for the builder's membership test; its output is unchanged, which a test pins against the shipped file. The builder cannot give every value its full range once the width reaches 5, or when max_value is too small for the width, which an installed package never reached. Such a table is now refused with ValueError when the shinglers are loaded rather than failing with KeyError in the middle of indexing, as are a max_value below 1, a negative width and non-int values. The release smoke test checks that LogBucket() loads the shipped table instead of only that the file exists.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #202, closes #215.
LogBucketcached its table asmcrit/cache/logbuckets.jsonno matter whichmax_valueandbucket_widthit was built for, and loaded that file whenever it existed. Since the package ships the file, every installation hashed with the 100,000 / 1 default table.SHINGLER_LOGBUCKETSandSHINGLER_LOGBUCKET_RANGEquietly did nothing —LogBucket(1024, 1)handed back 100,000 entries — and when the file was missing, a fresh one got written into the package directory, which can be read-only and is shared by every worker starting at the same time.The shipped file is now
logbuckets_100000_1.json, byte for byte the same table. Any other combination is built in memory once per process and never written anywhere. The builder uses a set for its membership test, so a full 100,000-entry table takes about 0.2 s instead of 1.3 s, and a test pins its output against the shipped file.config.shinglerhash already named the non-default values)Settings the builder can't serve properly (a range of 5 or more, a
max_valuetoo small for the range,max_value < 1, a negative range) now raiseValueErrorwhen the shinglers load, instead of aKeyErrorhalfway through indexing; non-int values raiseTypeError. The release smoke test checks thatLogBucket()really loads the shipped table, not just that the file is there.I checked the default path against a real corpus as well (7,244 samples, read-only): 3,655 functions recomputed from cached SMDA reports come out byte-identical to the stored MinHashes and to
main. Unit and mongo suites,ruffandtyare green.