1 Commits (8976709f24dc9baa603d030fa23484382f7ba767)

Author	SHA1	Message	Date
Callum Styan	4d3b15ae89	Implement probabilistic topk via count min sketch and min heap (#9882) ... This PR currently implements the following structures: - **Count Min Sketch**: http://dimacs.rutgers.edu/~graham/pubs/papers/cmencyc.pdf - **sketch-bf**: https://ietresearch.onlinelibrary.wiley.com/doi/epdf/10.1049/ell2.12482 improvements to the heap usage for Count Min Sketch - HeavyKeeper: https://www.usenix.org/system/files/conference/atc18/atc18-gong.pdf I also read and evaluated the following: - topkapi: https://www.cs.rice.edu/~as143/Papers/topkapi.pdf - space saving algorithm: https://www.cse.ust.hk/~raywong/comp5331/References/EfficientComputationOfFrequentAndTop-kElementsInDataStreams.pdf **Description**: As part of our work to ensure every query type is parallelizable we started exploring probabilistic data structures as a way to reduce memory consumption and allow merging of partial results after sharding of queries. In this PR I have implemented a few sketch structures as part of investigating structures that are applicable to topk queries. Unfortunately while many papers made claims of improved performance over the original count min sketch paper, topkapi and heavy keeper either didn't behave as expected or were not mergeable while keeping the same degree of accuracy. Count min sketch when using conservative updates (which vastly reduces error over counting, especially for low flows), also can't keep the same degree of accuracy, but it is still more accurate after merging than heavy keeper. I've also applied some of the principles outlined in the sketchbf paper related to reducing the # of heap operations required when observing events into the sketch, which sped up processing in my benchmark of 100k unique streams each with 1-2k randomized # of occurrences by nearly 3x. --------- Signed-off-by: Callum Styan <callumstyan@gmail.com> Co-authored-by: Karsten Jeschkies <karsten.jeschkies@grafana.com>	3 years ago