mirror of https://github.com/grafana/loki
fix(loki-canary): Send to Loki after updating `totalEntries`. (#7211)
**What this PR does / why we need it**: Fixes: #7142 (take a look for more details about the problem there) After trying few things to avoid this race and discussin it with Ed, This small changes seems to be more accurate and seems to fix the issue. This is unblocking rolling out new canary in our internal cells Update: For some reason I thought the fix didn't fix the original issue when I tested it with internal loki dev cell, almost a month ago. But after testing it for more than 2hr on same cell. I realized it is working fine (will leave it for a day like this just to confirm) Verified following: ``` {.... container="loki-canary"} |= "websocket missing" ``` and noticed it's happening no longer after the fix. Also verified the metrics ``` sum(increase(loki_canary_websocket_missing_entries_total{...}[$__range])) ``` It dropped to zero.   Update 2: After running it for whole day, looks there was two log entries missed on the websocket. My guess is since websocket is long-live connection and something can interrupt the connection loosing this message.  I also quickly checked how often this happens on other bigger envs (say ops) and looks like it's not that uncommon.  **Which issue(s) this PR fixes**: Fixes https://github.com/grafana/loki/issues/7142 **Special notes for your reviewer**: Tested it on internal loki dev cells **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`pull/7428/head
parent
2881c52da4
commit
ceb09efdf8
Loading…
Reference in new issue