Evenly spread memcached across nodes (#7183)

<!--  Thanks for sending a pull request!  Before submitting:

1. Read our CONTRIBUTING.md guide
2. Name your PR as `<Feature Area>: Describe your change`.
a. Do not end the title with punctuation. It will be added in the
changelog.
b. Start with an imperative verb. Example: Fix the latency between
System A and System B.
  c. Use sentence case, not title case.
d. Use a complete phrase or sentence. The PR title will appear in a
changelog, so help other people understand what your change will be.
3. Rebase your PR if it gets out of sync with main
-->

**What this PR does / why we need it**:
This PR enables the
[TopologySpreadConstraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/)
for Memcached chunks cache introduced at
https://github.com/grafana/jsonnet-libs/pull/832.
Now the chunks cache will be evenly spread across nodes but more than
one cache can run on the same node.



**Which issue(s) this PR fixes**:
Fixes #<issue number>

**Special notes for your reviewer**:

<!--
Note about CHANGELOG entries, if a change adds:
* an important feature
* fixes an issue present in a previous release, 
* causes a change in operation that would be useful for an operator of
Loki to know
then please add a CHANGELOG entry.

For documentation changes, build changes, simple fixes etc please skip
this step. We are attempting to curate a changelog of the most relevant
and important changes to be easier to ingest by end users of Loki.

Note about the upgrade guide, if this changes:
* default configuration values
* metric names or label names
* changes existing log lines such as the metrics.go query output line
* configuration parameters 
* anything to do with any API
* any other change that would require special attention or extra steps
to upgrade
Please document clearly what changed AND what needs to be done in the
upgrade guide.
-->
**Checklist**
- [ ] Documentation added
- [ ] Tests updated
- [ ] Is this an important fix or new feature? Add an entry in the
`CHANGELOG.md`.
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`

docs/sources/upgrading/_index.md

@@ -49,6 +49,15 @@ We introduced a new cache called `embedded-cache` which is an in-process cache s
 
 Currently `embedded-cache` with `distributed: true` can be enabled only for results cache.
 
+#### Evenly spread Memcached pods for chunks across kubernetes nodes
+
+We now evenly spread memcached_chunks pods across the available kubernetes nodes, but allowing more than one pod to be scheduled into the same node.
+If you want to run at most a single pod per node, set `$.memcached.memcached_chunks.use_topology_spread` to false.
+
+While we attempt to schedule at most 1 memcached_chunks pod per Kubernetes node with the `topology_spread_max_skew: 1` field,
+if no more nodes are available then multiple pods will be scheduled on the same node.
+This can potentially impact your service's reliability so consider tuning these values according to your risk tolerance.
+
 #### Evenly spread distributors across kubernetes nodes
 
 We now evenly spread distributors across the available kubernetes nodes, but allowing more than one distributors to be scheduled into the same node.

production/ksonnet/loki/memcached.libsonnet

@@ -6,6 +6,7 @@ memcached {
     name: 'memcached',
     max_item_size: '2m',
     memory_limit_mb: 4096,
+    use_topology_spread: true,
   },
 
   // Dedicated memcached instance used to temporarily cache index lookups.