Like Prometheus, but for logs.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
loki/integration/util/merger.go

55 lines
1.3 KiB

package util
import (
"fmt"
"dario.cat/mergo"
"gopkg.in/yaml.v2"
)
// YAMLMerger takes a set of given YAML fragments and merges them into a single YAML document.
// The order in which these fragments is supplied is maintained, so subsequent fragments will override preceding ones.
type YAMLMerger struct {
fragments [][]byte
}
func NewYAMLMerger() *YAMLMerger {
return &YAMLMerger{}
}
func (m *YAMLMerger) AddFragment(fragment []byte) {
m.fragments = append(m.fragments, fragment)
}
func (m *YAMLMerger) Merge() ([]byte, error) {
merged := make(map[interface{}]interface{})
for _, fragment := range m.fragments {
fragmentMap, err := yamlToMap(fragment)
if err != nil {
return nil, fmt.Errorf("failed to unmarshal given fragment %q to map: %w", fragment, err)
}
index-shipper: add support for multiple stores (#7754) Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> **What this PR does / why we need it**: Currently loki initializes a single instance of index-shipper to [handle all the table ranges](https://github.com/grafana/loki/blob/ff7b46297345b215fbf49c2cd4c364d125b6290b/pkg/storage/factory.go#L188) (from across periods) for a given index type `boltdb-shipper, tsdb`. Since index-shipper only has the object client handle to the store defined by `shared_store_type`, it limits the index uploads to a single store. Setting `shared_store_type` to a different store at a later point in time would mean losing access to the indexes stored in the previously configured store. With this PR, we initialize a separate index-shipper & table manager for each period if `shared_store_type` is not explicity configured. This offers the flexibility to store index in multiple stores (across providers). **Note**: - usage of `shared_store_type` in this commit text refers to one of these config options depending on the index in use: `-boltdb.shipper.shared-store`, `-tsdb.shipper.shared-store` - `shared_store_type` used to default to the `object_store` from the latest `period_config` if not explicitly configured. This PR removes these defaults in favor of supporting index uploads to multiple stores. **Which issue(s) this PR fixes**: Fixes #7276 **Special notes for your reviewer**: All the instances of downloads table manager operate on the same cacheDir. But it shouldn't be a problem as the tableRanges do not overlap across periods. **Checklist** - [X] Reviewed the `CONTRIBUTING.md` guide - [ ] Documentation added - [X] Tests updated - [x] `CHANGELOG.md` updated - [x] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` --------- Signed-off-by: Ashwanth Goli <iamashwanth@gmail.com> Co-authored-by: J Stickler <julie.stickler@grafana.com>
2 years ago
if err = mergo.Merge(&merged, fragmentMap, mergo.WithOverride, mergo.WithTypeCheck, mergo.WithAppendSlice); err != nil {
return nil, fmt.Errorf("failed to merge fragment %q with base: %w", fragment, err)
}
}
mergedYAML, err := yaml.Marshal(merged)
if err != nil {
return nil, err
}
return mergedYAML, nil
}
func yamlToMap(fragment []byte) (interface{}, error) {
var fragmentMap map[interface{}]interface{}
err := yaml.Unmarshal(fragment, &fragmentMap)
if err != nil {
return nil, err
}
return fragmentMap, nil
}