apitech/loki - loki - Gitea: Git with a cup of tea

Commit Graph

Author	SHA1	Message	Date
Ivan Kalita	e4ec8441d3	feat(metastore): use arrow for scanning and blooms (#20234 ) Changing metastore implementation to work with arrow.RecordBatch without intermediate materialization to structs. metastore.Sections now uses new metastore.GetIndexes, metastore.IndexSectionsReader and metastore.CollectSections. We'd like to run metastore queries in a distributed manner using engine v2. In order to achieve that we need to split the work that metastore.Sections does internally, schedule it on different workers, and collect the results. The engine achieves that via pipelines that produce arrow.RecordBatch-es. In this PR the metastore.Sections is updated such a way to make it easier to integrate metastore with the engine.	1 month ago

Author

SHA1

Message

Date

Ivan Kalita

e4ec8441d3

feat(metastore): use arrow for scanning and blooms (#20234 )

Changing metastore implementation to work with arrow.RecordBatch without intermediate materialization to structs.
metastore.Sections now uses new metastore.GetIndexes, metastore.IndexSectionsReader and metastore.CollectSections.

We'd like to run metastore queries in a distributed manner using engine v2. In order to achieve that we need to split the work that metastore.Sections does internally, schedule it on different workers, and collect the results. The engine achieves that via pipelines that produce arrow.RecordBatch-es. In this PR the metastore.Sections is updated such a way to make it easier to integrate metastore with the engine.

1 month ago

1 Commits (d9404337367df922bcbf4e2771e182288ea6717f)