2 Commits (b95bfabd422a9b458dbbbde3eda33a95ed768a3a)

Author	SHA1	Message	Date
Christian Haudum	119275aab7	chore(engine): Column naming conventions (#19396) ... In the new engine, we need fully qualified column names, since columns from different sources can have the same name. Right now, the distinction between columns with the same name is implemented using the `Metadata` field on the `arrow.Field`. However, it is quite cumbersome to parse the column type and data type from this generic map. This PR introduces package with naming conventions for columns, defined by name, data type, and column type. So, this information can be encoded into the `Name` field of the `arrow.Field`. The convention is defined as ``` [DATA_TYPE].[COLUMN_TYPE].[COLUMN_NAME] ``` #### Examples: * `utf8.label.service_name` * `timestamp_ns.builtin.timestamp` The column type can easily be converted into a `Scope`, which is defined by an origin and type. The mapping is as follows: ``` ColumnTypeBuiltin -> Scope{Record, Attribute} ColumnTypeMetadata -> Scope{Record, Builtin} ColumnTypeLabel -> Scope{Resource, Attribute} ColumnTypeParsed -> Scope{Generated, Attribute} ColumnTypeGenerated -> Scope{Generated, Builtin} ColumnTypeAmbiguous -> Scope{Unscoped, Attribute} ``` --- Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	3 months ago
Robert Fratto	2e418b13fd	chore(engine): unexport subpackages (#19384) ... This moves packages around to reduce the surface area of the public engine API: * `pkg/engine/planner` moves to `pkg/engine/internal/planner` * `pkg/engine/executor` moves to `pkg/engine/internal/executor` These packages were only used from `pkg/engine` and did not need to be public. We may make them public again in the future if we want to expose subcomponents of the engine. This move means that `pkg/engine/planner/internal/tree` became `pkg/engine/internal/planner/internal/tree`. To reduce the import path, I also moved that package to `pkg/engine/internal/util/tree`. Other than moving files and updating import paths, no code changes are made. Signed-off-by: Robert Fratto <robertfratto@gmail.com>	3 months ago
Robert Fratto	9129d8cdd0	chore(engine): use columnar logs.Reader for DataObjScan (#18577)	5 months ago
Christian Haudum	4305a6aa61	chore(engine): Make literals simple types (#17733) ... Representing data types as simple types (string, int64, float64) makes them more efficient to cast without conversion. Signed-off-by: Christian Haudum <christian.haudum@gmail.com>	6 months ago
Christian Haudum	3956ea1be7	chore(engine): Map line filter expressions to data object predicates (#18102) ... This PR converts line filter expressions into message predicates for the data object scan, instead of returning an error. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Ashwanth <iamashwanth@gmail.com>	7 months ago
Robert Fratto	a237444f0d	refactor(dataobj): invert dependency between dataobj and sections (#17762) ... Originally, the dataobj package was a higher-level API around sections. This design caused it to become a bottleneck: * Implementing any new public behaviour for a section required bubbling it up to the dataobj API for it to be exposed, making it tedious to add new sections or update existing ones. * The `dataobj.Builder` pattern was focused on constructing dataobjs for storing log data, which will cause friction as we build objects around other use cases. This PR builds on top of the foundation laid out by #17704 and #17708, fully inverting the dependency between dataobj and sections: * The `dataobj` package has no knowledge of what sections exist, and can now be used for writing and reading generic sections. Section packages now create higher-level APIs around the abstractions provided by `dataobj`. * Section packages are now public, and callers interact directly with these packages for writing and reading section-specific data. * All logic for a section (encoding, decoding, buffering, reading) is now fully self-contained inside the section package. Previously, the implementation of each section was spread across three packages (`pkg/dataobj/internal/encoding`, `pkg/dataobj/internal/sections/SECTION`, `pkg/dataobj`). * Cutting a section is now a decision made by the caller rather than the section implementation. Previously, the logs section builder would create multiple sections. For the most part, this change is a no-op, with two exceptions: 1. Section cutting is now performed by the caller; however, this shouldn't result in any issues. 2. Removing the high-level `dataobj.Stream` and `dataobj.Record` types will temporarily reduce the allocation gains from #16988. I will address this after this PR is merged.	8 months ago
Christian Haudum	cbb5be6d57	feat(engine): Implements predicate mapping for DataObjScan (#17621) ... This is an initial implementation of `buildLogsPredicate(expr physical.Expression) (dataobj.LogsPredicate, error)`. While there have been discussions about bypassing this approach in favor of directly using `dataset.Reader` and `dataset.Predicate`, this PR provides essential supported functionality to unblock progress in the interim. Specifically, it handles mapping common timestamp and metadata equality predicates. Signed-off-by: Christian Haudum <christian.haudum@gmail.com> Co-authored-by: Owen Diehl <ow.diehl@gmail.com>	8 months ago