Now, regression data is stored in sf1DataPath, which is local or remote.
For performance reason, we use local dir for community pipeline, however, we need prepare data for every machine,
this process is easy mistake. So we cache data from s3 in local transparently, thus, we just need to config one data source.
There are currently many types of ScanNodes in Doris. And most of the logic of these ScanNodes is the same, including:
Runtime filter
Predicate pushdown
Scanner generation and scheduling
So I intend to unify the common logic of all ScanNodes.
Different data sources only need to implement different Scanners for data access.
So that the future optimization for scan can be applied to the scan of all data sources,
while also reducing the code duplication.
This PR mainly adds 4 new class:
VScanner
All Scanners' parent class. The subclasses can inherit this class to implement specific data access methods.
VScanNode
The unified ScanNode, and is responsible for common logic including RuntimeFilter, predicate pushdown, Scanner generation and scheduling.
ScannerContext
ScannerContext is responsible for recording the execution status
of a group of Scanners corresponding to a ScanNode.
Including how many scanners are being scheduled, and maintaining
a producer-consumer blocks queue between scanners and scan nodes.
ScannerContext is also the scheduling unit of ScannerScheduler.
ScannerScheduler schedules a ScannerContext at a time,
and submits the Scanners to the scanner thread pool for data scanning.
ScannerScheduler
Unified responsible for all Scanner scheduling tasks
Test:
This work is still in progress and default is disabled.
I tested it with jmeter with 50 concurrency, but currently the scanner is just return without data.
The QPS can reach about 9000.
I can't compare it to origin implement because no data is read for now. I will test it when new olap scanner is ready.
Co-authored-by: morningman <morningman@apache.org>
* use suffix of directory as group name and use directory as dbname
We can rename tpcds_sf1 to tpcds_sf1_p1, then tpcds_sf1 will be in group
p1. We will group cases to p0, p1, p2, p3 in the future.
p0: function cases running in seconds.
p1: cases with expected out running in minutes, like tpcds_sf1
p2: cases with expected out running in hours, like tpcds_sf10 tpcds_sf100
p3: cases without without expected out to test core dump.
current nereids planner execute ssb will run into dead loop and crash be, this pr fix this problem and add some regression test case prevent execute ssb failed
Support register suite plugin to add third-party function.
See
1. register in: ${DORIS_HOME}/regression-test/plugins/plugin_example.groovy
2. usage: ${DORIS_HOME}/regression-test/suites/demo/test_plugin.groovy
3. doc: ${DORIS_HOME}/docs/zh-CN/developer-guide/regression-testing.md
1. add a new config in regression-conf.groovy
enableHdfs, default is false, to skip tests with hdfs
2. fix a bug that when double type column result is null, exception will be thrown
* [Feature][regression-test]CSV import and export support header (#8487)
1.Add two new types to stream load boker load: csv_with_names and csv_with_name_sand_types
2.Add two new types to export: csv_with_names and csv_with_names_and_types
support suite block to specify multiple groups.
TestAction support compare result to iterator, local file and http stream.
support print teamcity service message.
abandon the logical: generate groovy file for sql file
support 3 levels parrallel: script file, suite block, thread action
support specify JAVA_OPTS for boot shell
avoid jvm metaspace oom
use -d to run the suite in some directories, instead of -g. and -g is used to specify groups
Support a lot of actions for regression testing framework.
e.g. thread, lazyCheck, onSuccess, connect, selectUnionAll, timer
Demo exists in ${DORIS_HOME}/regression-test/suites/demo
Add scalable regression testing framework(#7584)
contains
- Test framework making by groovy, and support built-in **readable DSL** named as `Action`
- Demo exists in `${DORIS_HOME}/regression-test/data/demo`
- Chinese doc exist in `${DORIS_HOME}/docs/zh-CN/developer-guide/regression-testing.md`
English document coming soon
