7182f14645
In previous implementation, when doing list partition prune, we need to generation `rangeToId` every time we doing prune. But `rangeToId` is actually a static data that should be create-once-use-every-where. So for hive partition, I created the `rangeToId` and all other necessary data structures for partition prunning in partition cache, so that we can use it directly. In my test, the cost of partition prune for 10000 partitions reduce from 8s -> 0.2s. Aslo add "partition" info in explain string for hive table. ``` | 0:VEXTERNAL_FILE_SCAN_NODE | | predicates: `nation` = '0024c95b' | | inputSplitNum=1, totalFileSize=4750, scanRanges=1 | | partition=1/10000 | | numNodes=1 | | limit: 10 | ``` Bug fix: 1. Fix bug that es scan node can not filter data 2. Fix bug that query es with predicate like `where substring(test2,2) = "ext2";` will fail at planner phase. `Unexpected exception: org.apache.doris.analysis.FunctionCallExpr cannot be cast to org.apache.doris.analysis.SlotRef` TODO: 1. Some problem when quering es version 8: ` Unexpected exception: Index: 0, Size: 0`, will be fixed later.
66 lines
3.6 KiB
Bash
Executable File
66 lines
3.6 KiB
Bash
Executable File
#!/bin/bash
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
|
|
################################################################
|
|
# This script will restart all thirdparty containers
|
|
################################################################
|
|
|
|
set -eo pipefail
|
|
|
|
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd)"
|
|
|
|
# If you want to start multi group of these containers on same host,
|
|
# Change this to a specific string.
|
|
# Do not use "_" or other sepcial characters, only number and alphabeta.
|
|
# NOTICE: change this uid will modify the file in docker-compose.
|
|
CONTAINER_UID="doris--"
|
|
|
|
# elasticsearch
|
|
sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/elasticsearch/es.yaml
|
|
sudo docker compose -f "${ROOT}"/docker-compose/elasticsearch/es.yaml --env-file "${ROOT}"/docker-compose/elasticsearch/es.env down
|
|
sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/data/es6/
|
|
sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/data/es6/*
|
|
sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/data/es7/
|
|
sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/data/es7/*
|
|
sudo mkdir -p "${ROOT}"/docker-compose/elasticsearch/data/es8/
|
|
sudo rm -rf "${ROOT}"/docker-compose/elasticsearch/data/es8/*
|
|
sudo chmod -R 777 "${ROOT}"/docker-compose/elasticsearch/data
|
|
sudo docker compose -f "${ROOT}"/docker-compose/elasticsearch/es.yaml --env-file "${ROOT}"/docker-compose/elasticsearch/es.env up -d --remove-orphans
|
|
|
|
# mysql 5.7
|
|
sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml
|
|
sudo docker compose -f "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml --env-file "${ROOT}"/docker-compose/mysql/mysql-5.7.env down
|
|
sudo mkdir -p "${ROOT}"/docker-compose/mysql/data/
|
|
sudo rm "${ROOT}"/docker-compose/mysql/data/* -rf
|
|
sudo docker compose -f "${ROOT}"/docker-compose/mysql/mysql-5.7.yaml --env-file "${ROOT}"/docker-compose/mysql/mysql-5.7.env up -d
|
|
|
|
# pg 14
|
|
sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml
|
|
sudo docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14.env down
|
|
sudo mkdir -p "${ROOT}"/docker-compose/postgresql/data/data
|
|
sudo rm "${ROOT}"/docker-compose/postgresql/data/data/* -rf
|
|
sudo docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14.env up -d
|
|
|
|
# hive
|
|
# before start it, you need to download parquet file package, see "README" in "docker-compose/hive/scripts/"
|
|
sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/hive/hive-2x.yaml
|
|
sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/hive/hadoop-hive.env.tpl
|
|
sudo "${ROOT}"/docker-compose/hive/gen_env.sh
|
|
sudo docker compose -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive.env down
|
|
sudo docker compose -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive.env up -d
|