database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
10,591 Commits 1 Branch 0 Tags
f788acaa6e319b1fe052dff52d2483766f7e1750
Commit Graph

1702 Commits

Author SHA1 Message Date
yagagagaga
66f53a723e [typo](docs) Add a previously missed configuration item and other note (#19814) 
fix enable_query_memory_overcommit config doc
 2023-05-19 08:38:10 +08:00
yongjinhou
40ab4ce305 fix select resource groups bug (#19808) 2023-05-18 21:54:31 +08:00
Kang
294599ee45 [feature](jsonb) rename JSONB type name and function name to JSON (#19774) 
To be more compatible with MySQL, rename JSONB type name and function name to JSON.

The old JSONB type name and jsonb_xx function can still be used for backward compatibility.

There is a function jsonb_extract remained since json_extract is used by json string function and more work need to change it. It will be changed further.
 2023-05-18 16:16:52 +08:00
yuanyuan8983
5707ecd0e5 [typo](docs) modify the dynamic table document (#19750) 2023-05-18 09:24:17 +08:00
amory
51a12f4f11 [Doc](map-type) add map sql doc (#19612) 2023-05-18 09:09:41 +08:00
lsy3993
7f9cdc41f5 [typo](docs)add new describe for grant (#19656) 2023-05-18 09:00:39 +08:00
gnehil
b32ee4facc [typo](doc)fix audit plugin and spark load kerberos for yarn #19749 2023-05-18 08:37:41 +08:00
zhuwei
40909f49cb [typo](docs) spark load example revise and some other revise (#19756) 
Co-authored-by: zhuwei <zhuwei8421@gmail.com>
 2023-05-18 08:37:23 +08:00
caoliang-web
35ad081784 [typo](doc)Example of adding stream load import to current time (#19760) 2023-05-18 08:36:39 +08:00
zhengyu
4566281cc3 [fix](sink) disable lazy-open partition by default (#19769) 
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
 2023-05-18 07:28:04 +08:00
xueweizhang
97d4778ecf [enhancement](schema) dynamic_partition.time_unit support year (#19551) 
dynamic_partition.time_unit support year
 2023-05-17 23:49:15 +08:00
yongkang.zhong
8aa7f0e188 [fix](catalog) fix the include_database_list not in effect (#19589) 2023-05-17 22:56:21 +08:00
luozenglin
60d5c82f44 [fix](tvf) fix the inconsistency between tvf backends function and show backends result (#19697) 2023-05-17 22:55:46 +08:00
catpineapple
4ecd1c683d [doc](fix) cold hot separation doc fix (#19757) 2023-05-17 22:39:31 +08:00
Mingyu Chen
2536b57590 [doc](catalog) optimize catalog doc (#19601) 2023-05-17 21:45:08 +08:00
Tiewei Fang
1eb929e1ca [Bugfix](Jdbc Catalog) fix data type mapping of SQLServer Catalog (#19525) 
We map `money/smallmoney` types of SQLSERVER into decimal type of doris.
 2023-05-17 21:02:42 +08:00
bigben0204
f95c1d7cb6 [feat](profile) Add a new rest api to query instance host and ip information for query profile action in branch master(#18668) (#19643) 2023-05-17 10:52:47 +08:00
abmdocrt
9cc7af6062 [doc](doris future) Add mentor doc for doris future in community page (#19690) 2023-05-17 08:20:35 +08:00
abmdocrt
3a7bc3a7a8 [doc](retention) optimize retention doc (#19692) 2023-05-17 08:17:45 +08:00
xy720
0d11c4207a [docs](struct-type) add docs for struct and named_struct function (#19700) 2023-05-17 08:16:33 +08:00
xy720
12c21287a5 [docs](struct-type) Add docs for struct type (#19694) 2023-05-16 19:13:27 +08:00
Ziyu Wang
325a1d4b28 [vectorized](function) support array_count function (#18557) 
support array_count function.
array_count：Returns the number of non-zero and non-null elements in the given array.
 2023-05-16 17:00:01 +08:00
Weijie Guo
9535ed01aa [feature](tvf) Support compress file for tvf hdfs() and s3() (#19530) 
We can support this by add a new properties for tvf, like :

`select * from hdfs("uri" = "xxx", ..., "compress_type" = "lz4", ...)`

User can:

Specify compression explicitly by setting `"compression" = "xxx"`.
Doris can infer the compression type by the suffix of file name(e.g. `file1.gz`)
Currently, we only support reading compress file in `csv` format, and on BE side, we already support.
All need to do is to analyze the `"compress_type"` on FE side and pass it to BE.
 2023-05-16 08:50:43 +08:00
zhangdong
e48524009d [doc](fqdn)fqdn doc en (#19634) 2023-05-16 08:48:34 +08:00
yongkang.zhong
ac9e92e1aa [typo](docs) Optimize mac compilation documentation (#19629) 2023-05-15 20:34:47 +08:00
Liqf
c87e78dc35 [bug](jsonb) fix jsonb query bug When the json key value contains "." (#19185) 
Issue Number: close #19173

mysql> SELECT jsonb_extract('{"a.b.c":{"k1":"v31", "k2.a1": 300},"a":"opentelemetry"}', '$."a.b.c".k1');
+-------------------------------------------------------------------------------------------+
| jsonb_extract('{"a.b.c":{"k1":"v31", "k2.a1": 300},"a":"opentelemetry"}', '$."a.b.c".k1') |
+-------------------------------------------------------------------------------------------+
| "v31" |
+-------------------------------------------------------------------------------------------+
1 row in set (0.06 sec)
 2023-05-15 15:43:12 +08:00
Zhengguo Yang
6748ae4a57 [Feature] Collect the information statistics of the query hit (#18805) 
1. Show the query hit statistics for `baseall`

   ```sql
    MySQL [test_query_db]> show query stats from baseall;
    +-------+------------+-------------+
    | Field | QueryCount | FilterCount |
    +-------+------------+-------------+
    | k0    | 0          | 0           |
    | k1    | 0          | 0           |
    | k2    | 0          | 0           |
    | k3    | 0          | 0           |
    | k4    | 0          | 0           |
    | k5    | 0          | 0           |
    | k6    | 0          | 0           |
    | k10   | 0          | 0           |
    | k11   | 0          | 0           |
    | k7    | 0          | 0           |
    | k8    | 0          | 0           |
    | k9    | 0          | 0           |
    | k12   | 0          | 0           |
    | k13   | 0          | 0           |
    +-------+------------+-------------+
    14 rows in set (0.002 sec)

    MySQL [test_query_db]> select k0, k1,k2, sum(k3) from baseall  where k9 > 1 group by k0,k1,k2;
    +------+------+--------+-------------+
    | k0   | k1   | k2     | sum(`k3`)   |
    +------+------+--------+-------------+
    |    0 |    6 |  32767 |        3021 |
    |    1 |   12 |  32767 | -2147483647 |
    |    0 |    3 |   1989 |        1002 |
    |    0 |    7 | -32767 |        1002 |
    |    1 |    8 |    255 |  2147483647 |
    |    1 |    9 |   1991 | -2147483647 |
    |    1 |   11 |   1989 |       25699 |
    |    1 |   13 | -32767 |  2147483647 |
    |    1 |   14 |    255 |         103 |
    |    0 |    1 |   1989 |        1001 |
    |    0 |    2 |   1986 |        1001 |
    |    1 |   15 |   1992 |        3021 |
    +------+------+--------+-------------+
    12 rows in set (0.050 sec)

    MySQL [test_query_db]> show query stats from baseall;
    +-------+------------+-------------+
    | Field | QueryCount | FilterCount |
    +-------+------------+-------------+
    | k0    | 1          | 0           |
    | k1    | 1          | 0           |
    | k2    | 1          | 0           |
    | k3    | 1          | 0           |
    | k4    | 0          | 0           |
    | k5    | 0          | 0           |
    | k6    | 0          | 0           |
    | k10   | 0          | 0           |
    | k11   | 0          | 0           |
    | k7    | 0          | 0           |
    | k8    | 0          | 0           |
    | k9    | 1          | 1           |
    | k12   | 0          | 0           |
    | k13   | 0          | 0           |
    +-------+------------+-------------+
    14 rows in set (0.001 sec)
   ```

2. Show the query hit statistics summary for all the mv in a table

   ```sql
   MySQL [test_query_db]> show query stats from baseall all;
    +-----------+------------+
    | IndexName | QueryCount |
    +-----------+------------+
    | baseall   | 1          |
    +-----------+------------+
    1 row in set (0.005 sec)
   ```

3. Show the query hit statistics detail info for all the mv in a table

   ```sql
    MySQL [test_query_db]> show query stats from baseall all verbose;
    +-----------+-------+------------+-------------+
    | IndexName | Field | QueryCount | FilterCount |
    +-----------+-------+------------+-------------+
    | baseall   | k0    | 1          | 0           |
    |           | k1    | 1          | 0           |
    |           | k2    | 1          | 0           |
    |           | k3    | 1          | 0           |
    |           | k4    | 0          | 0           |
    |           | k5    | 0          | 0           |
    |           | k6    | 0          | 0           |
    |           | k10   | 0          | 0           |
    |           | k11   | 0          | 0           |
    |           | k7    | 0          | 0           |
    |           | k8    | 0          | 0           |
    |           | k9    | 1          | 1           |
    |           | k12   | 0          | 0           |
    |           | k13   | 0          | 0           |
    +-----------+-------+------------+-------------+
    14 rows in set (0.017 sec)
   ```

4. Show the query hit for a database

   ```sql
    MySQL [test_query_db]> show query stats for test_query_db;
    +----------------------------+------------+
    | TableName                  | QueryCount |
    +----------------------------+------------+
    | compaction_tbl             | 0          |
    | bigtable                   | 0          |
    | empty                      | 0          |
    | tempbaseall                | 0          |
    | test                       | 0          |
    | test_data_type             | 0          |
    | test_string_function_field | 0          |
    | baseall                    | 1          |
    | nullable                   | 0          |
    +----------------------------+------------+
    9 rows in set (0.005 sec)
   ```

5. Show query hit statistics for all the databases

   ```sql
    MySQL [(none)]> show query stats;
    +-----------------+------------+
    | Database        | QueryCount |
    +-----------------+------------+
    | test_query_db   | 1          |
    +-----------------+------------+
    1 rows in set (0.005 sec)
   ```
 2023-05-15 10:56:34 +08:00
zzzzzzzs
91d5e956a0 [typo](doc) Fixed typos in cluster-action.md (#19549) 2023-05-14 23:52:41 +08:00
Hong Liu
80886af828 [doc](grant)add the version for grant for user; (#19556) 2023-05-14 23:52:18 +08:00
zzzzzzzs
859b203b1d [typo](doc) Fixed typos in query-profile-action.md (#19552) 2023-05-14 23:51:58 +08:00
wudi
2b402483a9 add release shade and sdk doc (#19576) 2023-05-14 23:51:17 +08:00
DongLiang-0
f4aea2a6db [Doc](binlog-load) delete binlog-load doc side bar (#19593) 2023-05-14 23:50:55 +08:00
abmdocrt
be0f4abc71 [doc](doris-future)Add doc for doris future (#19617) 2023-05-14 20:22:05 +08:00
HHoflittlefish777
f8ef25bb10 [enhancement](load) lazy-open necessary partitions when load (#18874) 2023-05-14 16:09:55 +08:00
slothever
cd9d633c1b [doc](multi-catalog)add properties converter docs (#18287) 
update doc for #18005
 2023-05-12 21:03:30 +08:00
ElvinWei
26d1eb64d2 [Doc](statistics) add statistics documents (#19323) 
The stats feature will continue to be refined, and the documentation will change over time.
 2023-05-12 20:11:29 +08:00
zzzzzzzs
feef5afa0b [typo](doc) Fixed typos in SHOW-ROUTINE-LOAD.md (#19573) 2023-05-12 14:37:28 +08:00
lsy3993
e8f1ce4eaa [typo](docs) fix doc bug of 'collect_list' and 'deploy' (#19531) 2023-05-11 16:53:29 +08:00
herry2038
834bf2eab7 [feature](array) Add array_last lambda function (#18388) 
Add array_last lambda function
 2023-05-11 13:15:54 +08:00
Yulei-Yang
41d4ed8367 [Improvement](multicatalog) support show_partitions for hms catalog (#19242) 
* [Improvement](multicatalog) support show_partitions for hms catalog

* update according review advice
 2023-05-11 01:17:23 +08:00
lsy3993
840dbdc7c0 [typo](docs) add comment of partition and key/value column (#19448) 
* change docker compose to 'docker-compose'

* modify sql of mysql

* fix docker start and stop cmd

* new commit

* add comment of partition and key/value column

* Update cn doc format

---------

Co-authored-by: Luzhijing <82810928+luzhijing@users.noreply.github.com>
 2023-05-11 01:14:17 +08:00
Mryange
d20b5f90d8 [feature](executor) Automatically set the instance_num using the info from be. (#19345) 
1. fixed some error regressions (results error with big nstance_num due to incorrect order by).
2. if set parallel_fragment_exec_instance_num to 0, the concurrency in the Pipeline execution engine will automatically be set to half of the number of CPU cores.
3. add limit to parallel_fragment_exec_instance_num that it cannot be set to more than fe.conf::max_instance_num(Default: 128)
```
mysql [(none)]>set parallel_fragment_exec_instance_num = 514;
ERROR 1231 (42000): errCode = 2, detailMessage = Variable 'parallel_fragment_exec_instance_num' can't be set to the value of '514(Should not be set to more than 128)'
```
 2023-05-10 17:07:41 +08:00
Euporia
0dd35c81b4 [docs](data-model):add sql statements to import data (#19390) 
* [docs](data-model):add sql statements to import data
* [docs](data-model)synchronize documents in English
 2023-05-10 17:06:50 +08:00
Pxl
9b7a419aed [Chore](build) update some doc about build enviroment (#19325) 
update some doc about build enviroment
 2023-05-10 16:18:44 +08:00
zzzzzzzs
7631c82eff [typo](doc) Fixed typos in native-user-defined-function.md (#19459) 2023-05-10 16:05:30 +08:00
zzzzzzzs
e60129a28b [typo](doc) Fixed typos in variables.md (#19451) 
* [typo](doc) Fixed typos in variables.md

* Update variables.md
 2023-05-10 16:04:53 +08:00
DeadlineFen
a05dbd3f81 [chore](compile) Improves PCH cache hit ratio (#19469) 
Supplement the documentation of be-clion-dev, avoid the problem of undefined DORIS_JAVA_HOME and inability to find jni.h when using clion development without directly compiling through build.sh
Complete the classification of header files in pch.h and introduce some header files that are not frequently modified in doris.
Separate the declaration and definition in common/config.h. If you need to modify the default configuration now, please modify it in common/config.cpp.
gen_cpp/version.h is regenerated every time it is recompiled, which may cause PCH to fail, so now you need to get the version information indirectly rather than directly.
 2023-05-10 12:49:01 +08:00
Qi Chen
096aa25ca6 [improvement](orc-reader) Implements ORC lazy materialization (#18615) 
- Implements ORC lazy materialization, integrate with the implementation of https://github.com/apache/doris-thirdparty/pull/56 and https://github.com/apache/doris-thirdparty/pull/62.
- Refactor code: Move `execute_conjuncts()` and `execute_conjuncts_and_filter_block()` in `parquet_group_reader `to `VExprContext`, used by parquet reader and orc reader.
- Add session variables `enable_parquet_lazy_materialization` and `enable_orc_lazy_materialization` to control whether enable lazy materialization.
- Modify `build.sh` to update apache-orc submodule or download package every time.
 2023-05-09 23:33:33 +08:00
yongkang.zhong
7c7db9ce93 [typo](docs) Add an open page cache hint to the benchmark (#19449) 2023-05-09 21:28:39 +08:00
chenlinzhong
aeb3450151 [feature](graph)Support querying data from the Nebula graph database (#19209) 
Support querying data from the Nebula graph database
This feature comes from the needs of commercial customers who have used Doris and Nebula, hoping to connect these two databases

changes mainly include:

* add New Graph Database JDBC Type
* Adapt the type and map the graph to the Doris type
 2023-05-09 15:30:11 +08:00