Commit Graph

2612 Commits

Author SHA1 Message Date
8ef6b4d996 [fix](json) fix json int128 overflow (#22917)
* support int128 in jsonb

* fix jsonb int128 write

* fix jsonb to json int128

* fix json functions for int128

* add nereids function jsonb_extract_largeint

* add testcase for json int128

* change docs for json int128

* add nereids function jsonb_extract_largeint

* clang format

* fix check style

* using int128_t = __int128_t for all int128

* use fmt::format_to instead of snprintf digit by digit for int128

* clang format

* delete useless check

* add warn log

* clang format
2023-08-25 11:40:30 +08:00
ffadf09eec [fix](catalog)add custom jar (#23406)
- allow put custom jar in `${DORIS_HOME}/lib/java_extensions/custom_extension` such as `paimon-s3-0.4.0-incubating.jar`
- add some note for paimon and fqdn
2023-08-25 11:10:53 +08:00
71071ba057 [feature](move-memtable)[4/7] add stream sink file writer (#23416)
Co-authored-by: laihui <1353307710@qq.com>
2023-08-25 00:08:27 +08:00
adafcc1f7a [typo](docs) Fix Sidebar Category Name of CN Version and Capitalize Files Name (#23050) 2023-08-24 22:56:23 +08:00
e262c8b4be [chore](fe)correct some doc errors (#23398) 2023-08-24 17:05:13 +08:00
448b7755c6 [feature](jdbc catalog) support doris jdbc catalog array type (#23056) 2023-08-23 21:17:16 +08:00
22e373a799 [feature](vector-search) add 4 distance functions to support vector search (#23129) 2023-08-23 15:51:15 +08:00
c7b9eb5f9c [enhancement](bitmap)support bitmap type for non-key column in unique table (#23228) 2023-08-23 14:21:22 +08:00
d9993a7790 [doc](multi-catalog)add gcs broker load faq (#23229)
add gcs broker load faq

When the '_' is used  at buckt name, will report an error:  Missing required property: uri

the gcs bucket not match the s3 bucket name format: https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/userguide/bucketnamingrules.html

we'd better not use  '_' in bucket.

<!--Describe your changes.-->
2023-08-22 22:48:31 +08:00
a7675243d9 [fix](jdbc catalog) fix adaptation to Oracle special character / table names (#23080)
The changes of this PR for JdbcOracleClient are as follows:

#### bug fixes:
  1. Fix the problem that if there is an approximate table name for Schema synchronization with a table name with `/` characters, the synchronization Column will be confused
  2. Fix the NPE problem of metadata synchronization after enabling lower_case_table_names configuration

#### improvement:
  1. Modify the method of synchronizing Oracle User to Doris Database mapping, use `metadata.getSchemas` instead of `SELECT DISTINCT OWNER FROM all_tables`
  2. When synchronizing metadata, change `null` at the catalog level to `conn.getcatalog`
2023-08-22 15:25:42 +08:00
51db11ed0b [improve](jdbc catalog) Add a variable to accommodate the final keyword in ClickHouse Jdbc Catalog queries (#23282) 2023-08-22 12:13:36 +08:00
2a5aed46f7 add CloudCanal en doc (#22731) 2023-08-22 10:49:54 +08:00
81dd00f6e4 [Feature](Compaction) Support do full compaction by table id (#22010) 2023-08-21 11:54:51 +08:00
Pxl
a11e0e3bc4 [Bug](agg) fix QUANTILE_UNION many problems (#23181)
fix QUANTILE_UNION many problems
2023-08-21 10:04:27 +08:00
97fa840324 [feature](multi-catalog)support iceberg hadoop catalog external table query (#22949)
support iceberg hadoop catalog external table query
2023-08-20 19:29:25 +08:00
ac855e75f1 [fix](doc) fix from_unixtime error range #23176 2023-08-19 12:17:13 +08:00
0838ff4bf4 [fix](Outfile) fix bug that the fileSize is not correct when outfile is completed (#22951) 2023-08-18 22:31:44 +08:00
5b8a76a22e [doc](catalog)faq for lzo.jar not found (#23070) 2023-08-18 10:16:32 +08:00
6e51632ca9 [docs](kerberos)add FAQ cases and enable krb5 debug (#22821) 2023-08-17 14:25:09 +08:00
41bce29ae3 [docs](docs)Rename Title and URL of Bitwise Functions (#22722) 2023-08-17 11:18:02 +08:00
4512569a3a [docs](releasenote)Update en release note 2.0.0 (#23041) 2023-08-16 15:13:09 +08:00
a2095b7d9e [fix](docs) add enable_single_replica_load on be config doc (#22948) 2023-08-16 10:31:01 +08:00
fe08db191f [typo](docs) Optimize the release note 2.0.0 (#22926) 2023-08-15 20:09:56 +08:00
27f5b623e6 [Chore](docs)Add SSL Faq (#22956) 2023-08-15 09:49:39 +08:00
c67d1cc805 [docs](releasenote)2.0.0 release note (#22904) 2023-08-14 10:11:03 +08:00
e2b06cd0cf [opt](docs) Optimize docs to avoid user set wrong replication_allocation (#22767) 2023-08-14 09:38:22 +08:00
79a61ced42 [docs](load) fix indentation in stream load manual (#22807) 2023-08-13 10:16:11 +08:00
1f8cb3f54a [Chore](doc) Fix doc zh-CN typo (#22903)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-08-12 16:14:06 +08:00
2b81553879 [doc](docs) Add some docs of baidu cloud bos (#22833)
* [doc](docs) Add some docs of baidu cloud bos

* fix
2023-08-12 07:09:57 +08:00
5b09254fac [improvement](external statistics)Fix external stats collection bugs (#22788)
1. Collect external table row count when execute analyze database.
2. Support show cached table stats (row count)
3. Support alter external table column stats.
4. Refresh/Invalidate table row count stat memory cache when analyze task finished and drop table stats.
2023-08-11 21:58:24 +08:00
84ee814bc3 [docs](docs) Update invalid pics of release note 1.1.0 and 2.0-beta (#22804) 2023-08-11 20:08:21 +08:00
3e9ba632d7 [typo](docs) Add a guide to using SQL for the jdbc catalog (#22880) 2023-08-11 16:28:42 +08:00
0c38f42827 [fix](doc) Remove introduction to unstable features (#22832)
1. Remove introduction to unstable features
2. Rename some sub-titles to avoid mixed use of chiniese and english
2023-08-11 15:59:16 +08:00
94a7b44540 [Improvement](log) add config to controll compression of fe log & fe audit log (#22865)
fe log is large for a busy doris cluster, if you want to preserve some historical logs, it cost too much disk space.
enable compression is a good way to save space.
and a gzip compressed text file can be viewed without decompression.
2023-08-11 14:08:08 +08:00
caf496a67e [Chore](RoutineLoad)Change max_batch_interval minimum limit from 5 to 1 (#22858) 2023-08-11 12:02:20 +08:00
71807ceb5f [Enhancement](tvf) Table value function support reading local file (#17404)
I tested the local tvf with tpch queries. First, generate `lineitem` datasets with 6001215 rows, and load it into `lineitem` table by:
```
insert into lineitem select c11, c1, c4, c2, c3, c5, c6, c7, c8, c9, c10, c12, c13, c14, c15, c16 
from local(
        "file_path" = "tools/tpch-tools/bin/tpch-data/lineitem.tbl.1", 
        "backend_id" = "10003", 
        "format" = "csv", 
        "column_separator" = "|"
);
```
Then, run `q1` and `q16` tpch queries, the query result is correct.

It can also analyze the BE's log directly like:

```
mysql> select * from local(
        "file_path" = "log/be.out",
        "backend_id" = "10006",
        "format" = "csv")
       where c1 like "%start_time%" limit 10;
+--------------------------------------------------------+
| c1                                                     |
+--------------------------------------------------------+
| start time: 2023年 08月 07日 星期一 23:20:32 CST       |
| start time: 2023年 08月 07日 星期一 23:32:10 CST       |
| start time: 2023年 08月 08日 星期二 00:20:50 CST       |
| start time: 2023年 08月 08日 星期二 00:29:15 CST       |
+--------------------------------------------------------+
```
2023-08-10 20:07:42 +08:00
221e860cb7 [Feature](Routine Load)Support Partial Update (#22785) 2023-08-10 17:41:53 +08:00
f2658dc7bd [Feature](multi-catalog) Truncate char or varchar columns if size is smaller than file columns or not found in the file column schema. (#22318)
Truncate char or varchar columns if size is smaller than file columns or not found in the file column schema by session var `truncate_char_or_varchar_columns`.
2023-08-10 14:37:20 +08:00
57fb9799b5 [feature](agg) add aggregation function 'bitmap_agg' (#22768)
This function can be used to replace bitmap_union(to_bitmap(expr)), because bitmap_union(to_bitmap(expr)) need create many many small bitmaps firstly and then merge them into a single bitmap.
bitmap_agg will convert the column value into a bitmap directly. Its performance is better than bitmap_union(to_bitmap(expr)) . In our test , there is about 30% improvement.
2023-08-10 12:18:25 +08:00
c1bc2c289b [doc](stats) Add description for some new configure option in stats related docs (#22723) 2023-08-10 11:37:50 +08:00
eafdab0cfd [Enhancement](tvf) Add frontends_disks table-valued-function (#22568)
---------

Co-authored-by: yuxianbing <yuxianbing@yy.com>
Co-authored-by: yuxianbing <iloveqaz123>
2023-08-10 10:40:24 +08:00
HB
5147c096ef [Enhancement] Add an API to query session information for all FEs (#20134)
Currently, Doris only has one interface for querying specific FE session information, and many times we need to know how many session information there are in the current cluster, so I added this API.

`
GET /rest/v1/session/all

{
"msg": "success",
"code": 0,
"data": {
"column_names": ["FE", "Id", "User", "Host", "Cluster", "Db", "Command", "Time", "State", "Info"],
"rows": [{
"FE": "10.14.170.23",
"User": "root",
"Command": "Sleep",
"State": "",
"Cluster": "default_cluster",
"Host": "10.81.85.89:31465",
"Time": "230",
"Id": "0",
"Info": "",
"Db": "db1"
},
{
"FE": "10.14.170.24",
"User": "root",
"Command": "Sleep",
"State": "",
"Cluster": "default_cluster",
"Host": "10.81.85.88:61465",
"Time": "460",
"Id": "1",
"Info": "",
"Db": "db1"
}]
},
"count": 2
}
`
2023-08-09 19:02:45 +08:00
9422494064 [docs](docs)Rename Title and URL of HLL Functions (#22728) 2023-08-09 15:53:39 +08:00
58ef388c32 [docs](docs)Rename Title and URL of JSON Functions (#22732) 2023-08-09 15:53:25 +08:00
af5f3ae2a6 [docs](docs)Rename Title & URL and Change Category Name as Numeric of Math Functions (#22733) 2023-08-09 15:52:49 +08:00
2fb7aba9bc [docs](docs)Rename Title and URL of IP Functions (#22741) 2023-08-09 15:52:35 +08:00
910863b329 [docs](docs) Rename Window Functions (#22742) 2023-08-09 15:52:22 +08:00
780ba83d91 [docs](docs)Rename the Files Without Category of Sql Functions (#22746) 2023-08-09 15:51:47 +08:00
61e661d389 [docs](docs)Rename Title and URL of Table Functions (#22747) 2023-08-09 15:51:15 +08:00
c443bce141 [docs](docs)Delete Dash Between Title of Benchmark (#22751) 2023-08-09 15:51:01 +08:00