Commit Graph

2042 Commits

Author SHA1 Message Date
50e6c4216a [vectorized](function) suppoort date_trunc function truncate week mode (#18334)
support date_trunc could truncate week eg:
select date_trunc('2023-4-3 19:28:30', 'week');
2023-04-04 10:24:26 +08:00
8b85c55117 [vectorized](function) Support array_shuffle and shuffle function. (#18116)
---------

Co-authored-by: zhangyu209 <zhangyu209@meituan.com>
2023-04-04 08:53:13 +08:00
eb6dbc03e0 [typo](docs) add regression test doc & fix api doc (#18329) 2023-04-03 17:40:41 +08:00
aff260c06f [Enhancement](HttpServer) Support https interface (#16834)
1. Organize http documents
2. Add http interface authentication for FE
3. **Support https interface for FE**
4. Provide authentication interface
5. Add http interface authentication for BE
6. Support https interface for BE
2023-04-03 14:18:17 +08:00
ecd3fd07f6 [feature](colocate) support cross database colocate join (#18152) 2023-04-03 14:03:42 +08:00
961f5d1bb7 [feature](function)Add St_Angle/St_Azimuth function (#18293)
Add St_Angle/St_azimuth function:
St_Angle:
Enter three point, which represent two intersecting lines. Returns the angle between these lines. Point 2 and point 1 represent the first line and point 2 and point 3 represent the second line. The angle between these lines is in radians, in the range [0, 2pi). The angle is measured clockwise from the first line to the second line.

`

mysql> SELECT ST_Angle(ST_Point(1, 0),ST_Point(0, 0),ST_Point(0, 1));
+----------------------------------------------------------------------+
| st_angle(st_point(1.0, 0.0), st_point(0.0, 0.0), st_point(0.0, 1.0)) |
+----------------------------------------------------------------------+
| 4.71238898038469 |
+----------------------------------------------------------------------+
1 row in set (0.04 sec)
`

St_azimuth:
Enter two point, and returns the azimuth of the line segment formed by points 1 and 2. The azimuth is the angle in radians measured between the line from point 1 facing true North to the line segment from point 1 to point 2.
`

mysql> SELECT st_azimuth(ST_Point(0, 0),ST_Point(1, 0));
+----------------------------------------------------+
| st_azimuth(st_point(0.0, 0.0), st_point(1.0, 0.0)) |
+----------------------------------------------------+
| 1.5707963267948966 |
+----------------------------------------------------+
1 row in set (0.04 sec)
2023-04-03 13:01:59 +08:00
94e3472050 [bug](function) fix count equal function return incorrect value (#18200)
fix count equal function return incorrect value
2023-04-03 11:20:36 +08:00
03fc41ea51 [doc](catalog) add faq for hive catalog (#18298) 2023-04-03 09:01:49 +08:00
04929ff6d4 [fix](doc) suggest use window function to replace running_difference (#18281) 2023-04-02 16:35:10 +08:00
20b3bdb000 [vectorized](function) support array_first_index function (#18175)
mysql> select array_first_index(x->x+1>3, [2, 3, 4]);
+-------------------------------------------------------------------+
| array_first_index(array_map([x] -> x(0) + 1 > 3, ARRAY(2, 3, 4))) |
+-------------------------------------------------------------------+
|                                                                 2 |
+-------------------------------------------------------------------+

mysql> select array_first_index(x -> x is null, [null, 1, 2]);
+----------------------------------------------------------------------+
| array_first_index(array_map([x] -> x(0) IS NULL, ARRAY(NULL, 1, 2))) |
+----------------------------------------------------------------------+
|                                                                    1 |
+----------------------------------------------------------------------+

mysql> select array_first_index(x->power(x,2)>10, [1, 2, 3, 4]);
+---------------------------------------------------------------------------------+
| array_first_index(array_map([x] -> power(x(0), 2.0) > 10.0, ARRAY(1, 2, 3, 4))) |
+---------------------------------------------------------------------------------+
|                                                                               4 |
+---------------------------------------------------------------------------------+
2023-03-31 12:51:29 +08:00
525f15dddf [vectorized](function) support array_sortby function (#18071) 2023-03-30 11:07:49 +08:00
01d012bab7 [fix](memory) Remove page cache regular clear, disabled jemalloc prof by default (#18218)
Remove page cache regular clear
Now the page cache is turned off by default. If the user manually opens the page cache, it can be considered that the user can accept the memory usage of the page cache, and then can consider adding a manual clear command to the cache.

fix memory gc cancel top memory query

jemalloc prof is not enabled by default
2023-03-30 09:39:37 +08:00
7e575d4286 [typo](docs) fix the error of markdown syntax (#18219) 2023-03-30 07:22:14 +08:00
6964d9f99c [fix](function) resubmit-fix AES/SM3/SM4 encrypt/ decrypt algorithm initialization vector bug (#17907)
* Revert "[fix](function) fix AES/SM3/SM4 encrypt/ decrypt algorithm initialization vector bug (#17420)"

This reverts commit 397cc011c4f1ba5a25c770258c13f1cd3f28b47d.

* [fix-resubmit](function) fix AES/SM3/SM4 encrypt/ decrypt algorithm initialization vector bug (#17420)

ECB algorithm, block_encryption_mode does not take effect, it only takes effect when init vector is provided.
Solved: 192/256 supports calculation without init vector

For other algorithms, an error should be reported when there is no init vector

Initialization Vector. The default value for the block_encryption_mode system variable is aes-128-ecb, or ECB mode, which does not require an initialization vector. The alternative permitted block encryption modes CBC, CFB1, CFB8, CFB128, and OFB all require an initialization vector.

Reference: https://dev.mysql.com/doc/refman/8.0/en/encryption-functions.html#function_aes-decrypt

Note: This fix does not support smooth upgrades. during upgrade process, query may report error: funciton not found
2023-03-29 21:13:01 +08:00
c3fe113894 rename PaloFe to DorisFE (#18167) 2023-03-29 00:30:16 +08:00
012f7bd031 [feature](function)Add ST_Area function (#18138) 2023-03-28 19:36:09 +08:00
6bd2609294 [Enhancement](multi-catalog) add config for external meta cache loade… (#18117)
Add config for external cache-loader's max thread-pool size.
2023-03-28 15:10:19 +08:00
1956f04aa2 [feature](multi-catalog) add specified_database_list PROPERTY for jdbc/hms/iceberg catalog (#17803)
add specified_database_list PROPERTY for jdbc catalog, user can use many database specified by jdbc catalog
2023-03-28 14:04:41 +08:00
6af93016a8 [typo](docs) fix docs DROP-CATALOG.md (#18135) 2023-03-28 10:17:07 +08:00
60073ebc84 [typo](docs) fix docs install-deploy.md (#18132) 2023-03-28 10:16:51 +08:00
d8ab8662af [typo](docs) fix docs multi-catalog.md (#18133) 2023-03-28 10:16:38 +08:00
4e015fcfb2 [typo](docs) fix docs DROP-ROLE.md (#18143) 2023-03-28 10:16:21 +08:00
a7ce99d8b3 [typo](docs) fix docs SELECT.md (#18144) 2023-03-28 08:45:56 +08:00
ee80c12815 [feature](json) add json_extract function (#17808) 2023-03-27 21:19:47 +08:00
785e3e3bca [Enhancement](multi catalog) Support hive meta cache TTL (#18102)
Currently, if user modify the file on hdfs directly, no through hive. The changes of file will not be noticed by Doris and user
will get wrong data. Support the TTL(Time-to-Live) config of File Cache, so that the stale file info will be invalidated automatically after expiring.

1.Add a parameter configuration to set file cache ttl. "file.meta.cache.ttl-second".
2.Set the value corresponding to guava expireAfterAccess to the configuration value.

Co-authored-by: lexluo <lexluo@tencent.com>
2023-03-27 19:19:31 +08:00
bcf95cd920 [feature](function)Add ST_Angle_Sphere function (#17919) 2023-03-27 10:14:46 +08:00
5463ba6267 [doc](fqdn)fqdn and k8s doc (#17318) 2023-03-26 22:04:21 +08:00
3e8b3d68fc [BugFix](jdbc catalog) fix OOM when jdbc catalog querys large data from doris #18067
When using JDBC Catalog to query the Doris data, because Doris does not provide the cursor reading method (that is, fetchBatchSize is invalid), Doris will send the data to the client at one time, resulting in client OOM.

The MySQL protocol provides a stream reading method. Doris can use this method to avoid OOM. The requirements of using the stream method are setting fetchbatchsize =  Integer.MIN_VALUE and setting ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY
2023-03-26 20:02:03 +08:00
2a0890d803 [feature](datatype) add show data types stmt (#18111) 2023-03-26 12:37:06 +08:00
5df011cd43 [typo](doc)Add cancel create materialized view grammar #18084 2023-03-26 11:39:25 +08:00
360d3050bc [Feature](array-function) Support array_reverse_sort function (#17754)
Co-authored-by: zhangyu209 <zhangyu209@meituan.com>
2023-03-25 21:58:11 +08:00
50eeb2d9a4 [fix](json) change int to bigint for json function (#17769) 2023-03-25 21:57:29 +08:00
855852d582 [enhancement](timeout) fix set timeout failure and simplify timeout logic (#17837) 2023-03-25 21:56:06 +08:00
f36465e76e [enhancement](memory) optimize jemalloc heap profile doc (#18094) 2023-03-25 13:04:45 +08:00
2408ca5da8 [Bug](DECIMALV3) Fix wrong precision for plus/minus (#18052)
Result type for DECIMAL(x, y) plus/minus DECIMAL(m, n) should be DECIMAL(max(x - y, m - n) + max(y + n) + 1, max(y + n))
2023-03-25 09:42:39 +08:00
0523860877 [Enhancement](streamload) print profile for streamload (#18015)
When both enable_profile and enable_stream_load_profile_log is true, stream load profile is printed to the log
2023-03-24 20:17:33 +08:00
039688978c [docs](doc) Add autobucket doc (#16746)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-03-24 19:58:58 +08:00
562f572311 [enhancement](UDF) The user defined functions support global ('show functions'/'show create') operation (#16973) (#17964)
1. add the global keyword.

SHOW [GLOBAL] [FULL] [BUILTIN] FUNCTIONS [IN|FROM db] [LIKE 'function_pattern']

SHOW CREATE GLOBAL FUNCTION function_name(arg_type [, ...]);

2. show the details of the global udf.
2023-03-24 19:07:38 +08:00
eb7b59c1c6 [docs](plugins) Fix the information in auditlog plugin documentation #18073
The information in the document is incomplete, user may be get error message like:

mysql> INSTALL PLUGIN FROM "http://127.0.0.1:8039/auditloader.zip";
ERROR 1105 (HY000): errCode = 2, detailMessage = http://127.0.0.1:8039/auditloader.zip.md5. you should set md5sum in plugin properties or provide a md5 URI to check plugin file
2023-03-24 18:16:16 +08:00
089a91ecd5 [vectorized](function) support array_exists lambda function (#17931)
Co-authored-by: zhangyu209 <zhangyu209@meituan.com>
2023-03-23 11:11:39 +08:00
d9059ef070 [Docs](multi-catalog) add yarn.resourcemanager.principal for hive catalog with kerberos enabled. (#17930)
Co-authored-by: wangxiangyu@360shuke.com <wangxiangyu@360shuke.com>
2023-03-22 23:34:33 +08:00
e2e806a5e7 [improve](clickhouse jdbc) support clickhouse array type (#17993)
In this PR, I match the array type of ClickHouse to the array type of Doris's jdbc external.
2023-03-22 19:42:32 +08:00
ebef0c038d Revert "[fix](function) fix AES/SM3/SM4 encrypt/ decrypt algorithm initialization vector bug (#17420)" (#17887)
This reverts commit 397cc011c4f1ba5a25c770258c13f1cd3f28b47d.
2023-03-22 13:28:25 +08:00
d4ca7cb57a [chore](macOS) Specify the version of LLVM for Homebrew to install it (#17945)
Clang 16 was released last week and we haven't ported the codebase to it. If Homebrew bumped the version of LLVM, our workflows would fail.
2023-03-22 11:09:28 +08:00
173d68409c [enhencement](planner) update and delete support use alias for target table (#17914) 2023-03-22 11:07:39 +08:00
213735b5fb [doc](auth)ranger doc (#17927) 2023-03-22 09:38:54 +08:00
4193884a32 [feature](array_zip) Support array_zip function (#17696) 2023-03-21 18:44:30 +08:00
6c8ed9135d [fix](truncate) fix unable to truncate table due to wrong storage medium (#17917)
When setting FE config default_storage_medium to SSD, and set all BE storage path as SSD.
And table will be stored with storage medium SSD.
But there is a FE config storage_cooldown_second and its default value is 30 days.
So after 30 days, the storage medium of table will be changed to HDD, which is unexpected.

This PR removes the storage_cooldown_second, and use a max value to set the cooldown time of SSD
storage medium when the default_storage_medium is SSD.
2023-03-21 10:04:47 +08:00
bae9d8d7f2 [Feature-Wip](MySQL LOAD)Add trim quotes property for mysql load (#17775)
Add trim quotes property for mysql load to trim double quotes in the load files.
2023-03-21 00:32:58 +08:00
dc284b62d9 [vectorized](function) support array_filter function (#17832) 2023-03-20 23:18:10 +08:00