airborne12 fc2b9db0ad
[Feature](inverted index) add tokenize function for inverted index ( #21813 ) ...
In this PR, we introduce TOKENIZE function for inverted index, it is used as following:
```
SELECT TOKENIZE('I love my country', 'english');
```
It has two arguments, first is text which has to be tokenized, the second is parser type which can be **english**, **chinese** or **unicode**.
It also can be used with existing table, like this:
```
mysql> SELECT TOKENIZE(c,"chinese") FROM chinese_analyzer_test;
+---------------------------------------+
| tokenize(`c`, 'chinese') |
+---------------------------------------+
| ["来到", "北京", "清华大学"] |
| ["我爱你", "中国"] |
| ["人民", "得到", "更", "实惠"] |
+---------------------------------------+
```
2023-07-25 15:05:35 +08:00
2023-03-07 10:28:56 +08:00
2023-07-05 15:16:04 +08:00
2023-01-17 16:48:02 +08:00
2023-07-18 09:17:14 +08:00
2023-06-21 16:25:07 +08:00
2023-05-15 15:13:16 +08:00
2023-05-29 15:38:31 +08:00
2023-07-16 13:21:15 +08:00
2023-06-19 14:11:10 +08:00
2023-01-18 16:41:41 +08:00
2023-07-10 19:21:32 +08:00
2023-07-14 10:25:16 +08:00
2022-08-19 10:57:55 +08:00
2023-07-25 14:41:36 +08:00
2023-07-18 12:53:09 +08:00
2023-04-07 17:04:08 +08:00
2023-07-10 17:57:21 +08:00
2023-06-01 13:09:58 +08:00
2023-07-18 19:10:55 +08:00
2023-01-13 18:33:40 +08:00
2023-06-26 19:32:43 +08:00
2023-06-06 10:35:30 +08:00
2023-05-29 15:38:31 +08:00
2023-05-17 21:26:31 +08:00
2023-07-01 12:29:22 +08:00
2023-07-03 09:32:55 +08:00
2023-07-24 22:31:18 +08:00
2023-01-10 17:25:06 +08:00
2023-06-13 14:31:24 +08:00
2023-07-18 09:17:14 +08:00
2023-05-14 20:01:30 +08:00
2023-06-06 12:04:03 +08:00
2023-07-25 15:05:35 +08:00
2023-06-01 13:09:58 +08:00
2023-07-25 11:56:20 +08:00
2023-07-22 14:08:44 +08:00
2023-05-17 20:50:15 +08:00
2023-06-30 14:29:21 +08:00
2023-07-12 21:37:36 +08:00
2023-05-29 15:38:31 +08:00
2023-07-21 17:02:01 +08:00
2023-03-30 14:42:34 +08:00
2023-05-30 16:26:00 +08:00
2023-02-19 15:11:54 +08:00
2023-03-21 23:22:17 +08:00
2023-07-22 15:27:05 +08:00
2023-04-07 14:23:16 +08:00
2023-04-03 09:49:07 +08:00
2023-06-01 13:09:58 +08:00
2023-07-24 16:08:00 +08:00
2023-07-21 13:57:27 +08:00
2023-07-07 14:38:28 +08:00
2023-07-25 14:45:20 +08:00
2023-06-14 17:34:34 +08:00
2023-07-07 17:45:55 +08:00
2023-07-07 17:45:55 +08:00
2022-09-19 08:38:17 +08:00
2023-07-19 14:05:56 +08:00
2023-01-18 12:33:20 +08:00
2023-06-08 10:55:22 +08:00
2022-11-22 21:35:18 +08:00
2023-07-25 11:21:03 +08:00
2023-07-11 14:27:03 +08:00
2023-03-20 13:10:12 +08:00
2023-05-17 21:26:31 +08:00
2023-06-27 19:55:35 +08:00
2023-04-11 11:09:43 +08:00
2023-06-21 18:36:46 +08:00
2023-04-26 15:34:46 +08:00
2022-10-04 20:56:22 +08:00
2023-04-07 14:23:16 +08:00
2022-11-22 10:36:12 +08:00
2023-02-17 18:48:27 +08:00
2023-05-26 16:05:38 +08:00
2023-05-29 15:38:31 +08:00
2023-05-24 20:17:28 +08:00
2023-06-01 13:09:58 +08:00
2023-06-01 13:09:58 +08:00
2023-06-25 11:16:56 +08:00
2023-06-06 11:05:56 +08:00
2023-05-10 15:06:23 +08:00
2023-06-01 13:09:58 +08:00
2023-06-01 13:09:58 +08:00
2023-04-20 11:02:53 +08:00
2022-11-28 19:30:32 +08:00
2023-06-06 11:05:56 +08:00
2022-11-28 19:30:32 +08:00
2023-06-06 11:05:56 +08:00
2023-06-06 11:05:56 +08:00
2023-06-16 10:09:22 +08:00
2023-06-01 13:09:58 +08:00
2023-05-29 15:38:31 +08:00
2023-05-22 11:45:53 +08:00
2023-06-21 18:36:46 +08:00
2023-07-23 13:35:08 +08:00
2023-03-23 17:59:49 +08:00
2023-06-21 18:36:46 +08:00
2023-03-29 10:49:39 +08:00
2022-09-29 10:45:17 +08:00
2023-04-07 14:23:16 +08:00
fc2b9db0ad