Mingyu Chen
e99b33c274
[opt](file-meta-cache) reduce file meta cache size and disable cache for some cases ( #32340 )
...
File meta cache on BE is used to cache the meta for external table's file such as parquet footer.
This cache is counted by number, not memory consumption.
So if the cache object is big(eg, a large parquet footer), the total memory consumption of this cache
will be large and causing OOM.
This PR mainly changes:
1. Add a new method `exceed_prune_limit()` for `CachePolicy`
For `ObjLRUCache`, it always return true so that the minor of full gc on BE will prune the cache each time.
2. Reduce the default capability of file meta cache, from 20000 to 1000
Also change the default capability of hdfs file handle cache, from 20000 to 1000
4. Change judgement of whether enable file meta cache when querying
If the number of file need to be read is larger than the 1/3 of the file meta cache's capability, file meta cache
will be disabled for this query. Because cache is useless if there are too many files.
2024-03-21 14:07:22 +08:00
..
2023-12-29 10:11:44 +08:00
2022-11-18 21:02:54 +08:00
2023-04-19 23:11:48 +08:00
2024-01-12 11:59:52 +08:00
2023-11-09 18:05:34 +08:00
2023-11-23 15:18:49 +08:00
2023-04-19 23:11:48 +08:00
2023-11-28 13:02:30 +08:00
2023-10-25 15:47:51 +08:00
2022-10-20 08:56:01 +08:00
2023-03-13 11:11:59 +08:00
2023-11-28 13:02:30 +08:00
2023-03-13 11:11:59 +08:00
2023-11-09 18:05:34 +08:00
2023-03-14 20:40:37 +08:00
2023-10-20 22:08:17 +08:00
2023-11-24 10:09:31 +08:00
2023-11-28 13:02:30 +08:00
2023-12-03 09:55:52 +08:00
2023-04-19 23:11:48 +08:00
2023-11-23 15:18:49 +08:00
2023-11-15 18:24:36 +08:00
2023-04-19 23:11:48 +08:00
2023-11-09 18:05:34 +08:00
2023-06-19 15:29:42 +08:00
2023-06-17 10:39:02 +08:00
2023-04-19 23:11:48 +08:00
2023-06-05 10:48:07 +08:00
2023-04-19 23:11:48 +08:00
2023-06-13 15:13:57 +08:00
2023-04-19 23:11:48 +08:00
2023-05-08 10:42:06 +08:00
2023-11-09 18:05:34 +08:00
2023-11-09 18:05:34 +08:00
2023-11-28 13:02:30 +08:00
2023-03-08 10:42:20 +08:00
2023-04-19 23:11:48 +08:00
2023-04-19 23:11:48 +08:00
2023-04-19 23:11:48 +08:00
2024-01-12 11:53:57 +08:00
2023-11-15 21:34:49 +08:00
2023-11-22 22:08:13 +08:00
2023-10-11 21:07:35 +08:00
2024-01-16 18:37:06 +08:00
2023-12-22 12:48:37 +08:00
2023-10-26 14:09:25 +08:00
2023-07-19 15:43:14 +08:00
2023-03-29 09:00:52 +08:00
2023-04-19 23:11:48 +08:00
2023-04-19 23:11:48 +08:00
2024-03-15 17:57:58 +08:00
2024-03-15 17:57:58 +08:00
2023-04-19 23:11:48 +08:00
2023-04-19 23:11:48 +08:00
2023-11-28 13:02:30 +08:00
2023-04-19 23:11:48 +08:00
2022-10-18 13:10:13 +08:00
2023-04-19 23:11:48 +08:00
2023-09-09 00:30:11 +08:00
2023-11-29 17:05:14 +08:00
2023-12-06 00:26:34 +08:00
2023-04-19 23:11:48 +08:00
2023-11-28 13:02:30 +08:00
2024-02-23 19:03:28 +08:00
2023-08-26 00:16:00 +08:00
2023-08-26 00:16:00 +08:00
2023-04-19 23:11:48 +08:00
2023-04-19 23:11:48 +08:00
2024-03-18 11:23:01 +08:00
2023-11-28 13:02:30 +08:00
2023-11-28 13:02:30 +08:00
2023-04-19 23:11:48 +08:00
2023-06-13 09:41:22 +08:00
2024-03-14 16:05:08 +08:00
2024-03-14 16:05:08 +08:00
2023-12-15 15:57:08 +08:00
2023-06-30 14:29:21 +08:00
2024-02-01 19:00:50 +08:00
2023-09-22 12:23:43 +08:00
2023-11-28 13:02:30 +08:00
2023-11-28 13:02:30 +08:00
2023-08-25 11:40:30 +08:00
2023-11-28 13:02:30 +08:00
2024-02-29 19:51:47 +08:00
2023-04-19 23:11:48 +08:00
2023-04-19 23:11:48 +08:00
2023-12-06 23:29:46 +08:00
2023-07-29 12:56:31 +08:00
2022-11-12 16:06:15 +08:00
2023-11-28 13:02:30 +08:00
2023-08-22 18:31:40 +08:00
2024-03-15 18:04:02 +08:00
2023-12-22 12:05:20 +08:00
2023-11-28 13:02:30 +08:00
2023-12-02 20:33:54 +08:00
2023-12-02 20:33:54 +08:00
2023-02-15 15:58:43 +08:00
2023-01-20 13:33:01 +08:00
2024-01-12 11:53:57 +08:00
2023-11-28 13:02:30 +08:00
2024-01-04 18:27:27 +08:00
2023-08-25 21:24:55 +08:00
2024-03-21 14:07:22 +08:00
2024-03-21 14:07:22 +08:00
2023-11-23 15:18:49 +08:00
2023-04-19 23:11:48 +08:00
2024-01-06 21:17:22 +08:00
2023-04-19 23:11:48 +08:00
2023-04-19 23:11:48 +08:00
2023-02-03 11:28:48 +08:00
2023-07-08 14:49:44 +08:00
2023-07-08 14:49:44 +08:00
2023-11-14 18:53:04 +08:00
2023-11-14 10:30:42 +08:00
2023-11-14 10:30:42 +08:00
2023-09-29 22:38:52 +08:00
2023-04-19 23:11:48 +08:00
2023-06-12 10:00:35 +08:00
2024-03-21 14:07:22 +08:00
2023-11-12 11:57:46 +08:00
2023-09-06 15:39:19 +08:00
2023-09-06 15:39:19 +08:00
2023-11-09 18:05:34 +08:00
2023-11-12 11:57:46 +08:00
2023-04-24 14:51:51 +08:00
2024-01-31 23:53:39 +08:00
2024-01-31 23:53:39 +08:00
2023-07-20 20:48:55 +08:00
2023-06-11 14:00:04 +08:00
2024-03-16 20:55:28 +08:00
2024-03-16 20:55:28 +08:00
2023-09-20 17:21:45 +08:00
2023-09-20 17:21:45 +08:00
2023-12-06 00:26:34 +08:00
2023-04-19 23:11:48 +08:00
2023-11-28 13:02:30 +08:00
2023-11-09 18:05:34 +08:00
2023-09-27 16:14:49 +08:00
2023-10-10 18:23:07 +08:00
2023-10-10 18:23:07 +08:00
2023-10-20 22:08:17 +08:00
2024-02-05 22:23:16 +08:00
2023-04-19 23:11:48 +08:00
2023-06-28 16:49:45 +08:00
2023-11-28 13:02:30 +08:00
2023-05-11 10:17:11 +08:00
2023-11-14 10:30:42 +08:00
2023-11-28 13:02:30 +08:00
2023-09-29 22:38:52 +08:00
2023-11-28 13:02:30 +08:00
2023-04-19 23:11:48 +08:00
2023-04-19 23:11:48 +08:00
2023-09-29 22:38:52 +08:00
2023-04-19 23:11:48 +08:00
2023-08-31 19:46:38 +08:00
2023-11-28 13:02:30 +08:00
2024-02-29 08:42:35 +08:00
2024-02-29 08:42:35 +08:00
2023-10-24 02:51:43 +08:00
2023-04-19 23:11:48 +08:00
2023-04-19 23:11:48 +08:00
2023-06-06 12:03:01 +08:00
2024-02-29 12:38:03 +08:00
2023-09-21 17:26:41 +08:00
2023-07-01 12:18:23 +08:00
2023-03-13 11:11:59 +08:00
2023-12-28 11:00:38 +08:00
2023-04-19 23:11:48 +08:00
2023-11-10 10:07:02 +08:00
2024-03-14 09:20:50 +08:00
2024-03-14 09:19:40 +08:00
2024-02-18 14:45:25 +08:00
2023-04-19 23:11:48 +08:00
2023-11-23 15:18:49 +08:00
2023-04-19 23:11:48 +08:00