From dc7f7b6fb49337b8e539f908f4cabf8049b72ffd Mon Sep 17 00:00:00 2001 From: Johan Wikman Date: Wed, 17 Aug 2016 08:36:01 +0300 Subject: [PATCH] All changes 2.0.0...develop 973b983 Merge branch 'release-2.0.0' into develop 255dd23 Make spinlock functions take const argument 6e23bab Fix bitmask reallocation 338c189 Rename and clean up slavelag filter 3ea8f28 Fix possible NULL pointer dereference bfe6738 MXS-830: Add module information to logged messages 1fad962 Fix strncat usage d38997a Adjust log throttling policy 0be4e4b Add hashtable_item_strcasecmp 726100e Take hashtable convenience functions into use 5e7744a Fix typo in maxadmin.md c5778c8 Merge branch 'release-2.0.0' into develop b5762af Move from tmpnam to mkstemp d6f2c71 Add convenience functions to hashtable 359058a MXS-825: Add support for --execdir 636347c Enable runtime reconfiguration of log throttling ef9fba9 Improve log throttling documentation aef917a Implement log throttling e3a5349 Remove shardrouter.c 8051e80 Remove custom qc_sqlite allocation functions fd34d60 Initial implementation of the learning firewall a8752a8 Removed "filestem option" from example 1ef2519 Removed "filestem option" from example 0815cc8 Cleanup spinlock.h ab4dc99 Clean up hashtable.h ef2c078 Add prototypes for hashtable copy and free functions fb5cfaf Add 'log_throttling' configuration entry 300d823 Add proper prototypes for hashtable hash and cmp functions 1c649aa qc_mysqlembedded: Include skygw_...h without path. d276160 Add missing RPM scripts e70e644 Fix HTTPAuth installation 1b2b389 Combine utils into the server directory 3ff9913 Add missing utils headers to devel package 407efb2 Fix minor packaging problems 99aa6ad Split MaxScale into core, experimental and devel packages 1290386 Merge branch 'develop' of ssh://github.com/mariadb-corporation/maxscale-new into develop e59f148 Make scripts POSIX sh compatible 7319266 Fixed SHOW SLAVE STATUS in bonlog router f8d760a Update Binlogrouter.md 0a904ed Update Replication-Proxy-Binlog-Router-Tutorial.md 75d4202 Update Replication-Proxy-Binlog-Router-Tutorial.md b8651fc Add missing newline in listmanager.h c7ad047 Add note about user data caches to release notes 70ccc2b Merge branch 'release-2.0.0' into develop 575d1b6 Mistake - dummy session needs list markers set. 8364508 Merge branch 'develop' into binlog_server_semisync 868b902 Update MaxScale limitations 2c8b327 Store listener caches in separate directories 6e183ec Create unique user data caches for each listeners f643685 Don't free orphaned tee filter sessions 4179afa Allow binlogrouter to be used without a listener 7ad79af Add function for freeing a listener 677a0a2 Move authentication data from services to listeners 4f12af7 Merge remote-tracking branch 'origin/MXS-677' into develop 1419b81 Semi-Sync support to binlog server: code review updtate 0ea0f01 Semi-Sync support to binlog server: added missing routine 4aad909 Semi-Sync support to binlog server b824e1e Add authenticator support to httpd.c 705a688 Change tabs to spaces d0c419e Change method of adding list fields to e.g. DCB 25504fc Document the changed routing priority of hints 41666d1 Remove use_ssl_if_enabled global option a3584e9 Make routing hints have highest priority 34a1d24 Updated document with new binlog router option 01eedc5 Updated documentation with SSL usage 8a4c0f6 Update Replication-Proxy-Binlog-Router-Tutorial.md 4e374aa Update Replication-Proxy-Binlog-Router-Tutorial.md f3f3c57 Update Replication-Proxy-Binlog-Router-Tutorial.md 617b79f Binlog Server: error messages typo fix fa8dfae Binlog Server: error messages review 1b8819c Fix freeing of schemarouter session memory 07f49e1 MXS-788: new code review fix 1fd3b09 MXS-788: show services now displays SSL info 6ca2584 MXS-788 code review fix ae6a7d0 MXS-788 code review 43d3474 Master server SSL connection 90b2377 Use correct variable in listmanager pre-allocation 9a5b238 Fix listmanager pre-allocation 9c78625 Fix a memory leak when backend authentication fails e59a966 Fix hang in list_find_free ff30223 Fix freeing of shared data in schemarouter fc8f9d3 Add missing include in luafilter ecf7f53 Add missing NULL value to filter parameter array 636d849 Update memory allocation approach f0d1d38 Add new allocation functions 97d00a0 Fix writing of uninitialized data to logs e72c9b2 Merge branch 'release-2.0.0' into develop cf2b712 Merge branch 'release-2.0.0' into develop 8917c5c Change the logic behind valid list entry checks c10deff Improve documentation about version_string f59f1f7 Merge branch 'develop' of ssh://github.com/mariadb-corporation/maxscale-new into develop c88edb3 Backend authentication failure improvement abd5bee Revert "Backend authentication failure improvement" 5bb3107 Backend authentication failure improvement b7f434a Add new allocation functions 3f022fa Fix stupid mistake 99c4317 Merge remote-tracking branch 'origin/MXS-677' into develop 3c1ded6 Added connection/authentication failure error reporting in SHOW SLAVE STATUS 0a60f7b Tidy up and deal with review points. ba103ff blr_slave.c: Update strncpy usage 467331e blr_master.c: Strncpy usage updates d2b7c0c Merge remote-tracking branch 'origin/develop-nullauth-merge' into develop 5a8c1d0 qc: Measure execution time at the right place. bccdb93 Merge branch 'NullAuthDeny' into develop 2e6511c Add 5.5.5 prefix to all version strings that lack it 314655a Improve DCB and session initialization and list handling e1c43f0 MXS-655: Make MaxScale logging logrotate(8) compatible ce36afd MXS-626: Don't log a header unless maxlog enabled dcd47a7 blr_file.c: Replace uses of strncpy 6b8f576 bls_slave.c: Replace strncpy with memcpy 68a0039 Add list preallocation, tidy up, simplify init. cb37d1b Fix copyright etc headers. 11a400d Tidy; comment; fix bad copies and mistakes. 7e36ec4 Add list manager files. c4794e3 Initial code for list manager. 1b42e25 Merge remote-tracking branch 'origin/MXS-765' into develop d50f617 Fix problems, extend tests, respond to review. dcb4a91 Filter test folder removed 0b60dbe Add a couple of comments. 83cdba0 Fix overwriting problem. ba5d353 Fix overwriting problem. 53671cb Small fixes in response to review. 173d049 blr.c: Review strncpy usage 4ff6ef2 binlog_common.c: Replace strncpy with memcpy f238e03 maxbinlogcheck.s: Replace strncpy 9807f8d harness: Replace unnecessary use of strncpy 8c7fe6a avro: Modify strncpy usage 9b8008e Small improvements. b7f784f Fix mistakes in testqueuemanager.c cc26962 Restore missing poll.c code; add testqueuemanager.c. 2e91806 Format the filter harness 22059e6 Initial implementation connection queueing. c604dc2 readwritesplit.c: Improve COM_INIT_DB handling 454d920 schemarouter.c: Replace strncpy with strcpy 8e85d66 sharding_common.c: Too long a database name handled explicitly 77f4446 Astyle schemarouter 491f7c2 maxinfo.c: Replace strncpy with memcpy 6b98105 maxinfo: Reformat with astyle c1dbf08 Handle oversize user and database names 5fa4a0f Merge branch 'develop' of ssh://github.com/mariadb-corporation/maxscale-new into develop 706963b BLR_DBUSERS_TAIL new var in blr.h d75b9af Tweak comments, remove trailing blanks. ab2400a Optimise statistics gathering by inline & simpler fns. fb59ddc Remove unnecessary strncpy/strncat usage in Binlog Server bdcd551 resultset.c: Change strncpy to memcpy c6b1c5e Reject rather than cut too long a path 6d8f112 Remove unnecessary strncpy/strncat usage 18bf5ed Remove unnecessary strncpy usage dc0e2db Make maxpasswd more userfriendly c9c8695 Fix calculation of padded_len in encryptPassword 2cfd2c6 dbusers.c: Check strncpy usage 7ab9342 Make more thorough checks in secrets_readKeys be7d593 Format cli.c debugcli.c testroute.c webserver.c 1ee5efb config.c: Check usage of strncpy 3043b12 gq_utils.c: Unnecessary use of strncpy removed 77874ac Add help to maxkeys 38392a3 Update secrets_writeKeys documentation 2d1325c Make SSL optional in MaxScale's own communication bda00da Fix avro build failures b2cb31a Add more OOM macros 41ccf17 Fix strdup usage a48f732 Fix realloc calls 20771f6 Add forgotten extern "C" block 8faf35a Add maxscale allocation functions bb47890 Add macros for OOM logging afea388 Fix silly mistakes. 6dafd22 Make deny default for null auth; move code from common to auth. --- CMakeLists.txt | 164 +- COPYRIGHT | 2 +- Documentation/About/Limitations.md | 14 - Documentation/Documentation-Contents.md | 1 + Documentation/Filters/CCRFilter.md | 70 + Documentation/Filters/Gatekeeper.md | 53 + .../Building-MaxScale-from-Source-Code.md | 37 +- .../Getting-Started/Configuration-Guide.md | 44 +- Documentation/Reference/MaxAdmin.md | 26 +- .../MaxScale-2.1.0-Release-Notes.md | 85 + Documentation/Routers/Binlogrouter.md | 20 +- Documentation/Routers/ReadWriteSplit.md | 11 +- ...eplication-Proxy-Binlog-Router-Tutorial.md | 162 +- Documentation/check_links.sh | 13 +- LICENSE.TXT | 4 +- avro/CMakeLists.txt | 2 +- avro/maxavro.c | 2 +- avro/maxavro.h | 2 +- avro/maxavro_datablock.c | 2 +- avro/maxavro_file.c | 2 +- avro/maxavro_record.c | 2 +- avro/maxavro_schema.c | 2 +- avro/maxavro_write.c | 2 +- avro/maxavrocheck.c | 2 +- avro/test/test_values.c | 2 +- client/CMakeLists.txt | 2 +- client/maxadmin.c | 2 +- cmake/BuildMariaDBConnector.cmake | 2 - cmake/BuildPCRE2.cmake | 3 - cmake/CheckPlatform.cmake | 132 +- cmake/defaults.cmake | 9 + cmake/init_scripts.cmake | 36 + cmake/install_layout.cmake | 108 + cmake/macros.cmake | 74 +- cmake/package.cmake | 52 + cmake/package_deb.cmake | 12 +- cmake/package_rpm.cmake | 17 +- doxygate.in => etc/doxygate.in | 0 maxscale.conf.in => etc/maxscale.conf.in | 0 plugins/CMakeLists.txt | 10 +- plugins/nagios/check_maxscale_monitors.pl | 2 +- plugins/nagios/check_maxscale_resources.pl | 2 +- plugins/nagios/check_maxscale_threads.pl | 2 +- query_classifier/qc_dummy/CMakeLists.txt | 2 +- query_classifier/qc_dummy/qc_dummy.cc | 2 +- .../qc_mysqlembedded/CMakeLists.txt | 2 +- .../qc_mysqlembedded/qc_mysqlembedded.cc | 4 +- query_classifier/qc_sqlite/CMakeLists.txt | 3 +- query_classifier/qc_sqlite/qc_sqlite.c | 100 +- query_classifier/qc_sqlite/qc_sqlite3.c | 2 +- .../test/canonical_tests/canonizer.c | 2 +- query_classifier/test/classify.c | 2 +- query_classifier/test/compare.cc | 24 +- rabbitmq_consumer/CMakeLists.txt | 2 +- rabbitmq_consumer/consumer.c | 2 +- script/stacktrace | 12 +- script/unpack_rpm.sh | 32 - server/CMakeLists.txt | 1 + server/core/CMakeLists.txt | 10 +- server/core/adminusers.c | 2 +- server/core/alloc.c | 218 ++ server/core/atomic.c | 2 +- server/core/buffer.c | 80 +- server/core/config.c | 226 +- server/core/dbusers.c | 285 +- server/core/dcb.c | 502 ++- server/core/doxygen.c | 2 +- server/core/externcmd.c | 46 +- server/core/filter.c | 79 +- server/core/gateway.c | 144 +- server/core/gw_ssl.c | 2 +- server/core/gw_utils.c | 15 +- server/core/gwbitmask.c | 29 +- server/core/gwdirs.c | 21 +- server/core/hashtable.c | 149 +- server/core/hint.c | 25 +- server/core/housekeeper.c | 27 +- server/core/listener.c | 94 +- server/core/listmanager.c | 451 +++ server/core/load_utils.c | 49 +- server/core/log_manager.cc | 1166 ++++--- server/core/maxkeys.c | 69 +- server/core/maxpasswd.c | 158 +- server/core/maxscale_pcre2.c | 5 +- server/core/memlog.c | 31 +- server/core/misc.c | 2 +- server/core/mlist.c | 19 +- server/core/modutil.c | 114 +- server/core/monitor.c | 58 +- server/core/mysql_binlog.c | 2 +- server/core/mysql_utils.c | 14 +- server/core/poll.c | 58 +- server/core/query_classifier.c | 10 +- server/core/queuemanager.c | 144 +- server/core/random_jkiss.c | 2 +- server/core/resultset.c | 47 +- server/core/secrets.c | 103 +- server/core/server.c | 112 +- server/core/service.c | 322 +- server/core/session.c | 491 ++- {utils => server/core}/skygw_utils.cc | 217 +- server/core/slist.c | 22 +- server/core/spinlock.c | 13 +- server/core/statistics.c | 80 +- server/core/test/CMakeLists.txt | 9 + server/core/test/test_mysql_users.c | 44 +- server/core/test/testadminusers.c | 5 +- server/core/test/testbuffer.c | 6 +- server/core/test/testdcb.c | 2 +- server/core/test/testfeedback.c | 6 +- server/core/test/testfilter.c | 2 +- server/core/test/testgwbitmask.c | 5 +- server/core/test/testhash.c | 26 +- server/core/test/testhint.c | 7 +- server/core/test/testlog.c | 19 +- server/core/test/testlogorder.c | 7 +- server/core/test/testlogthrottling.cc | 262 ++ server/core/test/testmaxscalepcre2.c | 6 +- server/core/test/testmemlog.c | 2 +- server/core/test/testmodutil.c | 5 +- server/core/test/testpoll.c | 2 +- server/core/test/testqueuemanager.c | 251 ++ server/core/test/testserver.c | 9 +- server/core/test/testservice.c | 10 +- server/core/test/testsession.c | 2 +- server/core/test/testspinlock.c | 2 +- server/core/test/testusers.c | 2 +- server/core/thread.c | 2 +- server/core/users.c | 21 +- server/core/utils.c | 79 +- server/include/CMakeLists.txt | 7 + server/include/adminusers.h.in | 2 +- server/include/atomic.h | 2 +- server/include/buffer.h | 2 +- server/include/dbusers.h | 12 +- server/include/dcb.h | 29 +- server/include/def_monitor_event.h | 2 +- server/include/externcmd.h | 2 +- server/include/filter.h | 2 +- server/include/gw.h | 3 +- server/include/gw_authenticator.h | 2 +- server/include/gw_protocol.h | 2 +- server/include/gw_ssl.h | 2 +- server/include/gwbitmask.h | 4 +- server/include/gwdirs.h.in | 2 +- server/include/hashtable.h | 60 +- server/include/hint.h | 2 +- server/include/housekeeper.h | 2 +- server/include/listener.h | 13 +- server/include/listmanager.h | 119 + server/include/log_manager.h | 82 +- server/include/maxconfig.h | 8 +- server/include/maxscale.h | 2 +- server/include/maxscale/CMakeLists.txt | 5 + server/include/maxscale/alloc.h | 74 + server/include/maxscale/poll.h | 2 +- server/include/maxscale_pcre2.h | 2 +- server/include/memlog.h | 2 +- server/include/mlist.h | 2 +- server/include/modinfo.h | 2 +- server/include/modules.h | 2 +- server/include/modutil.h | 2 +- server/include/monitor.h | 2 +- server/include/mysql_binlog.h | 2 +- server/include/mysql_utils.h | 2 +- server/include/notification.h | 2 +- server/include/platform.h | 2 +- server/include/query_classifier.h | 2 +- server/include/queuemanager.h | 19 +- server/include/random_jkiss.h | 2 +- server/include/rdtsc.h | 2 +- server/include/resultset.h | 2 +- server/include/router.h | 2 +- server/include/secrets.h | 9 +- server/include/server.h | 2 +- server/include/service.h | 8 +- server/include/session.h | 29 +- {utils => server/include}/skygw_debug.h | 21 +- {utils => server/include}/skygw_types.h | 2 +- {utils => server/include}/skygw_utils.h | 17 +- server/include/slist.h | 2 +- server/include/spinlock.h | 17 +- server/include/statistics.h | 41 +- server/include/thread.h | 2 +- server/include/users.h | 2 +- server/include/utils.h | 2 +- ...ale_template.cnf => maxscale.cnf.template} | 0 server/modules/CMakeLists.txt | 1 + server/modules/authenticator/CMakeLists.txt | 26 +- server/modules/authenticator/cdc_plain_auth.c | 70 +- server/modules/authenticator/http_auth.c | 223 ++ server/modules/authenticator/max_admin_auth.c | 9 +- server/modules/authenticator/mysql_auth.c | 44 +- .../{null_auth.c => null_auth_allow.c} | 5 +- server/modules/authenticator/null_auth_deny.c | 154 + server/modules/filter/CMakeLists.txt | 66 +- .../filter/{slavelag.c => ccrfilter.c} | 85 +- .../modules/filter/dbfwfilter/CMakeLists.txt | 4 +- server/modules/filter/dbfwfilter/dbfwfilter.c | 171 +- server/modules/filter/dbfwfilter/dbfwfilter.h | 2 +- server/modules/filter/dbfwfilter/ruleparser.y | 2 +- server/modules/filter/dbfwfilter/token.l | 2 +- server/modules/filter/gatekeeper.c | 587 ++++ server/modules/filter/hint/CMakeLists.txt | 2 +- server/modules/filter/hint/hintfilter.c | 9 +- server/modules/filter/hint/hintparser.c | 27 +- server/modules/filter/luafilter.c | 31 +- server/modules/filter/mqfilter.c | 194 +- server/modules/filter/namedserverfilter.c | 37 +- server/modules/filter/qlafilter.c | 59 +- server/modules/filter/regexfilter.c | 45 +- server/modules/filter/tee.c | 66 +- server/modules/filter/test/CMakeLists.txt | 26 - server/modules/filter/test/README | 20 - server/modules/filter/test/doxygen.conf | 2303 -------------- .../filter/test/fwfilter/fwtest.cnf.in | 4 - .../filter/test/fwfilter/fwtest.expected | 8 - .../modules/filter/test/fwfilter/fwtest.input | 10 - .../filter/test/fwfilter/fwtest.output | 8 - .../filter/test/fwfilter/fwtest2.expected | 8 - .../filter/test/fwfilter/fwtest2.input | 10 - server/modules/filter/test/harness.cnf | 2 - server/modules/filter/test/harness.h | 371 --- server/modules/filter/test/harness_common.c | 1166 ------- server/modules/filter/test/harness_ui.c | 400 --- server/modules/filter/test/harness_util.c | 48 - .../filter/test/hintfilter/hint_testing.cnf | 3 - .../test/hintfilter/hint_testing.expected | 48 - .../filter/test/hintfilter/hint_testing.input | 48 - .../filter/test/hintfilter/hint_tests.sh | 9 - server/modules/filter/test/querysmall | 1 - .../filter/test/regexfilter/regextest.cnf | 5 - .../test/regexfilter/regextest.expected | 3 - .../filter/test/regexfilter/regextest.input | 3 - server/modules/filter/test/rules | 4 - server/modules/filter/test/tee_recursion.sh | 86 - server/modules/filter/test/tee_recursion1.cnf | 114 - server/modules/filter/test/tee_recursion2.cnf | 112 - server/modules/filter/test/testdriver.sh | 11 - server/modules/filter/testfilter.c | 9 +- server/modules/filter/topfilter.c | 67 +- server/modules/include/CMakeLists.txt | 5 + server/modules/include/avrorouter.h | 6 +- server/modules/include/binlog_common.h | 2 +- server/modules/include/blr.h | 73 +- server/modules/include/blr_constants.h | 2 +- server/modules/include/cdc.h | 2 +- server/modules/include/debugcli.h | 6 +- server/modules/include/httpd.h | 2 +- server/modules/include/maxadmin.h | 2 +- server/modules/include/maxinfo.h | 2 +- server/modules/include/maxscaled.h | 2 +- server/modules/include/mysql_auth.h | 2 +- .../include/mysql_client_server_protocol.h | 32 +- server/modules/include/mysqlhint.h | 2 +- server/modules/include/readconnection.h | 2 +- server/modules/include/readwritesplit.h | 2 +- server/modules/include/schemarouter.h | 2 +- server/modules/include/shardrouter.h | 2 +- server/modules/include/telnetd.h | 2 +- server/modules/monitor/CMakeLists.txt | 18 +- server/modules/monitor/galeramon.c | 15 +- server/modules/monitor/galeramon.h | 2 +- server/modules/monitor/mmmon.c | 15 +- server/modules/monitor/mmmon.h | 2 +- server/modules/monitor/mysql_mon.c | 20 +- server/modules/monitor/mysqlmon.h | 2 +- server/modules/monitor/ndbclustermon.c | 15 +- server/modules/monitor/ndbclustermon.h | 2 +- server/modules/protocol/CMakeLists.txt | 14 +- server/modules/protocol/cdc.c | 15 +- server/modules/protocol/httpd.c | 50 +- server/modules/protocol/maxscaled.c | 11 +- server/modules/protocol/mysql_backend.c | 54 +- server/modules/protocol/mysql_client.c | 141 +- server/modules/protocol/mysql_common.c | 45 +- server/modules/protocol/telnetd.c | 13 +- server/modules/protocol/testprotocol.c | 4 +- server/modules/routing/CMakeLists.txt | 8 +- server/modules/routing/avro/CMakeLists.txt | 12 +- server/modules/routing/avro/avro.c | 101 +- server/modules/routing/avro/avro_client.c | 31 +- server/modules/routing/avro/avro_file.c | 103 +- server/modules/routing/avro/avro_index.c | 2 +- server/modules/routing/avro/avro_rbr.c | 5 +- server/modules/routing/avro/avro_schema.c | 142 +- server/modules/routing/avro/cdc | 2 +- .../modules/routing/avro/cdc_kafka_producer | 2 +- .../modules/routing/avro/cdc_last_transaction | 2 +- server/modules/routing/avro/cdc_schema.go | 2 +- server/modules/routing/avro/cdc_users | 2 +- server/modules/routing/binlog/CMakeLists.txt | 4 +- server/modules/routing/binlog/binlog_common.c | 4 +- server/modules/routing/binlog/blr.c | 553 +++- server/modules/routing/binlog/blr_cache.c | 2 +- server/modules/routing/binlog/blr_file.c | 175 +- server/modules/routing/binlog/blr_master.c | 488 ++- server/modules/routing/binlog/blr_slave.c | 868 +++-- .../modules/routing/binlog/maxbinlogcheck.c | 196 +- .../modules/routing/binlog/test/testbinlog.c | 25 +- server/modules/routing/cli.c | 261 +- server/modules/routing/debugcli.c | 307 +- server/modules/routing/debugcmd.c | 136 +- server/modules/routing/maxinfo/CMakeLists.txt | 2 +- server/modules/routing/maxinfo/maxinfo.c | 887 +++--- .../modules/routing/maxinfo/maxinfo_error.c | 103 +- server/modules/routing/maxinfo/maxinfo_exec.c | 773 ++--- .../modules/routing/maxinfo/maxinfo_parse.c | 430 +-- server/modules/routing/readconnroute.c | 25 +- .../routing/readwritesplit/CMakeLists.txt | 2 +- .../routing/readwritesplit/readwritesplit.c | 261 +- .../readwritesplit/test/CMakeLists.txt | 7 - .../routing/readwritesplit/test/rwsplit.sh | 307 -- .../test/select_for_var_set.sql | 5 - .../test/set_autocommit_disabled.sql | 7 - .../test/test_after_autocommit_disabled.sql | 2 - .../test/test_autocommit_disabled1.sql | 7 - .../test/test_autocommit_disabled1b.sql | 7 - .../test/test_autocommit_disabled2.sql | 9 - .../test/test_autocommit_disabled3.sql | 9 - .../test/test_hints/CMakeLists.txt | 3 - .../test/test_hints/complex_tests | 48 - .../test/test_hints/error_tests | 39 - .../readwritesplit/test/test_hints/hints.txt | 9 - .../test/test_hints/rwsplit_hints.sh | 75 - .../test/test_hints/simple_tests | 18 - .../test/test_hints/stack_tests | 50 - .../test/test_hints/syntax_check.sh | 33 - .../test/test_implicit_commit1.sql | 8 - .../test/test_implicit_commit2.sql | 15 - .../test/test_implicit_commit3.sql | 9 - .../test/test_implicit_commit4.sql | 9 - .../test/test_implicit_commit5.sql | 14 - .../test/test_implicit_commit6.sql | 11 - .../test/test_implicit_commit7.sql | 10 - .../readwritesplit/test/test_sescmd.sql | 4 - .../readwritesplit/test/test_sescmd2.sql | 20 - .../readwritesplit/test/test_sescmd3.sql | 16 - .../test/test_temporary_table.sql | 5 - .../test/test_transaction_routing1.sql | 6 - .../test/test_transaction_routing2.sql | 11 - .../test/test_transaction_routing2b.sql | 11 - .../test/test_transaction_routing3.sql | 7 - .../test/test_transaction_routing3b.sql | 7 - .../test/test_transaction_routing4.sql | 9 - .../test/test_transaction_routing4b.sql | 9 - .../routing/schemarouter/CMakeLists.txt | 10 +- .../routing/schemarouter/schemarouter.c | 530 ++-- .../routing/schemarouter/sharding_common.c | 136 +- .../routing/schemarouter/shardrouter.c | 2811 ----------------- server/modules/routing/schemarouter/svcconn.c | 52 +- .../routing/schemarouter/test/CMakeLists.txt | 9 - .../routing/schemarouter/test/portblock.sh | 9 - .../schemarouter/test/prepare_shard.sh | 19 - .../routing/schemarouter/test/test.cmake.in | 17 - .../schemarouter/test/testschemarouter.c | 210 -- .../schemarouter/test/testschemarouter2.c | 326 -- server/modules/routing/testroute.c | 115 +- server/modules/routing/webserver.c | 558 ++-- utils/CMakeLists.txt | 5 - 360 files changed, 11641 insertions(+), 15856 deletions(-) create mode 100644 Documentation/Filters/CCRFilter.md create mode 100644 Documentation/Filters/Gatekeeper.md create mode 100644 Documentation/Release-Notes/MaxScale-2.1.0-Release-Notes.md create mode 100644 cmake/init_scripts.cmake create mode 100644 cmake/package.cmake rename doxygate.in => etc/doxygate.in (100%) rename maxscale.conf.in => etc/maxscale.conf.in (100%) delete mode 100755 script/unpack_rpm.sh create mode 100644 server/core/alloc.c create mode 100644 server/core/listmanager.c rename {utils => server/core}/skygw_utils.cc (88%) create mode 100644 server/core/test/testlogthrottling.cc create mode 100644 server/core/test/testqueuemanager.c create mode 100644 server/include/CMakeLists.txt create mode 100644 server/include/listmanager.h create mode 100644 server/include/maxscale/CMakeLists.txt create mode 100644 server/include/maxscale/alloc.h rename {utils => server/include}/skygw_debug.h (98%) rename {utils => server/include}/skygw_types.h (97%) rename {utils => server/include}/skygw_utils.h (93%) rename server/{maxscale_template.cnf => maxscale.cnf.template} (100%) create mode 100644 server/modules/authenticator/http_auth.c rename server/modules/authenticator/{null_auth.c => null_auth_allow.c} (97%) create mode 100644 server/modules/authenticator/null_auth_deny.c rename server/modules/filter/{slavelag.c => ccrfilter.c} (82%) create mode 100644 server/modules/filter/gatekeeper.c delete mode 100644 server/modules/filter/test/CMakeLists.txt delete mode 100644 server/modules/filter/test/README delete mode 100644 server/modules/filter/test/doxygen.conf delete mode 100644 server/modules/filter/test/fwfilter/fwtest.cnf.in delete mode 100644 server/modules/filter/test/fwfilter/fwtest.expected delete mode 100644 server/modules/filter/test/fwfilter/fwtest.input delete mode 100644 server/modules/filter/test/fwfilter/fwtest.output delete mode 100644 server/modules/filter/test/fwfilter/fwtest2.expected delete mode 100644 server/modules/filter/test/fwfilter/fwtest2.input delete mode 100644 server/modules/filter/test/harness.cnf delete mode 100644 server/modules/filter/test/harness.h delete mode 100644 server/modules/filter/test/harness_common.c delete mode 100644 server/modules/filter/test/harness_ui.c delete mode 100644 server/modules/filter/test/harness_util.c delete mode 100644 server/modules/filter/test/hintfilter/hint_testing.cnf delete mode 100644 server/modules/filter/test/hintfilter/hint_testing.expected delete mode 100644 server/modules/filter/test/hintfilter/hint_testing.input delete mode 100755 server/modules/filter/test/hintfilter/hint_tests.sh delete mode 100644 server/modules/filter/test/querysmall delete mode 100644 server/modules/filter/test/regexfilter/regextest.cnf delete mode 100644 server/modules/filter/test/regexfilter/regextest.expected delete mode 100644 server/modules/filter/test/regexfilter/regextest.input delete mode 100644 server/modules/filter/test/rules delete mode 100755 server/modules/filter/test/tee_recursion.sh delete mode 100644 server/modules/filter/test/tee_recursion1.cnf delete mode 100644 server/modules/filter/test/tee_recursion2.cnf delete mode 100755 server/modules/filter/test/testdriver.sh create mode 100644 server/modules/include/CMakeLists.txt delete mode 100644 server/modules/routing/readwritesplit/test/CMakeLists.txt delete mode 100755 server/modules/routing/readwritesplit/test/rwsplit.sh delete mode 100644 server/modules/routing/readwritesplit/test/select_for_var_set.sql delete mode 100644 server/modules/routing/readwritesplit/test/set_autocommit_disabled.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_after_autocommit_disabled.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_autocommit_disabled1.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_autocommit_disabled1b.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_autocommit_disabled2.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_autocommit_disabled3.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_hints/CMakeLists.txt delete mode 100644 server/modules/routing/readwritesplit/test/test_hints/complex_tests delete mode 100644 server/modules/routing/readwritesplit/test/test_hints/error_tests delete mode 100644 server/modules/routing/readwritesplit/test/test_hints/hints.txt delete mode 100755 server/modules/routing/readwritesplit/test/test_hints/rwsplit_hints.sh delete mode 100644 server/modules/routing/readwritesplit/test/test_hints/simple_tests delete mode 100644 server/modules/routing/readwritesplit/test/test_hints/stack_tests delete mode 100755 server/modules/routing/readwritesplit/test/test_hints/syntax_check.sh delete mode 100644 server/modules/routing/readwritesplit/test/test_implicit_commit1.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_implicit_commit2.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_implicit_commit3.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_implicit_commit4.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_implicit_commit5.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_implicit_commit6.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_implicit_commit7.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_sescmd.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_sescmd2.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_sescmd3.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_temporary_table.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_transaction_routing1.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_transaction_routing2.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_transaction_routing2b.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_transaction_routing3.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_transaction_routing3b.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_transaction_routing4.sql delete mode 100644 server/modules/routing/readwritesplit/test/test_transaction_routing4b.sql delete mode 100644 server/modules/routing/schemarouter/shardrouter.c delete mode 100644 server/modules/routing/schemarouter/test/CMakeLists.txt delete mode 100755 server/modules/routing/schemarouter/test/portblock.sh delete mode 100755 server/modules/routing/schemarouter/test/prepare_shard.sh delete mode 100644 server/modules/routing/schemarouter/test/test.cmake.in delete mode 100644 server/modules/routing/schemarouter/test/testschemarouter.c delete mode 100644 server/modules/routing/schemarouter/test/testschemarouter2.c delete mode 100644 utils/CMakeLists.txt diff --git a/CMakeLists.txt b/CMakeLists.txt index e3762c85c..39dd205ba 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -17,6 +17,7 @@ endif() # Set default values for cache entries and set the MaxScale version include(cmake/defaults.cmake) include(VERSION.cmake) +include(ExternalProject) set(CMAKE_BUILD_TYPE "RelWithDebInfo" CACHE STRING "Choose the type of build, options are: None(CMAKE_CXX_FLAGS or CMAKE_C_FLAGS used) Debug Release RelWithDebInfo MinSizeRel.") set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake") @@ -26,10 +27,9 @@ project(MaxScale) # Set the installation layout include(${CMAKE_SOURCE_DIR}/cmake/install_layout.cmake) -#Do the platform check -include(cmake/CheckPlatform.cmake) +# Do the platform check +include(cmake/CheckPlatform.cmake) -check_deps() check_dirs() find_package(OpenSSL) find_package(Valgrind) @@ -38,7 +38,7 @@ find_package(Pandoc) find_package(TCMalloc) find_package(Jemalloc) find_package(Git) -find_package(CURL) +find_package(CURL) find_package(RabbitMQ) find_package(LibUUID) find_package(Avro) @@ -171,7 +171,6 @@ set(CMAKE_CXX_FLAGS_DEBUG "${DEBUG_FLAGS} -DSS_DEBUG -DLOG_ASSERT") set(CMAKE_CXX_FLAGS_RELEASE "") set(CMAKE_CXX_FLAGS_RELWITHDEBINFO "-ggdb") -include_directories(utils) include_directories(avro) include_directories(server/include) include_directories(server/inih) @@ -190,108 +189,62 @@ if(NOT WITHOUT_MAXADMIN) endif() # Generate text versions of some documents -execute_process(COMMAND perl ${CMAKE_SOURCE_DIR}/Documentation/format.pl +execute_process(COMMAND perl ${CMAKE_SOURCE_DIR}/Documentation/format.pl ${CMAKE_SOURCE_DIR}/Documentation/Changelog.md ${CMAKE_BINARY_DIR}/Changelog.txt) -execute_process(COMMAND perl ${CMAKE_SOURCE_DIR}/Documentation/format.pl +execute_process(COMMAND perl ${CMAKE_SOURCE_DIR}/Documentation/format.pl ${CMAKE_SOURCE_DIR}/Documentation/Release-Notes/MaxScale-1.2.0-Release-Notes.md ${CMAKE_BINARY_DIR}/ReleaseNotes.txt) -execute_process(COMMAND perl ${CMAKE_SOURCE_DIR}/Documentation/format.pl +execute_process(COMMAND perl ${CMAKE_SOURCE_DIR}/Documentation/format.pl ${CMAKE_SOURCE_DIR}/Documentation/Upgrading/Upgrading-To-MaxScale-1.2.md ${CMAKE_BINARY_DIR}/UpgradingToMaxScale12.txt) -install(FILES ${CMAKE_BINARY_DIR}/Changelog.txt DESTINATION ${MAXSCALE_SHAREDIR}) -install(FILES ${CMAKE_BINARY_DIR}/ReleaseNotes.txt DESTINATION ${MAXSCALE_SHAREDIR}) -install(FILES ${CMAKE_BINARY_DIR}/UpgradingToMaxScale12.txt DESTINATION ${MAXSCALE_SHAREDIR}) -install(FILES server/maxscale_template.cnf DESTINATION ${MAXSCALE_SHAREDIR}) -if(WITH_MAXSCALE_CNF) - install(FILES server/maxscale_template.cnf DESTINATION ${MAXSCALE_CONFDIR} RENAME maxscale.cnf.template) +install_file(${CMAKE_BINARY_DIR}/Changelog.txt core) +install_file(${CMAKE_BINARY_DIR}/ReleaseNotes.txt core) +install_file(${CMAKE_BINARY_DIR}/UpgradingToMaxScale12.txt core) +install_file(server/maxscale.cnf.template core) +install_file(server/maxscale_binlogserver_template.cnf core) + +# Install the template into /etc +if(WITH_MAXSCALE_CNF AND (NOT TARGET_COMPONENT OR "core" STREQUAL "${TARGET_COMPONENT}")) + install_custom_file(server/maxscale.cnf.template ${MAXSCALE_CONFDIR} core) endif() -install(FILES server/maxscale_binlogserver_template.cnf DESTINATION ${MAXSCALE_SHAREDIR}) -install(PROGRAMS ${ERRMSG} DESTINATION ${MAXSCALE_VARDIR}/lib/maxscale) -install(FILES ${CMAKE_SOURCE_DIR}/COPYRIGHT DESTINATION ${MAXSCALE_SHAREDIR}) -install(FILES ${CMAKE_SOURCE_DIR}/README DESTINATION ${MAXSCALE_SHAREDIR}) -install(FILES ${CMAKE_SOURCE_DIR}/LICENSE.TXT DESTINATION ${MAXSCALE_SHAREDIR}) -install(FILES etc/lsyncd_example.conf DESTINATION ${MAXSCALE_SHAREDIR}) -install(FILES Documentation/maxscale.1 DESTINATION ${CMAKE_INSTALL_DATADIR}/man/man1) + +# This should be moved to the qc_mysqlembedded directory +# install(PROGRAMS ${ERRMSG} DESTINATION ${MAXSCALE_VARDIR}/lib/maxscale) + +install_file(${CMAKE_SOURCE_DIR}/README core) +install_file(${CMAKE_SOURCE_DIR}/LICENSE.TXT core) +install_file(etc/lsyncd_example.conf core) +install_manual(Documentation/maxscale.1 1 core) # Install startup scripts and ldconfig files if(WITH_SCRIPTS) - configure_file(${CMAKE_SOURCE_DIR}/maxscale.conf.in ${CMAKE_BINARY_DIR}/maxscale.conf @ONLY) - configure_file(${CMAKE_SOURCE_DIR}/etc/maxscale.service.in ${CMAKE_BINARY_DIR}/maxscale.service @ONLY) - if(DEB_BASED) - configure_file(${CMAKE_SOURCE_DIR}/etc/ubuntu/init.d/maxscale.in ${CMAKE_BINARY_DIR}/maxscale @ONLY) - else() - configure_file(${CMAKE_SOURCE_DIR}/etc/init.d/maxscale.in ${CMAKE_BINARY_DIR}/maxscale @ONLY) - endif() - if(PACKAGE) - message(STATUS "maxscale.conf will unpack to: /etc/ld.so.conf.d") - message(STATUS "startup scripts will unpack to to: /etc/init.d") - message(STATUS "systemd service files will unpack to to: /usr/lib/systemd/system") - install(PROGRAMS ${CMAKE_BINARY_DIR}/maxscale DESTINATION ${MAXSCALE_SHAREDIR} ) - install(FILES ${CMAKE_BINARY_DIR}/maxscale.conf DESTINATION ${MAXSCALE_SHAREDIR}) - install(FILES ${CMAKE_BINARY_DIR}/maxscale.service DESTINATION ${MAXSCALE_SHAREDIR}) - else() - install(PROGRAMS ${CMAKE_BINARY_DIR}/maxscale DESTINATION /etc/init.d) - install(FILES ${CMAKE_BINARY_DIR}/maxscale.conf DESTINATION /etc/ld.so.conf.d) - install(FILES ${CMAKE_BINARY_DIR}/maxscale.service DESTINATION /usr/lib/systemd/system) - message(STATUS "Installing maxscale.conf to: /etc/ld.so.conf.d") - message(STATUS "Installing startup scripts to: /etc/init.d") - message(STATUS "Installing systemd service files to: /usr/lib/systemd/system") - endif() + include(cmake/init_scripts.cmake) endif() # Only do packaging if configured if(PACKAGE) - execute_process(COMMAND uname -m COMMAND tr -d '\n' OUTPUT_VARIABLE CPACK_PACKAGE_ARCHITECTURE) # Install the files copied by the postinst script into the share folder - install(PROGRAMS ${CMAKE_BINARY_DIR}/maxscale DESTINATION ${MAXSCALE_SHAREDIR}) - install(FILES ${CMAKE_BINARY_DIR}/maxscale.conf DESTINATION ${MAXSCALE_SHAREDIR}) - install(PROGRAMS ${CMAKE_BINARY_DIR}/postinst DESTINATION ${MAXSCALE_SHAREDIR}) - install(PROGRAMS ${CMAKE_BINARY_DIR}/postrm DESTINATION ${MAXSCALE_SHAREDIR}) - if(${CMAKE_VERSION} VERSION_LESS 2.8.12) - message(WARNING "CMake version is ${CMAKE_VERSION}. Building of packages requires version 2.8.12 or greater.") - else() + install_program(${CMAKE_BINARY_DIR}/maxscale core) + install_file(${CMAKE_BINARY_DIR}/maxscale.conf core) + install_program(${CMAKE_BINARY_DIR}/postinst core) + install_program(${CMAKE_BINARY_DIR}/postrm core) - # Generic CPack configuration variables - SET(CPACK_SET_DESTDIR ON) - set(CPACK_STRIP_FILES FALSE) - set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "MaxScale") - set(CPACK_PACKAGE_VERSION_MAJOR "${MAXSCALE_VERSION_MAJOR}") - set(CPACK_PACKAGE_VERSION_MINOR "${MAXSCALE_VERSION_MINOR}") - set(CPACK_PACKAGE_VERSION_PATCH "${MAXSCALE_VERSION_PATCH}") - set(CPACK_PACKAGE_CONTACT "MariaDB Corporation Ab") - if(DISTRIB_SUFFIX) - set(CPACK_PACKAGE_FILE_NAME "maxscale-${MAXSCALE_VERSION}-${MAXSCALE_BUILD_NUMBER}.${DISTRIB_SUFFIX}.${CPACK_PACKAGE_ARCHITECTURE}") - else() - set(CPACK_PACKAGE_FILE_NAME "maxscale-${MAXSCALE_VERSION}-${MAXSCALE_BUILD_NUMBER}.${CPACK_PACKAGE_ARCHITECTURE}") - endif() - set(CPACK_PACKAGE_NAME "maxscale") - set(CPACK_PACKAGE_VENDOR "MariaDB Corporation Ab") - set(CPACK_PACKAGE_DESCRIPTION_FILE ${CMAKE_SOURCE_DIR}/etc/DESCRIPTION) - set(CPACK_PACKAGING_INSTALL_PREFIX "${CMAKE_INSTALL_PREFIX}") + # Include the package configuration + include(cmake/package.cmake) - # See if we are on a RPM-capable or DEB-capable system - find_program(RPMBUILD rpmbuild) - find_program(DEBBUILD dpkg-buildpackage) - set(CPACK_GENERATOR "TGZ") - - if(NOT ( ${RPMBUILD} STREQUAL "RPMBUILD-NOTFOUND" ) ) - include(cmake/package_rpm.cmake) - message(STATUS "Generating RPM packages") - endif() - if(NOT ( ${DEBBUILD} STREQUAL "DEBBUILD-NOTFOUND" ) ) - include(cmake/package_deb.cmake) - message(STATUS "Generating DEB packages for ${DEB_ARCHITECTURE}") - endif() - - include(CPack) - endif() + # CPack needs to be included after everything is configured + include(CPack) endif() +# +# Custom targets for MaxScale +# + # uninstall target # see http://www.cmake.org/Wiki/CMake_FAQ#Can_I_do_.22make_uninstall.22_with_CMake.3F configure_file( @@ -306,54 +259,37 @@ find_package(Doxygen) if(DOXYGEN_FOUND) configure_file( - "${CMAKE_CURRENT_SOURCE_DIR}/doxygate.in" + "${CMAKE_CURRENT_SOURCE_DIR}/etc/doxygate.in" "${CMAKE_CURRENT_BINARY_DIR}/doxygate" IMMEDIATE @ONLY) add_custom_target(documentation COMMAND ${DOXYGEN_EXECUTABLE} ${CMAKE_CURRENT_BINARY_DIR}/doxygate) - -endif() - -# Testall target with Valgrind -if(VALGRIND_FOUND) -add_custom_target(testall-valgrind - COMMAND ${CMAKE_COMMAND} -DBUILD_TESTS=Y -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_DIR} -DWITH_SCRIPTS=N ${CMAKE_SOURCE_DIR} - COMMAND make install - COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_SOURCE_DIR}/server/test/MaxScale_test.cnf ${CMAKE_BINARY_DIR}/etc/MaxScale.cnf - COMMAND /bin/sh -c "valgrind --track-fds=yes --leak-check=full --show-leak-kinds=all --log-file=${CMAKE_BINARY_DIR}/valgrind.log ${CMAKE_BINARY_DIR}/bin/maxscale -c ${CMAKE_BINARY_DIR} &>/dev/null" - COMMAND /bin/sh -c "make test || echo \"Test results written to: ${CMAKE_BINARY_DIR}/Testing/Temporary/\"" - COMMAND killall maxscale - COMMENT "Running full test suite with Valgrind..." VERBATIM) - endif() +# Generates PDF documentation add_custom_target(generate_pdf COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_SOURCE_DIR}/Documentation ${CMAKE_BINARY_DIR}/Documentation - COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_DIR}/Documentation ${CMAKE_COMMAND} + COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_DIR}/Documentation ${CMAKE_COMMAND} -DBUILD_DIR=${CMAKE_BINARY_DIR} - -DCMAKE_MODULE_PATH=${CMAKE_MODULE_PATH} + -DCMAKE_MODULE_PATH=${CMAKE_MODULE_PATH} -P generate-pdf.cmake - COMMENT "Generating PDF files" VERBATIM) + COMMENT "Generating PDF files" VERBATIM) +# Creates text versions of the release notes add_custom_target(generate_txt_release COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_SOURCE_DIR}/Documentation ${CMAKE_BINARY_DIR}/Documentation - COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_DIR}/Documentation ${CMAKE_COMMAND} + COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_DIR}/Documentation ${CMAKE_COMMAND} -DBUILD_DIR=${CMAKE_BINARY_DIR} - -DCMAKE_MODULE_PATH=${CMAKE_MODULE_PATH} + -DCMAKE_MODULE_PATH=${CMAKE_MODULE_PATH} -P generate-txt-release.cmake - COMMENT "Generating TXT release notes" VERBATIM) - + COMMENT "Generating TXT release notes" VERBATIM) +# Generates HTML documentation add_custom_target(generate_html COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_SOURCE_DIR}/Documentation ${CMAKE_BINARY_DIR}/Documentation - COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_DIR}/Documentation ${CMAKE_COMMAND} + COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_DIR}/Documentation ${CMAKE_COMMAND} -DBUILD_DIR=${CMAKE_BINARY_DIR} - -DCMAKE_MODULE_PATH=${CMAKE_MODULE_PATH} + -DCMAKE_MODULE_PATH=${CMAKE_MODULE_PATH} -P generate-html.cmake - COMMENT "Generating HTML files" VERBATIM) - -if(PACKAGE) - message(STATUS "You can install startup scripts and system configuration files for MaxScale by running the 'postinst' shell script located at ${CMAKE_INSTALL_PREFIX}.") - message(STATUS "To remove these installed files, run the 'postrm' shell script located in the same folder.") -endif() + COMMENT "Generating HTML files" VERBATIM) diff --git a/COPYRIGHT b/COPYRIGHT index 87fcf6625..0c1b7b88e 100644 --- a/COPYRIGHT +++ b/COPYRIGHT @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/Documentation/About/Limitations.md b/Documentation/About/Limitations.md index 19066bb52..ac05d17f0 100644 --- a/Documentation/About/Limitations.md +++ b/Documentation/About/Limitations.md @@ -12,20 +12,6 @@ limitations to the components which illustrate them. This section describes the limitations that are common to all configuration of plugins with MariaDB MaxScale. -### Crash if one of several listeners for a Service fails as startup - -If a service has multiple listeners and one of those listeners fails -at startup, MariaDB MaxScale will crash. - -A typical reason for a listener to fail is that it has been configured -with a non-existing socket path or a port that MariaDB MaxScale is not allowed -to use. - -Workaround: Ensure that socket paths and ports are valid. - -Issues [MXS-710](https://jira.mariadb.org/browse/MXS-710) and -[MXS-711](https://jira.mariadb.org/browse/MXS-711) relate to this. - ## Limitations with MySQL Protocol support Compression is not included in MySQL server handshake diff --git a/Documentation/Documentation-Contents.md b/Documentation/Documentation-Contents.md index 993c347cd..4904c92a5 100644 --- a/Documentation/Documentation-Contents.md +++ b/Documentation/Documentation-Contents.md @@ -84,6 +84,7 @@ Here are detailed documents about the filters MariaDB MaxScale offers. They cont - [Database Firewall Filter](Filters/Database-Firewall-Filter.md) - [RabbitMQ Filter](Filters/RabbitMQ-Filter.md) - [Named Server Filter](Filters/Named-Server-Filter.md) + - [Gatekeeper - Adaptive Firewall](Filters/Gatekeeper.md) ## Monitors diff --git a/Documentation/Filters/CCRFilter.md b/Documentation/Filters/CCRFilter.md new file mode 100644 index 000000000..f2ea452d8 --- /dev/null +++ b/Documentation/Filters/CCRFilter.md @@ -0,0 +1,70 @@ +# Consistent Critical Read Filter + +## Overview + +The Consistent Critical Read (CCR) filter allows consistent critical reads to be +done through MaxScale while still allowing scaleout of non-critical reads. + +When the filter detects a statement that would modify the database, it attaches a +routing hint to all following statements. This routing hint guides the routing +module to route the statement to the master server where data is guaranteed to be +in a up-to-date state. + +## Filter Options + +The CCR filter accepts the following options. + +|Option |Description | +|-----------|--------------------------------------------| +|ignorecase |Use case-insensitive matching (default) | +|case |Use case-sensitive matching | +|extended |Use extended regular expression syntax (ERE)| + +To use multiple filter options, list them in a comma-separated list. + +``` +options=case,extended +``` + +## Filter Parameters + +The CCR filter has no mandatory parameters. + +### `time` + +The time window in seconds during which queries are routed to the master. The +default value for this parameter is 60 seconds. + +When a data modifying SQL statement is processed, a timer is set to the value of +_time_. Once the timer has elapsed, all statements are routed normally. If a new +data modifying SQL statement is processed within the time window, the timer is +reset to the value of _time_. + +Enabling this parameter in combination with the _count_ parameter causes both the +time window and number of queries to be inspected. If either of the two +conditions are met, the query is re-routed to the master. + +### `count` + +The number of SQL statements to route to master after detecting a data modifying +SQL statement. This feature is disabled by default. + +After processing a data modifying SQL statement, a counter is set to the value of +_count_ and all statements are routed to the master. Each executed statement +after a data modifying SQL statement cause the counter to be decremented. Once +the counter reaches zero, the statements are routed normally. If a new data +modifying SQL statement is processed, the counter is reset to the value of +_count_. + +### `match` + +An optional parameter that can be used to control which statements trigger the +statement re-routing. The parameter value is a regular expression that is used to +match against the SQL text. Only non-SELECT statements are inspected. + +### `ignore` + +An optional parameter that can be used to control which statements don't trigger +the statement re-routing. This does the opposite of the _match_ parameter. The +parameter value is a regular expression that is used to match against the SQL +text. Only non-SELECT statements are inspected. diff --git a/Documentation/Filters/Gatekeeper.md b/Documentation/Filters/Gatekeeper.md new file mode 100644 index 000000000..dee55ef6f --- /dev/null +++ b/Documentation/Filters/Gatekeeper.md @@ -0,0 +1,53 @@ +# Gatekeeper Filter + +# Overview + +This filter will learn from input data read during a learning phase. After +learning the characteristics of the input, the filter can then be set into +the enforcing mode. In this mode the filter will block any queries that do +not conform to the training set. This is a simple yet effective way to prevent +unwanted queries from being executed in the database. + +When the filter receives a query in the learning mode, the filter converts it +into a canonicalized form i.e. creates a query digest. The query digest is then +stored for later use. + +When the filter is started in enforcing mode, the list of stored query digests +is used as a master list of allowed queries. The filter calculates a query +digest for all incoming queries and compares it to the master list. If the +query digest is in the master list, the query is executed normally. If it +isn't found from the list, the query is not executed, an error is returned +to the client and a warning message is logged. + +# Configuration + +## Filter Parameters + +### `mode` + +This is a mandatory parameter for the filter which controls the filter's operation. + +Accepted values are _learn_ and _enforce_. _learn_ sets the filter into the +learning mode, where the query digests are calculated and stored at the end of +each session. _enforce_ sets the filter into the enforcement mode where incoming +queries are compared to the previously calculated list of query digests. + +### `datadir` + +This is an optional parameter which controls where the calculated query digests +are stored. This parameter takes an absolute path to a directory as its argument. + +## Example Configuration + +Here is an example configuration of the filter with both the mandatory _mode_ +and the optional _datadir_ parameters defined. The filter is in learning mode +and after collecting enough samples, the filter is should be set into enforcing +mode. The filter stores the gathered query digests in `/var/lib/maxscale/gatekeeper/`. + +``` +[Smart Firewall] +type=filter +module=gatekeeper +mode=learn +datadir=/var/lib/maxscale/gatekeeper/ +``` diff --git a/Documentation/Getting-Started/Building-MaxScale-from-Source-Code.md b/Documentation/Getting-Started/Building-MaxScale-from-Source-Code.md index c2a9a409e..f459ce017 100644 --- a/Documentation/Getting-Started/Building-MaxScale-from-Source-Code.md +++ b/Documentation/Getting-Started/Building-MaxScale-from-Source-Code.md @@ -141,14 +141,13 @@ which allows us to build packages. The `-DCMAKE_INSTALL_PREFIX` was removed sinc we aren't installing MariaDB MaxScale, only packaging it. ``` -cmake ../MaxScale -DPACKAGE=Y -DBUILD_TESTS=Y +cmake ../MaxScale -DPACKAGE=Y ``` -Next step is to test and build the package. +Next step is to build the package. ``` make -make test make package ``` @@ -163,3 +162,35 @@ the LD_LIBRARY_PATH environment variable. make LD_LIBRARY_PATH=$PWD/server/core/ make package ``` + +## Installing and packaging optional components + +MaxScale is split into multiple components. The main component is the core MaxScale +package which contains MaxScale and all the modules. This is the default component +that is build, installed and packaged. There exist two other components, the _experimental_ +and the _devel_ components. The former contains all experimental modules which are +not considered as part of the core MaxScale package and they can be alpha or beta +quality modules. The latter of the optional components, _devel_, contains the +development files required for MaxScale module development. + +The component which is build is controlled by the TARGET_COMPONENT CMake variable. +The default value for this is _core_ which build the core MaxScale package. + +To build the experimental modules, you will need to set value of the TARGET_COMPONENT +CMake variable to the component name you wish to install or package. + +For example, to package the experimental module component, invoke CMake with +_-DTARGET_COMPONENT=experimental_: + +``` +cmake ../MaxScale -DPACKAGE=Y DTARGET_COMPONENT=experimental +make package +``` + +If you wish to create a monolithic package with all the components, set the value to an +empty string and build the package: + +``` +cmake ../MaxScale -DPACKAGE=Y DTARGET_COMPONENT="" +make package +``` diff --git a/Documentation/Getting-Started/Configuration-Guide.md b/Documentation/Getting-Started/Configuration-Guide.md index 8cd7c211a..274fccb8c 100644 --- a/Documentation/Getting-Started/Configuration-Guide.md +++ b/Documentation/Getting-Started/Configuration-Guide.md @@ -242,6 +242,41 @@ log_augmentation=1 To disable the augmentation use the value 0 and to enable it use the value 1. +#### `log_throttling` + +In some circumstances it is possible that a particular error (or warning) is logged +over and over again, if the cause for the error persistently remains. To prevent the log +from flooding, it is possible to specify how many times a particular error may be logged +within a time period, before the logging of that error is suppressed for a while. + +``` +# A valid value looks like +# log_throttling = X, Y, Z +# +# where each value is a positive integer and X means the number of times a specific +# error may be logged within a time period of Y milliseconds, before the logging of +# that error is suppressed for Z milliseconds. +log_throttling=8, 2000, 15000 +``` + +In the example above, the logging of a particular error will be suppressed for 15 seconds +if the error has been logged 8 times in 2 seconds. + +The default is `10, 1000, 10000`, which means that if the same error is logged 10 +times in one second, the logging of that error is suppressed for the following +10 seconds. + +To disable log throttling, add an entry with an empty value +``` +log_throttling= +``` +or one where any of the integers is 0. +``` +log_throttling=0, 0, 0 +``` + +Note that *notice*, *info* and *debug* messages are never throttled. + #### `logdir` Set the directory where the logfiles are stored. The folder needs to be both readable and writable by the user running MariaDB MaxScale. @@ -475,7 +510,8 @@ This parameter enables matching of "127.0.0.1" (localhost) against "%" wildcard #### `version_string` -This parameter sets a custom version string that is sent in the MySQL Handshake from MariaDB MaxScale to clients. +This parameter sets a custom version string that is sent in the MySQL Handshake +from MariaDB MaxScale to clients. Example: @@ -483,7 +519,11 @@ Example: version_string=5.5.37-MariaDB-RWsplit ``` -If not set, the default value is the server version of the embedded MySQL/MariaDB library. Example: 5.5.35-MariaDB +If not set, the default value is `5.5.5-10.0.0 MaxScale ` +where `` is the version of MaxScale. If the provided string +does not start with the number 5, a 5.5.5- prefix will be added to it. This +means that a _version_string_ value of _MaxScale-Service_ would result in a +_5.5.5-MaxScale-Service_ being sent to the client. #### `weightby` diff --git a/Documentation/Reference/MaxAdmin.md b/Documentation/Reference/MaxAdmin.md index 55ab1da2b..e341e8ce1 100644 --- a/Documentation/Reference/MaxAdmin.md +++ b/Documentation/Reference/MaxAdmin.md @@ -807,7 +807,14 @@ MariaDB MaxScale can log messages to syslog, to a log file or to both. The appro ## Rotating the log file -MariaDB MaxScale logs messages to a log file in the log directory of MariaDB MaxScale. As the log file grows continuously, it is recommended to periodically rotate it. When rotated, the current log file will be closed and a new one with a new name opened. The log file name contain a sequence number, which is incremented each time the log is rotated. +MariaDB MaxScale logs messages to a log file in the log directory of MariaDB MaxScale. As the log file grows continuously, it is recommended to periodically rotate it. When rotated, the current log file will be closed and a new one with the *same* name opened. + +To retain the ealier log entries, you need to first rename the log file and then instruct MaxScale to rotate it. + + $ mv maxscale.log maxscale1.log + $ # MaxScale continues to write to maxscale1.log + $ kill -SIGUSR1 + $ # MaxScale closes the file (i.e. maxscale1.log) and reopens maxscale.log There are two ways for rotating the log - *flush log maxscale* and *flush logs* - and the result is identical. The two alternatives are due to historical reasons; earlier MariaDB MaxScale had several different log files. @@ -827,6 +834,23 @@ From version 1.3 onwards, MariaDB MaxScale has a single log file where messages Please note that changes made via this interface will not persist across restarts of MariaDB MaxScale. To make a permanent change edit the maxscale.cnf file. +## Adjusting the Log Throttling + +From 2.0 onwards, MariaDB MaxScale will throttle messages that are logged too frequently, which typically is a sign that MaxScale encounters some error that just keeps on repeating. The aim is to prevent the log from flooding. The configuration specifies how many times a particular error may be logged during a period of a specified length, before it is suppressed for a period of a specified other length. + +The current log throttling configuration can be queried with + + MaxScale> show log_throttling + 10 1000 100000 + +where the numbers are the count, the length (in milliseconds) of the period during which the counting is made, and the length (in milliseconds) of the period the message is subsequently suppressed. + +The configuration can be set with + + MaxScale> set log_throttling 10 1000 10000 + +where numbers are specified in the same order as in the *show* case. Setting any of the values to 0, disables the throttling. + ## Reloading The Configuration A command, _reload config_, is available that will cause MariaDB MaxScale to reload the maxscale.cnf configuration file. diff --git a/Documentation/Release-Notes/MaxScale-2.1.0-Release-Notes.md b/Documentation/Release-Notes/MaxScale-2.1.0-Release-Notes.md new file mode 100644 index 000000000..b18c74c6c --- /dev/null +++ b/Documentation/Release-Notes/MaxScale-2.1.0-Release-Notes.md @@ -0,0 +1,85 @@ +# MariaDB MaxScale 2.1.0 Release Notes + +Release 2.1.0 is a Beta release. + +This document describes the changes in release 2.1.0, when compared to +release 2.0.X. + +For any problems you encounter, please consider submitting a bug +report at [Jira](https://jira.mariadb.org). + +## Changes Features + +### Logging + +Before version 2.1.0, MaxScale created in the log directory a log file +maxscaleN.log, where N initially was 1 and then was increased every time +MaxScale was instructued (by sending the signal SIGUSR1 or via maxadmin) +to rotate the log file. + +That has now been changed so that the name of the log file is *always* +maxscale.log and when MaxScale is instructed to rotate the log file, +MaxScale simply closes it and then reopens and truncates it. + +To retain the existing log entries, you should first move the file to +another name (MaxScale continues writing to it) and then instruct +MaxScale to rotate the the log file. + +``` + $ mv maxscale.log maxscale1.log + $ # MaxScale continues to write to maxscale1.log + $ kill -SIGUSR1 + $ # MaxScale closes the file (i.e. maxscale1.log) and reopens maxscale.log +``` + +This behaviour is now compatible with logrotate(8). + +Further, if MaxScale is configured to use shared memory for the log file, +the file is created into the directory "/dev/shm/maxscale". Earlier the +log file was created into the directory "/dev/shm/maxscale.PID", where PID +was the pid of the MaxScale process. + +In addition, there is now a mechanism that prevents the flooding of the log, in +case the same error occurs over and over again. That mechanism, which is enabled +by default, is configured using the new global configuration entry `log_throttling`. +For more information about this configuration entry, please see +[Global Settings](../Getting-Started/Configuration-Guide.md#global-settings). + +### User data cache + +The user data loaded from the backend databases is now stored on a per listener +basis instead of a per service basis. In earlier versions, each service had its own +cache directory in `/var/cache/maxscale`. This directory contains cached user +data which is used there is no connectivity to the backend cluster. + +In 2.1.0, each listener has its own sub-directory in the service cache +directory. The old caches in `/var/cache/maxscale` will need to be manually +removed if they are no longer used by older versions of MaxScale. + +## New Features + + +## Bug fixes + +[Here is a list of bugs fixed since the release of MaxScale 2.0.X.](https://jira.mariadb.org/browse/MXS-739?jql=project%20%3D%20MXS%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20in%20(Fixed%2C%20Done)%20AND%20fixVersion%20%3D%202.0.0) + + +## Known Issues and Limitations + +There are some limitations and known issues within this version of MaxScale. +For more information, please refer to the [Limitations](../About/Limitations.md) document. + +## Packaging + +RPM and Debian packages are provided for the Linux distributions supported +by MariaDB Enterprise. + +Packages can be downloaded [here](https://mariadb.com/resources/downloads). + +## Source Code + +The source code of MaxScale is tagged at GitHub with a tag, which is identical +with the version of MaxScale. For instance, the tag of version X.Y.Z of MaxScale +is X.Y.Z. Further, *master* always refers to the latest released non-beta version. + +The source code is available [here](https://github.com/mariadb-corporation/maxscale-bsl). diff --git a/Documentation/Routers/Binlogrouter.md b/Documentation/Routers/Binlogrouter.md index 0669e84d5..7af0bb2ef 100644 --- a/Documentation/Routers/Binlogrouter.md +++ b/Documentation/Routers/Binlogrouter.md @@ -91,6 +91,21 @@ The default value is off, set transaction_safety=on to enable the incomplete tra This defines whether (on | off) MariaDB MaxScale sends to the slave the heartbeat packet when there are no real binlog events to send. The default value if 'off', no heartbeat event is sent to slave server. If value is 'on' the interval value (requested by the slave during registration) is reported in the diagnostic output and the packet is send after the time interval without any event to send. +### `semisync` + +This parameter controls whether binlog server could ask Master server to start the Semi-Synchronous replication. +In order to get semi-sync working the Master server must have the *rpl_semi_sync_master* plugin installed. +The available plugin and the value of GLOBAL VARIABLE *rpl_semi_sync_master_enabled* are checked in the Master registration phase: if plugin is installed in the Master database the binlog server then requests the semi-sync option. +Note: + - the network replication stream from Master has two additional bytes before each binlog event. + - the Semi-Sync protocol requires an acknoledge packet to be sent back to Master only when requested: the semi-sync flag will have value of 1. + This flag is set only if *rpl_semi_sync_master_enabled=1* in the Master, otherwise it will always have value of 0 and no ack packet is sent back. + +Please note that semi-sync replication is only related to binlog server to Master communication. +### `ssl_cert_verification_depth` + +This parameter sets the maximum length of the certificate authority chain that will be accepted. Legal values are positive integers. This applies to SSL connection to master server that could be acivated either by writing options in master.ini or later via CHANGE MASTER TO. This parameter cannot be modified at runtime, default is 9. + A complete example of a service entry for a binlog router service would be as follows. ``` [Replication] @@ -100,7 +115,7 @@ A complete example of a service entry for a binlog router service would be as fo version_string=5.6.17-log user=maxscale passwd=Mhu87p2D - router_options=uuid=f12fcb7f-b97b-11e3-bc5e-0401152c4c22,server-id=3,user=repl,password=slavepass,master-id=1,heartbeat=30,binlogdir=/var/binlogs,transaction_safety=1,master_version=5.6.19-common,master_hostname=common_server,master_uuid=xxx-fff-cccc-common,master-id=999,mariadb10-compatibility=1,send_slave_heartbeat=1 + router_options=uuid=f12fcb7f-b97b-11e3-bc5e-0401152c4c22,server-id=3,user=repl,password=slavepass,master-id=1,heartbeat=30,binlogdir=/var/binlogs,transaction_safety=1,master_version=5.6.19-common,master_hostname=common_server,master_uuid=xxx-fff-cccc-common,master-id=999,mariadb10-compatibility=1,send_slave_heartbeat=1,ssl_cert_verification_depth=9,semisync=1 ``` The minimum set of router options that must be given in the configuration are are server-id and master-id, default values may be used for all other options. @@ -108,3 +123,6 @@ The minimum set of router options that must be given in the configuration are ar ## Examples The [Replication Proxy](../Tutorials/Replication-Proxy-Binlog-Router-Tutorial.md) tutorial will show you how to configure and administrate a binlogrouter installation. + + +Tutorial also includes SSL communication setup to the master server and SSL client connections setup to MaxScale Binlog Server diff --git a/Documentation/Routers/ReadWriteSplit.md b/Documentation/Routers/ReadWriteSplit.md index d59a7c49f..30e4003b4 100644 --- a/Documentation/Routers/ReadWriteSplit.md +++ b/Documentation/Routers/ReadWriteSplit.md @@ -4,7 +4,7 @@ This document provides a short overview of the **readwritesplit** router module ## Overview -The **readwritesplit** router is designed to increase the read-only processing capability of a cluster while maintaining consistency. This is achieved by splitting the query load into read and write queries. Read queries, which do not modify data, are spread across multiple nodes while all write queries will be sent to a single node. +The **readwritesplit** router is designed to increase the read-only processing capability of a cluster while maintaining consistency. This is achieved by splitting the query load into read and write queries. Read queries, which do not modify data, are spread across multiple nodes while all write queries will be sent to a single node. The router is designed to be used with a traditional Master-Slave replication cluster. It automatically detects changes in the master server and will use the current master server of the cluster. With a Galera cluster, one can achieve a resilient setup and easy master failover by using one of the Galera nodes as a Write-Master node, where all write queries are routed, and spreading the read load over all the nodes. @@ -150,7 +150,14 @@ reconnecting to MariaDB MaxScale once a new master is available. ## Routing hints -The readwritesplit router supports routing hints. For a detailed guide on hint syntax and functionality, please read [this](../Reference/Hint-Syntax.md) document. +The readwritesplit router supports routing hints. For a detailed guide on hint +syntax and functionality, please read [this](../Reference/Hint-Syntax.md) document. + +**Note**: Routing hints will always have the highest priority when a routing +decision is made. This means that it is possible to cause inconsistencies in +the session state and the actual data in the database by adding routing hints +to DDL/DML statements which are then directed to slave servers. Only use routing +hints when you are sure that they can cause no harm. ## Limitations diff --git a/Documentation/Tutorials/Replication-Proxy-Binlog-Router-Tutorial.md b/Documentation/Tutorials/Replication-Proxy-Binlog-Router-Tutorial.md index 0004b0b19..f61035ec2 100644 --- a/Documentation/Tutorials/Replication-Proxy-Binlog-Router-Tutorial.md +++ b/Documentation/Tutorials/Replication-Proxy-Binlog-Router-Tutorial.md @@ -1,5 +1,5 @@ # MariaDB MaxScale as a Binlog Server -MariaDB MaxScale is a database proxy that sits between a database layer and the clients of that database, the binlog router described here is somewhat different to that original concept, moving MariaDB MaxScale down to play a role within the database layer itself. +MariaDB MaxScale is a dynamic data routing platform that sits between a database layer and the clients of that database, the binlog router described here is somewhat different to that original concept, moving MariaDB MaxScale down to play a role within the database layer itself. In a traditional MySQL replication setup a single master server is created and a set of slaves MySQL instances are configured to pull the binlog files from that master to the slaves. There are some problems, however, in this setup; when the number of slaves grows an increasing load is placed on the master, to serve the binlogs to each slave. When the master server fails, some action must be performed on every slave server before a new server can become the master server. @@ -127,6 +127,25 @@ When MariaDB MaxScale starts an error message may appear if current binlog file During normal operations binlog events are not distributed to the slaves until a *COMMIT* is seen. The default value is off, set *transaction_safety=on* to enable the incomplete transactions detection. +### semisync + +This parameter controls whether binlog server could ask Master server to start the Semi-Synchronous replication. +In order to get semi-sync working the Master server must have the *rpl_semi_sync_master* plugin installed. +The available plugin and the value of GLOBAL VARIABLE *rpl_semi_sync_master_enabled* are checked in the Master registration phase: if plugin is installed in the Master database the binlog server then requests the semi-sync option. +Note: + - the network replication stream from Master has two additional bytes before each binlog event. + - the Semi-Sync protocol requires an acknoledge packet to be sent back to Master only when requested: the semi-sync flag will have value of 1. + This flag is set only if *rpl_semi_sync_master_enabled=1* in the Master, otherwise it will always have value of 0 and no ack packet is sent back. + +Please note that semi-sync replication is only related to binlog server to Master communication. + + +### ssl_cert_verification_depth + +This parameter sets the maximum length of the certificate authority chain that will be accepted. Legal values are positive integers. +This applies to SSL connection to master server that could be acivated either by writing options in master.ini or later via CHANGE MASTER TO. +This parameter cannot be modified at runtime, default is 9. + A complete example of a service entry for a binlog router service would be as follows. ``` @@ -137,7 +156,7 @@ A complete example of a service entry for a binlog router service would be as fo version_string=5.6.17-log user=maxscale passwd=Mhu87p2D - router_options=uuid=f12fcb7f-b97b-11e3-bc5e-0401152c4c22,server-id=3,user=repl,password=slavepass,master-id=1,heartbeat=30,binlogdir=/var/binlogs,transaction_safety=1,master_version=5.6.19-common,master_hostname=common_server,master_uuid=xxx-fff-cccc-common,master-id=999,mariadb10-compatibility=1 + router_options=uuid=f12fcb7f-b97b-11e3-bc5e-0401152c4c22,server-id=3,user=repl,password=slavepass,master-id=1,heartbeat=30,binlogdir=/var/binlogs,transaction_safety=1,master_version=5.6.19-common,master_hostname=common_server,master_uuid=xxx-fff-cccc-common,master-id=999,mariadb10-compatibility=1,ssl_cert_verification_depth=9,semisync=1 ``` The minimum set of router options that must be given in the configuration are are *server-id* and *master-id*, default values may be used for all other options. @@ -154,6 +173,21 @@ As per any service in MariaDB MaxScale a listener section is required to define The protocol used by slaves for connection to MariaDB MaxScale is the same *MySQLClient* protocol that is used for client applications to connect to databases, therefore the same MariaDB MaxScale protocol module can be used. +It's also possible to enable SSL from clients (MySQL clients or Slave servers) by adding SSL options in the listener, or in a new one: +``` + [Replication Listener_SSL] + type=listener + service=Replication + protocol=MySQLClient + port=5309 + ssl=required + ssl_key=/path_to/key.pem + ssl_cert=/path_to/cert.pem + ssl_ca_cert=/path_to/ca-cert.pem + #ssl_version=TLSv10 +``` +Check the [Configuration-Guide](../Getting-Started/Configuration-Guide.md) for SSL options details. + # MariaDB MaxScale replication diagnostics The binlog router module of MariaDB MaxScale produces diagnostic output that can be viewed via the `maxadmin` client application. Running the maxadmin command and issuing a show service command will produce a considerable amount of output that will show both the master connection status and statistics and also a block for each of the slaves currently connected. @@ -232,7 +266,67 @@ The binlog router module of MariaDB MaxScale produces diagnostic output that can Users data: 0x156c030 Total connections: 2 Currently connected: 2 --bash-4.1$ +``` +If a slave is connected to MaxScale with SSL, an entry will be present in the Slave report: +``` + Slaves: + Server-id: 106 + Hostname: SBslave6 + Slave UUID: 00019686-7777-7777-7777-777777777777 + Slave_host_port: 188.165.213.5:40365 + Username: massi + Slave DCB: 0x7fc01be3ba88 + Slave connected with SSL: Established + +``` + +Another Binlog Server diagnostic output comes from SHOW SLAVE STATUS MySQL command + +``` +MySQL [(none)]> show slave status\G +*************************** 1. row *************************** + Slave_IO_State: Binlog Dump + Master_Host: 88.26.197.94 + Master_User: repl + Master_Port: 3306 + Connect_Retry: 60 + Master_Log_File: mysql-bin.003140 + Read_Master_Log_Pos: 16682679 + Relay_Log_File: mysql-bin.003140 + Relay_Log_Pos: 16682679 + Relay_Master_Log_File: mysql-bin.003140 + Slave_IO_Running: Yes + Slave_SQL_Running: Yes + Replicate_Do_DB: + Replicate_Ignore_DB: + Replicate_Do_Table: + Replicate_Ignore_Table: + Replicate_Wild_Do_Table: + Replicate_Wild_Ignore_Table: + Last_Errno: 0 + Last_Error: + Skip_Counter: 0 + Exec_Master_Log_Pos: 16682679 + Relay_Log_Space: 16682679 + Until_Condition: None + Until_Log_File: + Until_Log_Pos: 0 + Master_SSL_Allowed: Yes + Master_SSL_CA_File: /home/maxscale/packages/certificates/client/ca.pem + Master_SSL_CA_Path: + Master_SSL_Cert: /home/maxscale/packages/certificates/client/client-cert.pem + Master_SSL_Cipher: + Master_SSL_Key: /home/maxscale/packages/certificates/client/client-key.pem + Seconds_Behind_Master: 0 +Master_SSL_Verify_Server_Cert: No + Last_IO_Errno: 0 + Last_IO_Error: + Last_SQL_Errno: 0 + Last_SQL_Error: + Replicate_Ignore_Server_Ids: + Master_Server_Id: 1111 + Master_UUID: 6aae714e-b975-11e3-bc33-0401152c3d01 + Master_Info_File: /home/maxscale/binlog/first/binlogs/master.ini ``` # Binlog router compatibility @@ -261,6 +355,12 @@ Please note that is such condition the only user for MySQL protocol connection t master_user=repl master_password=somepass filestem=repl-bin + # Master SSL communication options + master_ssl=0 + master_ssl_key=/home/mpinto/packages/certificates/client/client-key.pem + master_ssl_cert=/home/mpinto/packages/certificates/client/client-cert.pem + master_ssl_ca=/home/mpinto/packages/certificates/client/ca.pem + #master_tls_version=TLSv12 Enabling replication from a master server requires: @@ -306,6 +406,15 @@ The supported options are: MASTER_LOG_FILE MASTER_LOG_POS + and SSL options as well: + MASTER_SSL (0|1) + MASTER_SSL_CERT (path to certificate file) + MASTER_SSL_KEY (path to key file) + MASTER_SSL_CA (path to CA cerificate file) + MASTER_TLS_VERSION (allowed level of encryption used) + +Further details about level of encryption or certificates could be found here [Configuration Guuide](../Getting-Started/Configuration-Guide.md) + There are some constraints related to *MASTER_LOG_FILE* and *MASTER_LOG_POS*: *MASTER_LOG_FILE* could be changed to next binlog in sequence with *MASTER_LOG_POS=4* or to current one at current position. @@ -348,6 +457,53 @@ This command removes *master.ini* file, blanks all master configuration in memor Note: existing binlog files are not touched by this command. +Examples with SSL options: + + MySQL [(none)]> CHANGE MASTER TO MASTER_SSL = 1, MASTER_SSL_CERT='/home/maxscale/packages/certificates/client/client-cert.pem', MASTER_SSL_CA='/home/maxscale/packages/certificates/client/ca.pem', MASTER_SSL_KEY='/home/maxscale/packages/certificates/client/client-key.pem', MASTER_TLS_VERSION='TLSv12'; + + MySQL [(none)]> CHANGE MASTER TO MASTER_TLS_VERSION='TLSv12'; + + MySQL [(none)]> CHANGE MASTER TO MASTER_SSL = 0; + + +#### Some constraints: + - In order to enable/re-enable Master SSL comunication the MASTER_SSL=1 option is required and all certificate options must be explicitey set in the same CHANGE MASTER TO command. + - New certificate options changes take effect after maxScale restart or after MASTER_SSL=1 with the new options. + +Note: + - SHOW SLAVE STATUS displays all the options but MASTER_TLS_VERSION value. + - Maxadmin, 'show services' or 'show service $binlog_service' displays all the options when SSL is on. + - STOP SLAVE is required for CHANGE MASTER TO command (any option) + - START SLAVE will use new SSL options for Master SSL communication setup. + +Examples: + mysql client + + MySQL> SHOW SLAVE STATUS\G + + Master_SSL_Allowed: Yes + Master_SSL_CA_File: /home/mpinto/packages/certificates/client/ca.pem + Master_SSL_CA_Path: + Master_SSL_Cert: /home/mpinto/packages/certificates/client/client-cert.pem + Master_SSL_Cipher: + Master_SSL_Key: /home/mpinto/packages/certificates/client/client-key.pem + + maxadmin client + + MaxScale>'show service BinlogServer' + + Service: BinlogServer + Router: binlogrouter (0x7fd8c3002b40) + State: Started + Master connection DCB: 0x7fd8c8fce228 + Master SSL is ON: + Master SSL CA cert: /home/mpinto/packages/certificates/client/ca.pem + Master SSL Cert: /home/mpinto/packages/certificates/client/client-cert.pem + Master SSL Key: /home/mpinto/packages/certificates/client/client-key.pem + Master SSL tls_ver: TLSv12 + + + ### Slave servers setup Examples of *CHANGE MASTER TO* command issued on a slave server that wants to gets replication events from MariaDB MaxScale binlog router: diff --git a/Documentation/check_links.sh b/Documentation/check_links.sh index 4e6a1ed4a..e93064c8f 100755 --- a/Documentation/check_links.sh +++ b/Documentation/check_links.sh @@ -1,28 +1,27 @@ +#!/bin/bash # # Copyright (c) 2016 MariaDB Corporation Ab # # Use of this software is governed by the Business Source License included # in the LICENSE.TXT file and at www.mariadb.com/bsl. # -# Change Date: 2019-01-01 +# Change Date: 2019-07-01 # # On the date above, in accordance with the Business Source License, use # of this software will be governed by version 2 or later of the General # Public License. # -#!/bin/bash - # Check that all links to Markdown format files point to existing local files homedir=$(pwd) -for file in $(find . -name '*.md') +find . -name '*.md'|while read file do - cd "$(dirname $file)" - for i in `grep -o '\[.*\]([^#].*[.]md)' $(basename $file)| sed -e 's/\[.*\](\(.*\))/\1/'` + cd "$(dirname "$file")" + grep -o '\[.*\]([^#].*[.]md)' "$(basename "$file")"| sed -e 's/\[.*\](\(.*\))/\1/'|while read i do - if [ ! -f $i ] + if [ ! -f "$i" ] then echo "Link $i in $file is not correct!" fi diff --git a/LICENSE.TXT b/LICENSE.TXT index 21734b80a..f395a2e82 100644 --- a/LICENSE.TXT +++ b/LICENSE.TXT @@ -7,7 +7,7 @@ Use Limitation: Usage of the software is free when your application uses the Software with a total of less than three database server instances for production purposes. -Change Date: 2019-01-01 +Change Date: 2019-07-01 Change License: Version 2 or later of the GNU General Public License as published by the Free Software Foundation. @@ -17,7 +17,7 @@ please visit: https://mariadb.com/products/mariadb-enterprise -------------------------------------------------------------------------- -Business Source License 0.9 +Business Source License 1.0 You are granted limited license to the Software under this Business Source License. Please read this Business Source License carefully, particularly diff --git a/avro/CMakeLists.txt b/avro/CMakeLists.txt index 2019a98d9..d76785788 100644 --- a/avro/CMakeLists.txt +++ b/avro/CMakeLists.txt @@ -4,5 +4,5 @@ target_link_libraries(maxavro maxscale-common jansson) add_executable(maxavrocheck maxavrocheck.c) target_link_libraries(maxavrocheck maxavro) -install(TARGETS maxavrocheck DESTINATION ${MAXSCALE_BINDIR}) +install_executable(maxavrocheck core) add_subdirectory(test) diff --git a/avro/maxavro.c b/avro/maxavro.c index 380788c19..251257fbb 100644 --- a/avro/maxavro.c +++ b/avro/maxavro.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/avro/maxavro.h b/avro/maxavro.h index f6b976544..715eb1ee5 100644 --- a/avro/maxavro.h +++ b/avro/maxavro.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/avro/maxavro_datablock.c b/avro/maxavro_datablock.c index 2c1cc359e..60d0d3c48 100644 --- a/avro/maxavro_datablock.c +++ b/avro/maxavro_datablock.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/avro/maxavro_file.c b/avro/maxavro_file.c index 6b8ac80b1..457664333 100644 --- a/avro/maxavro_file.c +++ b/avro/maxavro_file.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/avro/maxavro_record.c b/avro/maxavro_record.c index 82e05f873..a5e0e06f9 100644 --- a/avro/maxavro_record.c +++ b/avro/maxavro_record.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/avro/maxavro_schema.c b/avro/maxavro_schema.c index d26ea75e6..56c877b9b 100644 --- a/avro/maxavro_schema.c +++ b/avro/maxavro_schema.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/avro/maxavro_write.c b/avro/maxavro_write.c index 29574a58d..389fb06d7 100644 --- a/avro/maxavro_write.c +++ b/avro/maxavro_write.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/avro/maxavrocheck.c b/avro/maxavrocheck.c index 1c1940471..b3c639720 100644 --- a/avro/maxavrocheck.c +++ b/avro/maxavrocheck.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/avro/test/test_values.c b/avro/test/test_values.c index f279cdb4b..7a5c0752e 100644 --- a/avro/test/test_values.c +++ b/avro/test/test_values.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/client/CMakeLists.txt b/client/CMakeLists.txt index 32ab702ea..99e9e4a4f 100644 --- a/client/CMakeLists.txt +++ b/client/CMakeLists.txt @@ -7,4 +7,4 @@ if(HIST) else() message(STATUS "Could not find editline library. MaxAdmin will be built without it.") endif() -install(TARGETS maxadmin DESTINATION ${MAXSCALE_BINDIR}) +install_executable(maxadmin core) diff --git a/client/maxadmin.c b/client/maxadmin.c index cb4375b6c..65d26d6aa 100644 --- a/client/maxadmin.c +++ b/client/maxadmin.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/cmake/BuildMariaDBConnector.cmake b/cmake/BuildMariaDBConnector.cmake index 4b09b499e..9707dae2c 100644 --- a/cmake/BuildMariaDBConnector.cmake +++ b/cmake/BuildMariaDBConnector.cmake @@ -5,8 +5,6 @@ # sets the variables set by FindMariaDBConnector.cmake so that it appears that # the system has the connector. -include(ExternalProject) - set(MARIADB_CONNECTOR_C_REPO "https://github.com/MariaDB/mariadb-connector-c.git" CACHE STRING "MariaDB Connector-C Git repository") diff --git a/cmake/BuildPCRE2.cmake b/cmake/BuildPCRE2.cmake index dcf3840e8..57216730d 100644 --- a/cmake/BuildPCRE2.cmake +++ b/cmake/BuildPCRE2.cmake @@ -5,9 +5,6 @@ # need to add a dependeny on the 'pcre2' target by adding add_dependencies( pcre2) # to the CMakeLists.txt. You don't need to link against the pcre2 library # because the static symbols will be in MaxScale. - -include(ExternalProject) - ExternalProject_Add(pcre2 SOURCE_DIR ${CMAKE_SOURCE_DIR}/pcre2/ CMAKE_ARGS -DCMAKE_C_FLAGS=-fPIC -DBUILD_SHARED_LIBS=N -DPCRE2_BUILD_PCRE2GREP=N -DPCRE2_BUILD_TESTS=N BINARY_DIR ${CMAKE_BINARY_DIR}/pcre2/ diff --git a/cmake/CheckPlatform.cmake b/cmake/CheckPlatform.cmake index d1bf2e950..a35bd4339 100644 --- a/cmake/CheckPlatform.cmake +++ b/cmake/CheckPlatform.cmake @@ -1,46 +1,92 @@ #Checks for all the C system headers found in all the files - include(CheckFunctionExists) - include(CheckLibraryExists) - include(CheckIncludeFiles) +include(CheckFunctionExists) +include(CheckLibraryExists) +include(CheckIncludeFiles) - check_include_files(arpa/inet.h HAVE_ARPA_INET) - check_include_files(crypt.h HAVE_CRYPT) - check_include_files(ctype.h HAVE_CTYPE) - check_include_files(dirent.h HAVE_DIRENT) - check_include_files(dlfcn.h HAVE_DLFCN) - check_include_files(errno.h HAVE_ERRNO) - check_include_files(execinfo.h HAVE_EXECINFO) - check_include_files(fcntl.h HAVE_FCNTL) - check_include_files(ftw.h HAVE_FTW) - check_include_files(getopt.h HAVE_GETOPT) - check_include_files(ini.h HAVE_INI) - check_include_files(math.h HAVE_MATH) - check_include_files(memlog.h HAVE_MEMLOG) - check_include_files(netdb.h HAVE_NETDB) - check_include_files(netinet/in.h HAVE_NETINET_IN) - check_include_files(openssl/aes.h HAVE_OPENSSL_AES) - check_include_files(openssl/sha.h HAVE_OPENSSL_SHA) - check_include_files(pthread.h HAVE_PTHREAD) - check_include_files(pwd.h HAVE_PWD) - check_include_files(rdtsc.h HAVE_RDTSC) - check_include_files(regex.h HAVE_REGEX) - check_include_files(signal.h HAVE_SIGNAL) - check_include_files(stdarg.h HAVE_STDARG) - check_include_files(stdbool.h HAVE_STDBOOL) - check_include_files(stdint.h HAVE_STDINT) - check_include_files(stdio.h HAVE_STDIO) - check_include_files(stdlib.h HAVE_STDLIB) - check_include_files(string.h HAVE_STRING) - check_include_files(strings.h HAVE_STRINGS) - check_include_files(sys/epoll.h HAVE_SYS_EPOLL) - check_include_files(sys/ioctl.h HAVE_SYS_IOCTL) - check_include_files(syslog.h HAVE_SYSLOG) - check_include_files(sys/param.h HAVE_SYS_PARAM) - check_include_files(sys/socket.h HAVE_SYS_SOCKET) - check_include_files(sys/stat.h HAVE_SYS_STAT) - check_include_files(sys/time.h HAVE_SYS_TIME) - check_include_files(sys/types.h HAVE_SYS_TYPES) - check_include_files(sys/un.h HAVE_SYS_UN) - check_include_files(time.h HAVE_TIME) - check_include_files(unistd.h HAVE_UNISTD) +check_include_files(arpa/inet.h HAVE_ARPA_INET) +check_include_files(crypt.h HAVE_CRYPT) +check_include_files(ctype.h HAVE_CTYPE) +check_include_files(dirent.h HAVE_DIRENT) +check_include_files(dlfcn.h HAVE_DLFCN) +check_include_files(errno.h HAVE_ERRNO) +check_include_files(execinfo.h HAVE_EXECINFO) +check_include_files(fcntl.h HAVE_FCNTL) +check_include_files(ftw.h HAVE_FTW) +check_include_files(getopt.h HAVE_GETOPT) +check_include_files(ini.h HAVE_INI) +check_include_files(math.h HAVE_MATH) +check_include_files(memlog.h HAVE_MEMLOG) +check_include_files(netdb.h HAVE_NETDB) +check_include_files(netinet/in.h HAVE_NETINET_IN) +check_include_files(openssl/aes.h HAVE_OPENSSL_AES) +check_include_files(openssl/sha.h HAVE_OPENSSL_SHA) +check_include_files(pthread.h HAVE_PTHREAD) +check_include_files(pwd.h HAVE_PWD) +check_include_files(rdtsc.h HAVE_RDTSC) +check_include_files(regex.h HAVE_REGEX) +check_include_files(signal.h HAVE_SIGNAL) +check_include_files(stdarg.h HAVE_STDARG) +check_include_files(stdbool.h HAVE_STDBOOL) +check_include_files(stdint.h HAVE_STDINT) +check_include_files(stdio.h HAVE_STDIO) +check_include_files(stdlib.h HAVE_STDLIB) +check_include_files(string.h HAVE_STRING) +check_include_files(strings.h HAVE_STRINGS) +check_include_files(sys/epoll.h HAVE_SYS_EPOLL) +check_include_files(sys/ioctl.h HAVE_SYS_IOCTL) +check_include_files(syslog.h HAVE_SYSLOG) +check_include_files(sys/param.h HAVE_SYS_PARAM) +check_include_files(sys/socket.h HAVE_SYS_SOCKET) +check_include_files(sys/stat.h HAVE_SYS_STAT) +check_include_files(sys/time.h HAVE_SYS_TIME) +check_include_files(sys/types.h HAVE_SYS_TYPES) +check_include_files(sys/un.h HAVE_SYS_UN) +check_include_files(time.h HAVE_TIME) +check_include_files(unistd.h HAVE_UNISTD) + +# Check for libraries MaxScale depends on +find_library(HAVE_LIBAIO NAMES aio) +if(NOT HAVE_LIBAIO) + message(FATAL_ERROR "Could not find libaio") +endif() + +find_library(HAVE_LIBSSL NAMES ssl) +if(NOT HAVE_LIBSSL) + message(FATAL_ERROR "Could not find libssl") +endif() + +find_library(HAVE_LIBCRYPT NAMES crypt) +if(NOT HAVE_LIBCRYPT) + message(FATAL_ERROR "Could not find libcrypt") +endif() + +find_library(HAVE_LIBCRYPTO NAMES crypto) +if(NOT HAVE_LIBCRYPTO) + message(FATAL_ERROR "Could not find libcrypto") +endif() + +find_library(HAVE_LIBZ NAMES z) +if(NOT HAVE_LIBZ) + message(FATAL_ERROR "Could not find libz") +endif() + +find_library(HAVE_LIBM NAMES m) +if(NOT HAVE_LIBM) + message(FATAL_ERROR "Could not find libm") +endif() + +find_library(HAVE_LIBDL NAMES dl) +if(NOT HAVE_LIBDL) + message(FATAL_ERROR "Could not find libdl") +endif() + +find_library(HAVE_LIBRT NAMES rt) +if(NOT HAVE_LIBRT) + message(FATAL_ERROR "Could not find librt") +endif() + +find_library(HAVE_LIBPTHREAD NAMES pthread) +if(NOT HAVE_LIBPTHREAD) + message(FATAL_ERROR "Could not find libpthread") +endif() diff --git a/cmake/defaults.cmake b/cmake/defaults.cmake index 8da43b593..aa58ff536 100644 --- a/cmake/defaults.cmake +++ b/cmake/defaults.cmake @@ -53,3 +53,12 @@ set(WITH_TCMALLOC FALSE CACHE BOOL "Use tcmalloc as the memory allocator") # Use jemalloc as the memory allocator set(WITH_JEMALLOC FALSE CACHE BOOL "Use jemalloc as the memory allocator") + +# Install experimental modules +set(INSTALL_EXPERIMENTAL TRUE CACHE BOOL "Install experimental modules") + +# Default package name +set(PACKAGE_NAME "maxscale" CACHE STRING "Name of the generated package") + +# Which component to build (core, experimental, devel) +set(TARGET_COMPONENT "core" CACHE STRING "Which component to build (core, experimental, devel)") diff --git a/cmake/init_scripts.cmake b/cmake/init_scripts.cmake new file mode 100644 index 000000000..1f07ee8de --- /dev/null +++ b/cmake/init_scripts.cmake @@ -0,0 +1,36 @@ +# Check which init.d script to install +find_file(RPM_FNC functions PATHS /etc/rc.d/init.d) +find_file(DEB_FNC init-functions PATHS /lib/lsb) + +if(${RPM_FNC} MATCHES "NOTFOUND" AND ${DEB_FNC} MATCHES "NOTFOUND") + message(FATAL_ERROR "Cannot find required init-functions in /lib/lsb/ or /etc/rc.d/init.d/, please confirm that your system files are OK.") +elseif(${RPM_FNC} MATCHES "NOTFOUND") + set(HAVE_LSB_FUNCTIONS TRUE CACHE BOOL "If init.d script uses /lib/lsb/init-functions instead of /etc/rc.d/init.d/functions.") +else() + set(HAVE_LSB_FUNCTIONS FALSE CACHE BOOL "If init.d script uses /lib/lsb/init-functions instead of /etc/rc.d/init.d/functions.") +endif() + +configure_file(${CMAKE_SOURCE_DIR}/etc/maxscale.conf.in ${CMAKE_BINARY_DIR}/maxscale.conf @ONLY) +configure_file(${CMAKE_SOURCE_DIR}/etc/maxscale.service.in ${CMAKE_BINARY_DIR}/maxscale.service @ONLY) + +if(HAVE_LSB_FUNCTIONS) + configure_file(${CMAKE_SOURCE_DIR}/etc/ubuntu/init.d/maxscale.in ${CMAKE_BINARY_DIR}/maxscale @ONLY) +else() + configure_file(${CMAKE_SOURCE_DIR}/etc/init.d/maxscale.in ${CMAKE_BINARY_DIR}/maxscale @ONLY) +endif() + +if(PACKAGE) + message(STATUS "maxscale.conf will unpack to: /etc/ld.so.conf.d") + message(STATUS "startup scripts will unpack to to: /etc/init.d") + message(STATUS "systemd service files will unpack to to: /usr/lib/systemd/system") + install_file(${CMAKE_BINARY_DIR}/maxscale core) + install_file(${CMAKE_BINARY_DIR}/maxscale.conf core) + install_file(${CMAKE_BINARY_DIR}/maxscale.service core) +else() + install(PROGRAMS ${CMAKE_BINARY_DIR}/maxscale DESTINATION /etc/init.d COMPONENT core) + install(FILES ${CMAKE_BINARY_DIR}/maxscale.conf DESTINATION /etc/ld.so.conf.d COMPONENT core) + install(FILES ${CMAKE_BINARY_DIR}/maxscale.service DESTINATION /usr/lib/systemd/system COMPONENT core) + message(STATUS "Installing maxscale.conf to: /etc/ld.so.conf.d") + message(STATUS "Installing startup scripts to: /etc/init.d") + message(STATUS "Installing systemd service files to: /usr/lib/systemd/system") +endif() diff --git a/cmake/install_layout.cmake b/cmake/install_layout.cmake index a14ef05c5..9171c238f 100644 --- a/cmake/install_layout.cmake +++ b/cmake/install_layout.cmake @@ -9,3 +9,111 @@ set(MAXSCALE_DOCDIR ${CMAKE_INSTALL_DOCDIR}/maxscale CACHE PATH "Documentation i # These are the only hard-coded absolute paths set(MAXSCALE_VARDIR /var CACHE PATH "Data file path (usually /var/)") set(MAXSCALE_CONFDIR /etc CACHE PATH "Configuration file installation path (/etc/)") + +# +# Installation functions for MaxScale +# +# Do not directly install files with install(...) etc. commands. Use these +# functions for all executables, modules and files that are to be installed +# as a part of a package. These functions make additional checks that all +# required parameters are present for the modules and make sure that the +# targets are installed into right directories where MaxScale can find them. +# + +# Installation functions for executables and modules. +# +# @param Name of the CMake target +# @param Component where this executable should be included +function(install_executable target component) + if(NOT TARGET_COMPONENT OR "${component}" STREQUAL "${TARGET_COMPONENT}") + install(TARGETS ${target} DESTINATION ${MAXSCALE_BINDIR} COMPONENT "${component}") + endif() +endfunction() + +# Installation function for modules +# +# @param Name of the CMake target +# @param Component where this module should be included +function(install_module target component) + get_target_property(TGT_VERSION ${target} VERSION) + if (${TGT_VERSION} MATCHES "NOTFOUND") + message(AUTHOR_WARNING "Module '${target}' is missing the VERSION parameter!") + endif() + + # If TARGET_COMPONENT is defined, only parts of that component are installed + if(NOT TARGET_COMPONENT OR "${component}" STREQUAL "${TARGET_COMPONENT}") + install(TARGETS ${target} DESTINATION ${MAXSCALE_LIBDIR} COMPONENT "${component}") + endif() +endfunction() + +# Installation functions for interpreted scripts. +# +# @param Script to install +# @param Component where this script should be included +function(install_script target component) + + # If TARGET_COMPONENT is defined, only parts of that component are installed + if(NOT TARGET_COMPONENT OR "${component}" STREQUAL "${TARGET_COMPONENT}") + install(PROGRAMS ${target} DESTINATION ${MAXSCALE_BINDIR} COMPONENT "${component}") + endif() +endfunction() + +# Installation functions for files and programs. These all go to the share directory +# of MaxScale which for packages is /usr/share/maxscale. +# +# @param File to install +# @param Component where this file should be included +function(install_file file component) + + # If TARGET_COMPONENT is defined, only parts of that component are installed + if(NOT TARGET_COMPONENT OR "${component}" STREQUAL "${TARGET_COMPONENT}") + install(FILES ${file} DESTINATION ${MAXSCALE_SHAREDIR} COMPONENT "${component}") + endif() +endfunction() + +function(install_program file component) + + # If TARGET_COMPONENT is defined, only parts of that component are installed + if(NOT TARGET_COMPONENT OR "${component}" STREQUAL "${TARGET_COMPONENT}") + install(PROGRAMS ${file} DESTINATION ${MAXSCALE_SHAREDIR} COMPONENT "${component}") + endif() +endfunction() + +# Install man pages +# +# @param Manual file to install +# @param The page number where this should be installed e.g. man1 +# @param Component where this manual should be included +function(install_manual file page component) + + # If TARGET_COMPONENT is defined, only parts of that component are installed + if(NOT TARGET_COMPONENT OR "${component}" STREQUAL "${TARGET_COMPONENT}") + install(PROGRAMS ${file} DESTINATION ${CMAKE_INSTALL_DATADIR}/man/man${page} COMPONENT "${component}") + endif() +endfunction() + +# Install headers +# +# @param Header to install +# @param Component where this header should be included +function(install_header header component) + + # If TARGET_COMPONENT is defined, only parts of that component are installed + if(NOT TARGET_COMPONENT OR "${component}" STREQUAL "${TARGET_COMPONENT}") + install(FILES ${header} DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/maxscale COMPONENT "${component}") + endif() +endfunction() + + +# Install custom file to a custom destination +# +# @param File to install +# @param Destination where to install the file +# @param Component where this file should be included +function(install_custom_file file dest component) + + # If TARGET_COMPONENT is defined, only parts of that component are installed + if(NOT TARGET_COMPONENT OR "${component}" STREQUAL "${TARGET_COMPONENT}") + install(FILES ${file} DESTINATION ${dest} COMPONENT "${component}") + endif() +endfunction() diff --git a/cmake/macros.cmake b/cmake/macros.cmake index 4c1a6efbe..9ce92589d 100644 --- a/cmake/macros.cmake +++ b/cmake/macros.cmake @@ -1,83 +1,11 @@ +# Helper functions function(debugmsg MSG) if(DEBUG_OUTPUT) message(STATUS "DEBUG: ${MSG}") endif() endfunction() -macro(check_deps) - - - # Check for libraries MaxScale depends on - find_library(HAVE_LIBAIO NAMES aio) - if(NOT HAVE_LIBAIO) - message(FATAL_ERROR "Could not find libaio") - endif() - - find_library(HAVE_LIBSSL NAMES ssl) - if(NOT HAVE_LIBSSL) - message(FATAL_ERROR "Could not find libssl") - endif() - - find_library(HAVE_LIBCRYPT NAMES crypt) - if(NOT HAVE_LIBCRYPT) - message(FATAL_ERROR "Could not find libcrypt") - endif() - - find_library(HAVE_LIBCRYPTO NAMES crypto) - if(NOT HAVE_LIBCRYPTO) - message(FATAL_ERROR "Could not find libcrypto") - endif() - - find_library(HAVE_LIBZ NAMES z) - if(NOT HAVE_LIBZ) - message(FATAL_ERROR "Could not find libz") - endif() - - find_library(HAVE_LIBM NAMES m) - if(NOT HAVE_LIBM) - message(FATAL_ERROR "Could not find libm") - endif() - - find_library(HAVE_LIBDL NAMES dl) - if(NOT HAVE_LIBDL) - message(FATAL_ERROR "Could not find libdl") - endif() - - find_library(HAVE_LIBRT NAMES rt) - if(NOT HAVE_LIBRT) - message(FATAL_ERROR "Could not find librt") - endif() - - find_library(HAVE_LIBPTHREAD NAMES pthread) - if(NOT HAVE_LIBPTHREAD) - message(FATAL_ERROR "Could not find libpthread") - endif() - - -endmacro() - macro(check_dirs) - # Check which init.d script to install - if(WITH_SCRIPTS) - find_file(RPM_FNC functions PATHS /etc/rc.d/init.d) - if(${RPM_FNC} MATCHES "RPM_FNC-NOTFOUND") - find_file(DEB_FNC init-functions PATHS /lib/lsb) - if(${DEB_FNC} MATCHES "DEB_FNC-NOTFOUND") - message(FATAL_ERROR "Cannot find required init-functions in /lib/lsb/ or /etc/rc.d/init.d/, please confirm that your system files are OK.") - else() - set(DEB_BASED TRUE CACHE BOOL "If init.d script uses /lib/lsb/init-functions instead of /etc/rc.d/init.d/functions.") - endif() - else() - set(DEB_BASED FALSE CACHE BOOL "If init.d script uses /lib/lsb/init-functions instead of /etc/rc.d/init.d/functions.") - endif() - unset(DEB_FNC) - unset(RPM_FNC) - endif() - - #Check RabbitMQ headers and libraries - if(BUILD_RABBITMQ) - find_package(RabbitMQ) - endif() endmacro() diff --git a/cmake/package.cmake b/cmake/package.cmake new file mode 100644 index 000000000..a63c49049 --- /dev/null +++ b/cmake/package.cmake @@ -0,0 +1,52 @@ +# Common packaging configuration + +execute_process(COMMAND uname -m COMMAND tr -d '\n' OUTPUT_VARIABLE CPACK_PACKAGE_ARCHITECTURE) + +# Generic CPack configuration variables +set(CPACK_SET_DESTDIR ON) +set(CPACK_PACKAGE_RELOCATABLE FALSE) +set(CPACK_STRIP_FILES FALSE) +set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "MaxScale") +set(CPACK_PACKAGE_VERSION_MAJOR "${MAXSCALE_VERSION_MAJOR}") +set(CPACK_PACKAGE_VERSION_MINOR "${MAXSCALE_VERSION_MINOR}") +set(CPACK_PACKAGE_VERSION_PATCH "${MAXSCALE_VERSION_PATCH}") +set(CPACK_PACKAGE_CONTACT "MariaDB Corporation Ab") +set(CPACK_PACKAGE_VENDOR "MariaDB Corporation Ab") +set(CPACK_PACKAGE_DESCRIPTION_FILE ${CMAKE_SOURCE_DIR}/etc/DESCRIPTION) +set(CPACK_PACKAGING_INSTALL_PREFIX "${CMAKE_INSTALL_PREFIX}") + +# If we're building something other than the main package, append the target name +# to the package name. +if(DEFINED TARGET_COMPONENT AND NOT TARGET_COMPONENT STREQUAL "core" AND NOT TARGET_COMPONENT STREQUAL "") + set(CPACK_PACKAGE_NAME "${PACKAGE_NAME}-${TARGET_COMPONENT}") +else() + set(CPACK_PACKAGE_NAME "${PACKAGE_NAME}") +endif() + +if(DISTRIB_SUFFIX) + set(CPACK_PACKAGE_FILE_NAME "${CPACK_PACKAGE_NAME}-${MAXSCALE_VERSION}-${MAXSCALE_BUILD_NUMBER}.${DISTRIB_SUFFIX}.${CMAKE_SYSTEM_PROCESSOR}") +else() + set(CPACK_PACKAGE_FILE_NAME "${CPACK_PACKAGE_NAME}-${MAXSCALE_VERSION}-${MAXSCALE_BUILD_NUMBER}.${CMAKE_SYSTEM_PROCESSOR}") +endif() + +# See if we are on a RPM-capable or DEB-capable system +find_program(RPMBUILD rpmbuild) +find_program(DEBBUILD dpkg-buildpackage) + +message(STATUS "Generating TGZ packages") +set(CPACK_GENERATOR "TGZ") + +if(NOT ( ${RPMBUILD} STREQUAL "RPMBUILD-NOTFOUND" ) ) + include(cmake/package_rpm.cmake) + message(STATUS "Generating RPM packages") + set(PACKAGE_SUFFIX "rpm") + set(RPM TRUE CACHE INTERNAL "RPM based installation") +elseif(NOT ( ${DEBBUILD} STREQUAL "DEBBUILD-NOTFOUND" ) ) + include(cmake/package_deb.cmake) + message(STATUS "Generating DEB packages for ${DEB_ARCHITECTURE}") + set(PACKAGE_SUFFIX "deb") + set(DEB TRUE CACHE INTERNAL "DEB based installation") +endif() + +message(STATUS "You can install startup scripts and system configuration files for MaxScale by running the 'postinst' shell script located at ${CMAKE_INSTALL_PREFIX}.") +message(STATUS "To remove these installed files, run the 'postrm' shell script located in the same folder.") diff --git a/cmake/package_deb.cmake b/cmake/package_deb.cmake index ab3e91d64..c0612dc2c 100644 --- a/cmake/package_deb.cmake +++ b/cmake/package_deb.cmake @@ -1,6 +1,14 @@ # DEB specific CPack configuration parameters set(CPACK_GENERATOR "${CPACK_GENERATOR};DEB") -set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${CMAKE_BINARY_DIR}/postinst;{CMAKE_BINARY_DIR}/postrm") execute_process(COMMAND dpkg --print-architecture OUTPUT_VARIABLE DEB_ARCHITECTURE OUTPUT_STRIP_TRAILING_WHITESPACE) set(CPACK_DEBIAN_PACKAGE_ARCHITECTURE ${DEB_ARCHITECTURE}) -set(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON) +set(CPACK_DEBIAN_PACKAGE_FILE_NAME "${CPACK_PACKAGE_NAME}") + +if(TARGET_COMPONENT STREQUAL "core") + set(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON) + set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${CMAKE_BINARY_DIR}/postinst;${CMAKE_BINARY_DIR}/postrm") +endif() + +if(EXTRA_PACKAGE_DEPENDENCIES) + set(CPACK_DEBIAN_PACKAGE_DEPENDS "${EXTRA_PACKAGE_DEPENDENCIES}") +endif() diff --git a/cmake/package_rpm.cmake b/cmake/package_rpm.cmake index fabf2a650..ec6e9bed4 100644 --- a/cmake/package_rpm.cmake +++ b/cmake/package_rpm.cmake @@ -1,13 +1,13 @@ # RPM specific CPack configuration parameters + set(CPACK_GENERATOR "${CPACK_GENERATOR};RPM") set(CPACK_RPM_PACKAGE_RELEASE ${MAXSCALE_BUILD_NUMBER}) -set(CPACK_RPM_POST_INSTALL_SCRIPT_FILE ${CMAKE_BINARY_DIR}/postinst) -set(CPACK_RPM_POST_UNINSTALL_SCRIPT_FILE ${CMAKE_BINARY_DIR}/postrm) -set(CPACK_RPM_PACKAGE_NAME "maxscale") set(CPACK_RPM_PACKAGE_VENDOR "MariaDB Corporation Ab") set(CPACK_RPM_PACKAGE_LICENSE "MariaDB BSL") -set(CPACK_RPM_EXCLUDE_FROM_AUTO_FILELIST_ADDITION "/etc /etc/ld.so.conf.d /etc/init.d /etc/rc.d/init.d") +set(CPACK_RPM_EXCLUDE_FROM_AUTO_FILELIST_ADDITION "/etc /etc/ld.so.conf.d /etc/init.d /etc/rc.d/init.d /usr/share/man /usr/share/man1") set(CPACK_RPM_SPEC_MORE_DEFINE "%define ignore \#") +set(CPACK_RPM_PACKAGE_NAME "${CPACK_PACKAGE_NAME}") +set(CPACK_RPM_PACKAGE_FILE_NAME "${CPACK_PACKAGE_FILE_NAME}") set(IGNORED_DIRS "%ignore /etc" @@ -23,3 +23,12 @@ set(IGNORED_DIRS "%ignore ${CMAKE_INSTALL_PREFIX}/share/man/man1") set(CPACK_RPM_USER_FILELIST "${IGNORED_DIRS}") + +if(TARGET_COMPONENT STREQUAL "core") + set(CPACK_RPM_POST_INSTALL_SCRIPT_FILE ${CMAKE_BINARY_DIR}/postinst) + set(CPACK_RPM_POST_UNINSTALL_SCRIPT_FILE ${CMAKE_BINARY_DIR}/postrm) +endif() + +if(EXTRA_PACKAGE_DEPENDENCIES) + set(CPACK_RPM_PACKAGE_REQUIRES "${EXTRA_PACKAGE_DEPENDENCIES}") +endif() diff --git a/doxygate.in b/etc/doxygate.in similarity index 100% rename from doxygate.in rename to etc/doxygate.in diff --git a/maxscale.conf.in b/etc/maxscale.conf.in similarity index 100% rename from maxscale.conf.in rename to etc/maxscale.conf.in diff --git a/plugins/CMakeLists.txt b/plugins/CMakeLists.txt index fc18cc602..ebff248a2 100644 --- a/plugins/CMakeLists.txt +++ b/plugins/CMakeLists.txt @@ -1,5 +1,5 @@ -install(FILES nagios/check_maxscale_monitors.pl DESTINATION ${MAXSCALE_SHAREDIR}/plugins/nagios) -install(FILES nagios/check_maxscale_resources.pl DESTINATION ${MAXSCALE_SHAREDIR}/plugins/nagios) -install(FILES nagios/check_maxscale_threads.pl DESTINATION ${MAXSCALE_SHAREDIR}/plugins/nagios) -install(FILES nagios/maxscale_commands.cfg DESTINATION ${MAXSCALE_SHAREDIR}/plugins/nagios) -install(FILES nagios/server1.cfg DESTINATION ${MAXSCALE_SHAREDIR}/plugins/nagios) +install_custom_file(nagios/check_maxscale_monitors.pl ${MAXSCALE_SHAREDIR}/plugins/nagios/ core) +install_custom_file(nagios/check_maxscale_resources.pl ${MAXSCALE_SHAREDIR}/plugins/nagios/ core) +install_custom_file(nagios/check_maxscale_threads.pl ${MAXSCALE_SHAREDIR}/plugins/nagios/ core) +install_custom_file(nagios/maxscale_commands.cfg ${MAXSCALE_SHAREDIR}/plugins/nagios/ core) +install_custom_file(nagios/server1.cfg ${MAXSCALE_SHAREDIR}/plugins/nagios/ core) diff --git a/plugins/nagios/check_maxscale_monitors.pl b/plugins/nagios/check_maxscale_monitors.pl index 6fb266e5b..34fa1a756 100755 --- a/plugins/nagios/check_maxscale_monitors.pl +++ b/plugins/nagios/check_maxscale_monitors.pl @@ -5,7 +5,7 @@ # Use of this software is governed by the Business Source License included # in the LICENSE.TXT file and at www.mariadb.com/bsl. # -# Change Date: 2019-01-01 +# Change Date: 2019-07-01 # # On the date above, in accordance with the Business Source License, use # of this software will be governed by version 2 or later of the General diff --git a/plugins/nagios/check_maxscale_resources.pl b/plugins/nagios/check_maxscale_resources.pl index 9ab7ec8e6..239a4963e 100755 --- a/plugins/nagios/check_maxscale_resources.pl +++ b/plugins/nagios/check_maxscale_resources.pl @@ -5,7 +5,7 @@ # Use of this software is governed by the Business Source License included # in the LICENSE.TXT file and at www.mariadb.com/bsl. # -# Change Date: 2019-01-01 +# Change Date: 2019-07-01 # # On the date above, in accordance with the Business Source License, use # of this software will be governed by version 2 or later of the General diff --git a/plugins/nagios/check_maxscale_threads.pl b/plugins/nagios/check_maxscale_threads.pl index 034d99290..88cbe4c54 100755 --- a/plugins/nagios/check_maxscale_threads.pl +++ b/plugins/nagios/check_maxscale_threads.pl @@ -5,7 +5,7 @@ # Use of this software is governed by the Business Source License included # in the LICENSE.TXT file and at www.mariadb.com/bsl. # -# Change Date: 2019-01-01 +# Change Date: 2019-07-01 # # On the date above, in accordance with the Business Source License, use # of this software will be governed by version 2 or later of the General diff --git a/query_classifier/qc_dummy/CMakeLists.txt b/query_classifier/qc_dummy/CMakeLists.txt index 81b2235d8..b7712a67b 100644 --- a/query_classifier/qc_dummy/CMakeLists.txt +++ b/query_classifier/qc_dummy/CMakeLists.txt @@ -1,4 +1,4 @@ add_library(qc_dummy SHARED qc_dummy.cc) set_target_properties(qc_dummy PROPERTIES VERSION "1.0.0") set_target_properties(qc_dummy PROPERTIES LINK_FLAGS -Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/qc_dummy.map) -install(TARGETS qc_dummy COMPONENT lib DESTINATION ${MAXSCALE_LIBDIR}) +install_module(qc_dummy core) diff --git a/query_classifier/qc_dummy/qc_dummy.cc b/query_classifier/qc_dummy/qc_dummy.cc index 7d469101c..82f1381dd 100644 --- a/query_classifier/qc_dummy/qc_dummy.cc +++ b/query_classifier/qc_dummy/qc_dummy.cc @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/query_classifier/qc_mysqlembedded/CMakeLists.txt b/query_classifier/qc_mysqlembedded/CMakeLists.txt index b8ee02e6f..489f7fe60 100644 --- a/query_classifier/qc_mysqlembedded/CMakeLists.txt +++ b/query_classifier/qc_mysqlembedded/CMakeLists.txt @@ -14,5 +14,5 @@ if (BUILD_QC_MYSQLEMBEDDED) set_target_properties(qc_mysqlembedded PROPERTIES VERSION "1.0.0") set_target_properties(qc_mysqlembedded PROPERTIES LINK_FLAGS -Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/qc_mysqlembedded.map) #set_target_properties(qc_mysqlembedded PROPERTIES LINK_FLAGS -Wl,-z,defs) - install(TARGETS qc_mysqlembedded COMPONENT lib DESTINATION ${MAXSCALE_LIBDIR}) + install_module(qc_mysqlembedded libmysqld-parser) endif() diff --git a/query_classifier/qc_mysqlembedded/qc_mysqlembedded.cc b/query_classifier/qc_mysqlembedded/qc_mysqlembedded.cc index df86d4f47..b863b3f38 100644 --- a/query_classifier/qc_mysqlembedded/qc_mysqlembedded.cc +++ b/query_classifier/qc_mysqlembedded/qc_mysqlembedded.cc @@ -55,8 +55,8 @@ #include #include -#include "../utils/skygw_types.h" -#include "../utils/skygw_debug.h" +#include +#include #include #include #include diff --git a/query_classifier/qc_sqlite/CMakeLists.txt b/query_classifier/qc_sqlite/CMakeLists.txt index 2cae784d1..b3d028723 100644 --- a/query_classifier/qc_sqlite/CMakeLists.txt +++ b/query_classifier/qc_sqlite/CMakeLists.txt @@ -30,4 +30,5 @@ set_target_properties(qc_sqlite PROPERTIES LINK_FLAGS -Wl,--version-script=${CMA # # target_link_libraries(qc_sqlite maxscale-common) # set_target_properties(qc_sqlite PROPERTIES LINK_FLAGS -Wl,-z,defs) -install(TARGETS qc_sqlite COMPONENT lib DESTINATION ${MAXSCALE_LIBDIR}) + +install_module(qc_sqlite core) diff --git a/query_classifier/qc_sqlite/qc_sqlite.c b/query_classifier/qc_sqlite/qc_sqlite.c index 3ee36d6c1..6bd271279 100644 --- a/query_classifier/qc_sqlite/qc_sqlite.c +++ b/query_classifier/qc_sqlite/qc_sqlite.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -21,6 +21,7 @@ #include #include #include +#include #include "builtin_functions.h" //#define QC_TRACE_ENABLED @@ -106,65 +107,6 @@ static thread_local struct QC_SQLITE_INFO* info; } this_thread; - -/** - * MaxScale allocation functions. - * - * In the environments where MaxScale runs, malloc and friends will not fail, but - * in low-memory situations some process will be killed. These functions now - * never return NULL, but abort the process should the allocation failed for some - * unknown reason. - */ -static void* mxs_malloc(size_t size) -{ - void* p = malloc(size); - if (!p) - { - raise(SIGABRT); - } - - return p; -} - -static void mxs_free(void* p) -{ - free(p); -} - -static void* mxs_realloc(void* p, size_t size) -{ - p = realloc(p, size); - if (!p) - { - raise(SIGABRT); - } - - return p; -} - -static void* mxs_calloc(size_t n, size_t size) -{ - void* p = calloc(n, size); - if (!p) - { - raise(SIGABRT); - } - - return p; -} - -static char* mxs_strdup(const char* s1) -{ - char* s2 = strdup(s1); - if (!s2) - { - raise(SIGABRT); - } - - return s2; -} - - /** * HELPERS */ @@ -262,13 +204,15 @@ static char** copy_string_array(char** strings, int* pn) ++(*pn); } - ss = (char**) mxs_malloc((*pn + 1) * sizeof(char*)); + ss = (char**) MXS_MALLOC((*pn + 1) * sizeof(char*)); + MXS_ABORT_IF_NULL(ss); ss[*pn] = 0; for (int i = 0; i < *pn; ++i) { - ss[i] = mxs_strdup(strings[i]); + ss[i] = MXS_STRDUP(strings[i]); + MXS_ABORT_IF_NULL(ss[i]); } return ss; @@ -280,7 +224,8 @@ static void enlarge_string_array(size_t n, size_t len, char*** ppzStrings, size_ { int capacity = *pCapacity ? *pCapacity * 2 : 4; - *ppzStrings = (char**) mxs_realloc(*ppzStrings, capacity * sizeof(char**)); + *ppzStrings = (char**) MXS_REALLOC(*ppzStrings, capacity * sizeof(char**)); + MXS_ABORT_IF_NULL(*ppzStrings); *pCapacity = capacity; } } @@ -328,7 +273,8 @@ static QC_SQLITE_INFO* get_query_info(GWBUF* query) static QC_SQLITE_INFO* info_alloc(void) { - QC_SQLITE_INFO* info = mxs_malloc(sizeof(*info)); + QC_SQLITE_INFO* info = MXS_MALLOC(sizeof(*info)); + MXS_ABORT_IF_NULL(info); info_init(info); @@ -618,7 +564,8 @@ static void append_affected_field(QC_SQLITE_INFO* info, const char* s) info->affected_fields_capacity *= 2; } - info->affected_fields = mxs_realloc(info->affected_fields, info->affected_fields_capacity); + info->affected_fields = MXS_REALLOC(info->affected_fields, info->affected_fields_capacity); + MXS_ABORT_IF_NULL(info->affected_fields); } if (info->affected_fields_len != 0) @@ -878,7 +825,8 @@ static void update_affected_fields_from_select(QC_SQLITE_INFO* info, static void update_database_names(QC_SQLITE_INFO* info, const char* zDatabase) { - char* zCopy = mxs_strdup(zDatabase); + char* zCopy = MXS_STRDUP(zDatabase); + MXS_ABORT_IF_NULL(zCopy); exposed_sqlite3Dequote(zCopy); enlarge_string_array(1, info->database_names_len, @@ -889,7 +837,8 @@ static void update_database_names(QC_SQLITE_INFO* info, const char* zDatabase) static void update_names(QC_SQLITE_INFO* info, const char* zDatabase, const char* zTable) { - char* zCopy = mxs_strdup(zTable); + char* zCopy = MXS_STRDUP(zTable); + MXS_ABORT_IF_NULL(zCopy); exposed_sqlite3Dequote(zCopy); enlarge_string_array(1, info->table_names_len, &info->table_names, &info->table_names_capacity); @@ -898,7 +847,8 @@ static void update_names(QC_SQLITE_INFO* info, const char* zDatabase, const char if (zDatabase) { - zCopy = mxs_malloc(strlen(zDatabase) + 1 + strlen(zTable) + 1); + zCopy = MXS_MALLOC(strlen(zDatabase) + 1 + strlen(zTable) + 1); + MXS_ABORT_IF_NULL(zCopy); strcpy(zCopy, zDatabase); strcat(zCopy, "."); @@ -909,7 +859,8 @@ static void update_names(QC_SQLITE_INFO* info, const char* zDatabase, const char } else { - zCopy = mxs_strdup(zCopy); + zCopy = MXS_STRDUP(zCopy); + MXS_ABORT_IF_NULL(zCopy); } enlarge_string_array(1, info->table_fullnames_len, @@ -1399,7 +1350,8 @@ void mxs_sqlite3StartTable(Parse *pParse, /* Parser context */ update_names(info, NULL, name); } - info->created_table_name = mxs_strdup(info->table_names[0]); + info->created_table_name = MXS_STRDUP(info->table_names[0]); + MXS_ABORT_IF_NULL(info->created_table_name); } else { @@ -2607,7 +2559,8 @@ static char* qc_sqlite_get_created_table_name(GWBUF* query) { if (info->created_table_name) { - created_table_name = mxs_strdup(info->created_table_name); + created_table_name = MXS_STRDUP(info->created_table_name); + MXS_ABORT_IF_NULL(created_table_name); } } else @@ -2792,7 +2745,10 @@ static char* qc_sqlite_get_affected_fields(GWBUF* query) affected_fields = ""; } - return mxs_strdup(affected_fields); + affected_fields = MXS_STRDUP(affected_fields); + MXS_ABORT_IF_NULL(affected_fields); + + return affected_fields; } static char** qc_sqlite_get_database_names(GWBUF* query, int* sizep) diff --git a/query_classifier/qc_sqlite/qc_sqlite3.c b/query_classifier/qc_sqlite/qc_sqlite3.c index f9e4b2038..db8d54455 100644 --- a/query_classifier/qc_sqlite/qc_sqlite3.c +++ b/query_classifier/qc_sqlite/qc_sqlite3.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/query_classifier/test/canonical_tests/canonizer.c b/query_classifier/test/canonical_tests/canonizer.c index 223ebc023..5609ecec7 100644 --- a/query_classifier/test/canonical_tests/canonizer.c +++ b/query_classifier/test/canonical_tests/canonizer.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/query_classifier/test/classify.c b/query_classifier/test/classify.c index 846700490..43449f969 100644 --- a/query_classifier/test/classify.c +++ b/query_classifier/test/classify.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/query_classifier/test/compare.cc b/query_classifier/test/compare.cc index ad4ab97e0..b289881a6 100644 --- a/query_classifier/test/compare.cc +++ b/query_classifier/test/compare.cc @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -298,8 +298,18 @@ bool compare_parse(QUERY_CLASSIFIER* pClassifier1, GWBUF* pCopy1, bool success = false; const char HEADING[] = "qc_parse : "; + struct timespec start; + struct timespec finish; + + clock_gettime(CLOCK_MONOTONIC_RAW, &start); qc_parse_result_t rv1 = pClassifier1->qc_parse(pCopy1); + clock_gettime(CLOCK_MONOTONIC_RAW, &finish); + update_time(&global.time1, start, finish); + + clock_gettime(CLOCK_MONOTONIC_RAW, &start); qc_parse_result_t rv2 = pClassifier2->qc_parse(pCopy2); + clock_gettime(CLOCK_MONOTONIC_RAW, &finish); + update_time(&global.time2, start, finish); stringstream ss; ss << HEADING; @@ -335,20 +345,8 @@ bool compare_get_type(QUERY_CLASSIFIER* pClassifier1, GWBUF* pCopy1, bool success = false; const char HEADING[] = "qc_get_type : "; - struct timespec start; - struct timespec finish; - - // As long as we cannot explicitly parse a query, we rely upon the - // knowledge that it will be parsed the first time a qc-function is called. - clock_gettime(CLOCK_MONOTONIC_RAW, &start); uint32_t rv1 = pClassifier1->qc_get_type(pCopy1); - clock_gettime(CLOCK_MONOTONIC_RAW, &finish); - update_time(&global.time1, start, finish); - - clock_gettime(CLOCK_MONOTONIC_RAW, &start); uint32_t rv2 = pClassifier2->qc_get_type(pCopy2); - clock_gettime(CLOCK_MONOTONIC_RAW, &finish); - update_time(&global.time2, start, finish); stringstream ss; ss << HEADING; diff --git a/rabbitmq_consumer/CMakeLists.txt b/rabbitmq_consumer/CMakeLists.txt index 25a438ad7..486c92cd5 100644 --- a/rabbitmq_consumer/CMakeLists.txt +++ b/rabbitmq_consumer/CMakeLists.txt @@ -48,7 +48,7 @@ elseif(NOT (${DEBBUILD} MATCHES "NOTFOUND" ) ) set(CPACK_GENERATOR "${CPACK_GENERATOR};DEB") execute_process(COMMAND dpgk --print-architecture OUTPUT_VARIABLE DEB_ARCHITECTURE) set(CPACK_DEBIAN_PACKAGE_ARCHITECTURE ${DEB_ARCHITECTURE}) - set (CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON) + set (CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON) message(STATUS "Generating DEB packages for ${DEB_ARCHITECTURE}") endif() diff --git a/rabbitmq_consumer/consumer.c b/rabbitmq_consumer/consumer.c index 5895b982d..af7e05fe2 100644 --- a/rabbitmq_consumer/consumer.c +++ b/rabbitmq_consumer/consumer.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/script/stacktrace b/script/stacktrace index 328233017..4d183f960 100755 --- a/script/stacktrace +++ b/script/stacktrace @@ -24,7 +24,7 @@ function parse_stack_trace { local prefix=$1 local file=$2 - cat $file | egrep "$STACK_LINE" | sed "s/$DATE_TIME//" | \ + egrep "$STACK_LINE" < "$file" | sed "s/$DATE_TIME//" | \ while read line do local path=${line%%(*} @@ -34,7 +34,7 @@ function parse_stack_trace { if [ -e "${path}" ] then - file ${path} | fgrep -q executable + file "${path}" | fgrep -q executable let rc=$? local address; @@ -47,9 +47,9 @@ function parse_stack_trace { address=${address%\)*} fi - addr2line -e ${path} ${address} + addr2line -e "${path}" "${address}" else - echo ${line} + echo "${line}" fi done } @@ -58,7 +58,7 @@ function main { local prefix local key - while [[ $# > 1 ]] + while [[ $# -gt 1 ]] do key="$1" @@ -84,4 +84,4 @@ function main { parse_stack_trace "$prefix" "$file" } -main $* +main "$@" diff --git a/script/unpack_rpm.sh b/script/unpack_rpm.sh deleted file mode 100755 index b25680b28..000000000 --- a/script/unpack_rpm.sh +++ /dev/null @@ -1,32 +0,0 @@ -#!/bin/sh - -#This script unpacks the RPM to the provided directory - -unpack_to(){ - cd $2 && rpm2cpio $1 | cpio -id; -} - - -if [[ $# -lt 2 ]] -then - echo "Usage: unpack_rpm.sh " - exit 0 -fi - -SOURCE=$1 -DEST=$2 -FILES=$(ls $SOURCE |grep -i .*mariadb.*`uname -m`.*.rpm) - -if [[ ! -d $DEST ]] -then - mkdir -p $DEST -fi - -echo "Unpacking RPMs to: $DEST" - -for rpm in $FILES -do - echo "Unpacking $rpm..." - unpack_to $SOURCE/$rpm $DEST -done - diff --git a/server/CMakeLists.txt b/server/CMakeLists.txt index 69361c9d6..df0edd60a 100644 --- a/server/CMakeLists.txt +++ b/server/CMakeLists.txt @@ -4,3 +4,4 @@ include_directories(${MARIADB_CONNECTOR_INCLUDE_DIR}) add_subdirectory(core) add_subdirectory(modules) add_subdirectory(inih) +add_subdirectory(include) diff --git a/server/core/CMakeLists.txt b/server/core/CMakeLists.txt index 0f5c7d937..25b3578b2 100644 --- a/server/core/CMakeLists.txt +++ b/server/core/CMakeLists.txt @@ -1,4 +1,4 @@ -add_library(maxscale-common SHARED adminusers.c atomic.c buffer.c config.c dbusers.c dcb.c filter.c externcmd.c gwbitmask.c gwdirs.c gw_utils.c hashtable.c hint.c housekeeper.c load_utils.c log_manager.cc maxscale_pcre2.c memlog.c misc.c mlist.c modutil.c monitor.c queuemanager.c query_classifier.c poll.c random_jkiss.c resultset.c secrets.c server.c service.c session.c slist.c spinlock.c thread.c users.c utils.c ${CMAKE_SOURCE_DIR}/utils/skygw_utils.cc statistics.c listener.c gw_ssl.c mysql_utils.c mysql_binlog.c) +add_library(maxscale-common SHARED adminusers.c alloc.c atomic.c buffer.c config.c dbusers.c dcb.c filter.c externcmd.c gwbitmask.c gwdirs.c gw_utils.c hashtable.c hint.c housekeeper.c listmanager.c load_utils.c log_manager.cc maxscale_pcre2.c memlog.c misc.c mlist.c modutil.c monitor.c queuemanager.c query_classifier.c poll.c random_jkiss.c resultset.c secrets.c server.c service.c session.c slist.c spinlock.c thread.c users.c utils.c skygw_utils.cc statistics.c listener.c gw_ssl.c mysql_utils.c mysql_binlog.c) target_link_libraries(maxscale-common ${MARIADB_CONNECTOR_LIBRARIES} ${LZMA_LINK_FLAGS} ${PCRE2_LIBRARIES} ${CURL_LIBRARIES} ssl aio pthread crypt dl crypto inih z rt m stdc++) @@ -9,8 +9,8 @@ elseif(WITH_TCMALLOC) endif() add_dependencies(maxscale-common pcre2 connector-c) -install(TARGETS maxscale-common DESTINATION ${MAXSCALE_LIBDIR}) set_target_properties(maxscale-common PROPERTIES VERSION "1.0.0") +install_module(maxscale-common core) add_executable(maxscale gateway.c) add_dependencies(maxscale pcre2) @@ -22,15 +22,15 @@ elseif(WITH_TCMALLOC) endif() target_link_libraries(maxscale maxscale-common) -install(TARGETS maxscale DESTINATION ${MAXSCALE_BINDIR}) +install_executable(maxscale core) add_executable(maxkeys maxkeys.c) target_link_libraries(maxkeys maxscale-common) -install(TARGETS maxkeys DESTINATION ${MAXSCALE_BINDIR}) +install_executable(maxkeys core) add_executable(maxpasswd maxpasswd.c) target_link_libraries(maxpasswd maxscale-common) -install(TARGETS maxpasswd DESTINATION ${MAXSCALE_BINDIR}) +install_executable(maxpasswd core) if(BUILD_TESTS) add_subdirectory(test) diff --git a/server/core/adminusers.c b/server/core/adminusers.c index 0c5dd9f64..6ceb16293 100644 --- a/server/core/adminusers.c +++ b/server/core/adminusers.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/alloc.c b/server/core/alloc.c new file mode 100644 index 000000000..6ac095181 --- /dev/null +++ b/server/core/alloc.c @@ -0,0 +1,218 @@ +/* + * Copyright (c) 2016 MariaDB Corporation Ab + * + * Use of this software is governed by the Business Source License included + * in the LICENSE.TXT file and at www.mariadb.com/bsl. + * + * Change Date: 2019-07-01 + * + * On the date above, in accordance with the Business Source License, use + * of this software will be governed by version 2 or later of the General + * Public License. + */ + +#include +#include +#include + +/** + * @brief Allocates memory; behaves exactly like `malloc`. + * + * Usually `mxs_malloc` is not used, but the macro `MXS_MALLOC` instead. + * + * @note The returned pointer can be passed to `mxs_realloc`, `mxs_realloc_a` + * and `mxs_free`. + * + * @param size The amount of memory to allocate. + * @param caller The name of the function calling this function. + * @return A pointer to the allocated memory. + */ +void *mxs_malloc(size_t size/*, const char *caller*/) +{ + void *ptr = malloc(size); + + if (!ptr) + { + //MXS_OOM_MESSAGE(caller); + MXS_OOM(); + } + + return ptr; +} + +/** + * @brief Allocates memory; behaves exactly like `calloc`. + * + * Usually `mxs_calloc` is not used, but the macro `MXS_CALLOC` instead. + * + * @note The returned pointer can be passed to `mxs_realloc`, `mxs_realloc_a` + * and `mxs_free`. + * + * @param nmemb The number of elements. + * @param size The size of each element. + * @param caller The name of the function calling this function. + * @return A pointer to the allocated memory. + */ +void *mxs_calloc(size_t nmemb, size_t size/*, const char *caller*/) +{ + void *ptr = calloc(nmemb, size); + + if (!ptr) + { + //MXS_OOM_MESSAGE(caller); + MXS_OOM(); + } + + return ptr; +} + +/** + * @brief Re-allocates memory; behaves exactly like `realloc`. + * + * Usually `mxs_realloc` is not used, but the macro `MXS_REALLOC` instead. + * + * @note The returned pointer can be passed to `mxs_realloc`, `mxs_realloc_a` + * and `mxs_free`. + * + * @param ptr Pointer to memory earlier allocated by `mxs_malloc`, + * `mxs_calloc`, `mxs_realloc`, `mxs_strdup`, `mxs_strndup` + or or their `_a` equivalents. + * @param size What size the memory block should be changed to. + * @param caller The name of the function calling this function. + * @return A pointer to the allocated memory. + */ +void *mxs_realloc(void *ptr, size_t size/*, const char *caller*/) +{ + ptr = realloc(ptr, size); + + if (!ptr) + { + //MXS_OOM_MESSAGE(caller); + MXS_OOM(); + } + + return ptr; +} + +/** + * @brief Duplicates a string; behaves exactly like `strdup`. + * + * Usually `mxs_strdup` is not used, but the macro `MXS_STRDUP` instead. + * + * @note The returned pointer can be passed to `mxs_realloc` and `mxs_free`. + * + * @param s1 The string to be duplicated. + * @param caller The name of the function calling this function. + * @return A copy of the string. + */ +char *mxs_strdup(const char *s1/*, const char *caller*/) +{ + char *s2 = strdup(s1); + + if (!s2) + { + //MXS_OOM_MESSAGE(caller); + MXS_OOM(); + } + + return s2; +} + +/** + * @brief Duplicates a string; behaves exactly like `strndup`. + * + * Usually `mxs_strndup` is not used, but the macro `MXS_STRNDUP` instead. + * + * @note The returned pointer can be passed to `mxs_realloc` and `mxs_free`. + * + * @param s1 The string to be duplicated. + * @param n At most n bytes should be copied. + * @param caller The name of the function calling this function. + * @return A copy of the string. + */ +char *mxs_strndup(const char *s1, size_t n/*, const char *caller*/) +{ + char *s2 = strndup(s1, n); + + if (!s2) + { + //MXS_OOM_MESSAGE(caller); + MXS_OOM(); + } + + return s2; +} + +/** + * @brief Frees memory. + * + * Usually `mxs_free` is not used, but the macro `MXS_FREE` instead. + * + * @note The memory must have earlier been allocated with `mxs_malloc`, + * `mxs_calloc`, `mxs_realloc`, `mxs_strdup`, `mxs_strndup`, or + * their `_a` equivalents. + * + * @param ptr Pointer to the memory to be freed. + * @param caller The name of the function calling this function. + */ +void mxs_free(void *ptr/*, const char *caller*/) +{ + free(ptr); +} + +/** + * @brief Duplicates a string. + * + * Behaves exactly like `strdup`, except that `mxs_strdup_a` _always_ + * returns a non-NULL pointer. In case `mxs_strdup_a` cannot do that, + * it _as_ the process. + * + * Usually `mxs_strdup_a` is not used, but the macro `MXS_STRDUP_A` + * instead. + * + * @note The returned pointer can be passed to `mxs_realloc` and `mxs_free`. + * + * @param s1 The string to be duplicated. + * @param caller The name of the function calling this function. + * @return A copy of the string. + */ +char *mxs_strdup_a(const char *s1/*, const char *caller*/) +{ + char *s2 = mxs_strdup(s1/*, caller*/); + + if (!s2) + { + abort(); + } + + return s2; +} + +/** + * @breif Duplicates a string. + * + * Behaves exactly like `strndup` except that `mxs_strndup_a` _always_ + * returns a non-NULL pointer. In case `mxs_strndup_a` cannot do that, it + * _as_ the process. + * + * Usually `mxs_strndup_a` is not used, but the macro MXS_STRNDUP_A + * instead. + * + * @note The returned pointer can be passed to `mxs_realloc` and `mxs_free`. + * + * @param s1 The string to be duplicated. + * @param n At most n bytes should be copied. + * @param caller The name of the function calling this function. + * @return A copy of the string. + */ +char *mxs_strndup_a(const char *s1, size_t n/*, const char *caller*/) +{ + char *s2 = mxs_strndup(s1, n/*, caller*/); + + if (!s2) + { + abort(); + } + + return s2; +} diff --git a/server/core/atomic.c b/server/core/atomic.c index 9c15d2782..fe5c5eb45 100644 --- a/server/core/atomic.c +++ b/server/core/atomic.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/buffer.c b/server/core/buffer.c index ce4b794b9..feed261b8 100644 --- a/server/core/buffer.c +++ b/server/core/buffer.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -37,15 +37,16 @@ * * @endverbatim */ -#include #include +#include +#include +#include #include #include #include #include #include #include -#include #if defined(BUFFER_TRACE) #include @@ -83,25 +84,24 @@ gwbuf_alloc(unsigned int size) SHARED_BUF *sbuf; /* Allocate the buffer header */ - if ((rval = (GWBUF *)malloc(sizeof(GWBUF))) == NULL) + if ((rval = (GWBUF *)MXS_MALLOC(sizeof(GWBUF))) == NULL) { goto retblock; } /* Allocate the shared data buffer */ - if ((sbuf = (SHARED_BUF *)malloc(sizeof(SHARED_BUF))) == NULL) + if ((sbuf = (SHARED_BUF *)MXS_MALLOC(sizeof(SHARED_BUF))) == NULL) { - free(rval); + MXS_FREE(rval); rval = NULL; goto retblock; } /* Allocate the space for the actual data */ - if ((sbuf->data = (unsigned char *)malloc(size)) == NULL) + if ((sbuf->data = (unsigned char *)MXS_MALLOC(size)) == NULL) { - ss_dassert(sbuf->data != NULL); - free(rval); - free(sbuf); + MXS_FREE(rval); + MXS_FREE(sbuf); rval = NULL; goto retblock; } @@ -174,7 +174,7 @@ gwbuf_add_to_hashtable(GWBUF *buf) { total += strlen(strings[i]); } - tracetext = (char *)malloc(total); + tracetext = (char *)MXS_MALLOC(total); if (tracetext) { char *ptr = tracetext; @@ -188,7 +188,7 @@ gwbuf_add_to_hashtable(GWBUF *buf) if (NULL == buffer_hashtable) { buffer_hashtable = hashtable_alloc(10000, bhashfn, bcmpfn); - hashtable_memory_fns(buffer_hashtable, NULL, NULL, NULL, (HASHMEMORYFN)free); + hashtable_memory_fns(buffer_hashtable, NULL, NULL, NULL, hashtable_item_free); } hashtable_add(buffer_hashtable, buf, (void *)tracetext); } @@ -283,8 +283,8 @@ gwbuf_free_one(GWBUF *buf) if (atomic_add(&buf->sbuf->refcount, -1) == 1) { - free(buf->sbuf->data); - free(buf->sbuf); + MXS_FREE(buf->sbuf->data); + MXS_FREE(buf->sbuf); bo = buf->gwbuf_bufobj; while (bo != NULL) @@ -297,9 +297,9 @@ gwbuf_free_one(GWBUF *buf) { prop = buf->properties; buf->properties = prop->next; - free(prop->name); - free(prop->value); - free(prop); + MXS_FREE(prop->name); + MXS_FREE(prop->value); + MXS_FREE(prop); } /** Release the hint */ while (buf->hint) @@ -311,7 +311,7 @@ gwbuf_free_one(GWBUF *buf) #if defined(BUFFER_TRACE) gwbuf_remove_from_hashtable(buf); #endif - free(buf); + MXS_FREE(buf); } /** @@ -329,12 +329,8 @@ gwbuf_clone(GWBUF *buf) { GWBUF *rval; - if ((rval = (GWBUF *)calloc(1, sizeof(GWBUF))) == NULL) + if ((rval = (GWBUF *)MXS_CALLOC(1, sizeof(GWBUF))) == NULL) { - ss_dassert(rval != NULL); - char errbuf[STRERROR_BUFLEN]; - MXS_ERROR("Memory allocation failed due to %s.", - strerror_r(errno, errbuf, sizeof(errbuf))); return NULL; } @@ -393,12 +389,8 @@ GWBUF *gwbuf_clone_portion(GWBUF *buf, CHK_GWBUF(buf); ss_dassert(start_offset + length <= GWBUF_LENGTH(buf)); - if ((clonebuf = (GWBUF *)malloc(sizeof(GWBUF))) == NULL) + if ((clonebuf = (GWBUF *)MXS_MALLOC(sizeof(GWBUF))) == NULL) { - ss_dassert(clonebuf != NULL); - char errbuf[STRERROR_BUFLEN]; - MXS_ERROR("Memory allocation failed due to %s.", - strerror_r(errno, errbuf, sizeof(errbuf))); return NULL; } atomic_add(&buf->sbuf->refcount, 1); @@ -729,16 +721,9 @@ void gwbuf_add_buffer_object(GWBUF* buf, buffer_object_t* newb; CHK_GWBUF(buf); - newb = (buffer_object_t *)malloc(sizeof(buffer_object_t)); - ss_dassert(newb != NULL); + newb = (buffer_object_t *)MXS_MALLOC(sizeof(buffer_object_t)); + MXS_ABORT_IF_NULL(newb); - if (newb == NULL) - { - char errbuf[STRERROR_BUFLEN]; - MXS_ERROR("Memory allocation failed due to %s.", - strerror_r(errno, errbuf, sizeof(errbuf))); - return; - } newb->bo_id = id; newb->bo_data = data; newb->bo_donefun_fp = donefun_fp; @@ -798,7 +783,7 @@ static buffer_object_t* gwbuf_remove_buffer_object(GWBUF* buf, buffer_object_t* next = bufobj->bo_next; /** Call corresponding clean-up function to clean buffer object's data */ bufobj->bo_donefun_fp(bufobj->bo_data); - free(bufobj); + MXS_FREE(bufobj); return next; } @@ -813,18 +798,21 @@ static buffer_object_t* gwbuf_remove_buffer_object(GWBUF* buf, buffer_object_t* int gwbuf_add_property(GWBUF *buf, char *name, char *value) { - BUF_PROPERTY *prop; + name = MXS_STRDUP(name); + value = MXS_STRDUP(value); - if ((prop = malloc(sizeof(BUF_PROPERTY))) == NULL) + BUF_PROPERTY *prop = (BUF_PROPERTY *)MXS_MALLOC(sizeof(BUF_PROPERTY)); + + if (!name || !value || !prop) { - ss_dassert(prop != NULL); - char errbuf[STRERROR_BUFLEN]; - MXS_ERROR("Memory allocation failed due to %s.", - strerror_r(errno, errbuf, sizeof(errbuf))); + MXS_FREE(name); + MXS_FREE(value); + MXS_FREE(prop); return 0; } - prop->name = strdup(name); - prop->value = strdup(value); + + prop->name = name; + prop->value = value; spinlock_acquire(&buf->gwbuf_lock); prop->next = buf->properties; buf->properties = prop; diff --git a/server/core/config.c b/server/core/config.c index 2559fec02..da646f8fd 100644 --- a/server/core/config.c +++ b/server/core/config.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -51,6 +51,8 @@ #include #include #include +#include +#include #include #include #include @@ -72,6 +74,7 @@ #include #include #include +#include #define PCRE2_CODE_UNIT_WIDTH 8 #include @@ -91,7 +94,7 @@ static SSL_LISTENER *make_ssl_structure(CONFIG_CONTEXT *obj, bool require_cert, int config_truth_value(char *str); int config_get_ifaddr(unsigned char *output); -int config_get_release_string(char* release); +static int config_get_release_string(char* release); FEEDBACK_CONF *config_get_feedback_data(); void config_add_param(CONFIG_CONTEXT*, char*, char*); bool config_has_duplicate_sections(const char* config); @@ -118,8 +121,8 @@ static char *service_params[] = "passwd", "enable_root_user", "max_connections", - /* "max_queued_connections", */ - /* "queued_connection_timeout", */ + "max_queued_connections", + "queued_connection_timeout", "connection_timeout", "auth_all_servers", "strip_db_esc", @@ -206,9 +209,10 @@ static char *server_params[] = */ char* config_clean_string_list(char* str) { - char *dest; size_t destsize = strlen(str) + 1; - if ((dest = malloc(destsize)) != NULL) + char *dest = MXS_MALLOC(destsize); + + if (dest) { pcre2_code* re; pcre2_match_data* data; @@ -224,7 +228,7 @@ char* config_clean_string_list(char* str) MXS_ERROR("[%s] Regular expression compilation failed at %d: %s", __FUNCTION__, (int)err_offset, errbuf); pcre2_code_free(re); - free(dest); + MXS_FREE(dest); return NULL; } @@ -235,10 +239,10 @@ char* config_clean_string_list(char* str) (PCRE2_SPTR) replace, PCRE2_ZERO_TERMINATED, (PCRE2_UCHAR*) dest, &destsize)) == PCRE2_ERROR_NOMEMORY) { - char* tmp = realloc(dest, destsize * 2); + char* tmp = MXS_REALLOC(dest, destsize * 2); if (tmp == NULL) { - free(dest); + MXS_FREE(dest); dest = NULL; break; } @@ -255,10 +259,6 @@ char* config_clean_string_list(char* str) pcre2_code_free(re); pcre2_match_data_free(data); } - else - { - MXS_ERROR("[%s] Memory allocation failed.", __FUNCTION__); - } return dest; } @@ -305,12 +305,12 @@ handler(void *userdata, const char *section, const char *name, const char *value if (!ptr) { - if ((ptr = (CONFIG_CONTEXT *)malloc(sizeof(CONFIG_CONTEXT))) == NULL) + if ((ptr = (CONFIG_CONTEXT *)MXS_MALLOC(sizeof(CONFIG_CONTEXT))) == NULL) { return 0; } - ptr->object = strdup(section); + ptr->object = MXS_STRDUP_A(section); ptr->parameters = NULL; ptr->next = cntxt->next; ptr->element = NULL; @@ -325,9 +325,8 @@ handler(void *userdata, const char *section, const char *name, const char *value char *tmp; int paramlen = strlen(p1->value) + strlen(value) + 2; - if ((tmp = realloc(p1->value, sizeof(char) * (paramlen))) == NULL) + if ((tmp = MXS_REALLOC(p1->value, sizeof(char) * (paramlen))) == NULL) { - MXS_ERROR("[%s] Memory allocation failed.", __FUNCTION__); return 0; } strcat(tmp, ","); @@ -338,19 +337,19 @@ handler(void *userdata, const char *section, const char *name, const char *value MXS_ERROR("[%s] Cleaning configuration parameter failed.", __FUNCTION__); return 0; } - free(tmp); + MXS_FREE(tmp); return 1; } p1 = p1->next; } - if ((param = (CONFIG_PARAMETER *)malloc(sizeof(CONFIG_PARAMETER))) == NULL) + if ((param = (CONFIG_PARAMETER *)MXS_MALLOC(sizeof(CONFIG_PARAMETER))) == NULL) { return 0; } - param->name = strdup(name); - param->value = strdup(value); + param->name = MXS_STRDUP_A(name); + param->value = MXS_STRDUP_A(value); param->next = ptr->parameters; param->qfd_param_type = UNDEFINED_TYPE; ptr->parameters = param; @@ -379,6 +378,10 @@ config_load(char *file) return false; } + /* Temporary - should use configuration values and test return value (bool) */ + dcb_pre_alloc(1000); + session_pre_alloc(250); + global_defaults(); feedback_defaults(); @@ -440,7 +443,7 @@ config_reload() if (gateway.version_string) { - free(gateway.version_string); + MXS_FREE(gateway.version_string); } global_defaults(); @@ -472,13 +475,12 @@ process_config_context(CONFIG_CONTEXT *context) int error_count = 0; HASHTABLE* monitorhash; - if ((monitorhash = hashtable_alloc(5, simple_str_hash, strcmp)) == NULL) + if ((monitorhash = hashtable_alloc(5, hashtable_item_strhash, hashtable_item_strcmp)) == NULL) { MXS_ERROR("Failed to allocate, monitor configuration check hashtable."); return 0; } - hashtable_memory_fns(monitorhash, (HASHMEMORYFN) strdup, NULL, - (HASHMEMORYFN) free, NULL); + hashtable_memory_fns(monitorhash, hashtable_item_strdup, NULL, hashtable_item_free, NULL); /** * Process the data and create the services and servers defined @@ -745,7 +747,7 @@ CONFIG_PARAMETER* config_clone_param( { CONFIG_PARAMETER* p2; - p2 = (CONFIG_PARAMETER*) malloc(sizeof(CONFIG_PARAMETER)); + p2 = (CONFIG_PARAMETER*) MXS_MALLOC(sizeof(CONFIG_PARAMETER)); if (p2 == NULL) { @@ -772,10 +774,10 @@ void free_config_parameter(CONFIG_PARAMETER* p1) { while (p1) { - free(p1->name); - free(p1->value); + MXS_FREE(p1->name); + MXS_FREE(p1->value); CONFIG_PARAMETER* p2 = p1->next; - free(p1); + MXS_FREE(p1); p1 = p2; } } @@ -793,10 +795,10 @@ free_config_context(CONFIG_CONTEXT *context) while (context) { - free(context->object); + MXS_FREE(context->object); free_config_parameter(context->parameters); obj = context->next; - free(context); + MXS_FREE(context); context = obj; } } @@ -974,7 +976,73 @@ handle_global_item(const char *name, const char *value) } else if (strcmp(name, "query_classifier_args") == 0) { - gateway.qc_args = strdup(value); + gateway.qc_args = MXS_STRDUP_A(value); + } + else if (strcmp(name, "log_throttling") == 0) + { + if (*value == 0) + { + MXS_LOG_THROTTLING throttling = { 0, 0, 0 }; + + mxs_log_set_throttling(&throttling); + } + else + { + char *v = MXS_STRDUP_A(value); + + char *count = v; + char *window_ms = NULL; + char *suppress_ms = NULL; + + window_ms = strchr(count, ','); + if (window_ms) + { + *window_ms = 0; + ++window_ms; + + suppress_ms = strchr(window_ms, ','); + if (suppress_ms) + { + *suppress_ms = 0; + ++suppress_ms; + } + } + + if (!count || !window_ms || !suppress_ms) + { + MXS_ERROR("Invalid value for the `log_throttling` configuration entry: \"%s\". " + "No throttling will now be performed.", value); + MXS_NOTICE("The format of the value for 'log_throttling' is \"X, Y, Z\", where " + "X is the maximum number of times a particular error can be logged " + "in the time window of Y milliseconds, before the logging is suppressed " + "for Z milliseconds."); + } + else + { + int c = atoi(count); + int w = atoi(window_ms); + int s = atoi(suppress_ms); + + if ((c >= 0) && (w >= 0) && (s >= 0)) + { + MXS_LOG_THROTTLING throttling; + throttling.count = c; + throttling.window_ms = w; + throttling.suppress_ms = s; + + mxs_log_set_throttling(&throttling); + } + else + { + MXS_ERROR("Invalid value for the `log_throttling` configuration entry: \"%s\". " + "No throttling will now be performed.", value); + MXS_NOTICE("The configuration entry 'log_throttling' requires as value three positive " + "integers (or 0)."); + } + } + + MXS_FREE(v); + } } else { @@ -1007,10 +1075,10 @@ free_ssl_structure(SSL_LISTENER *ssl) if (ssl) { SSL_CTX_free(ssl->ctx); - free(ssl->ssl_key); - free(ssl->ssl_cert); - free(ssl->ssl_ca_cert); - free(ssl); + MXS_FREE(ssl->ssl_key); + MXS_FREE(ssl->ssl_cert); + MXS_FREE(ssl->ssl_ca_cert); + MXS_FREE(ssl); } } @@ -1035,7 +1103,7 @@ make_ssl_structure (CONFIG_CONTEXT *obj, bool require_cert, int *error_count) { if (!strcmp(ssl, "required")) { - if ((new_ssl = calloc(1, sizeof(SSL_LISTENER))) == NULL) + if ((new_ssl = MXS_CALLOC(1, sizeof(SSL_LISTENER))) == NULL) { return NULL; } @@ -1133,7 +1201,7 @@ make_ssl_structure (CONFIG_CONTEXT *obj, bool require_cert, int *error_count) return new_ssl; } *error_count += local_errors; - free(new_ssl); + MXS_FREE(new_ssl); } else if (strcmp(ssl, "disabled") != 0) { @@ -1160,11 +1228,11 @@ handle_feedback_item(const char *name, const char *value) } else if (strcmp(name, "feedback_user_info") == 0) { - feedback.feedback_user_info = strdup(value); + feedback.feedback_user_info = MXS_STRDUP_A(value); } else if (strcmp(name, "feedback_url") == 0) { - feedback.feedback_url = strdup(value); + feedback.feedback_url = MXS_STRDUP_A(value); } if (strcmp(name, "feedback_timeout") == 0) { @@ -1197,7 +1265,7 @@ global_defaults() gateway.auth_write_timeout = DEFAULT_AUTH_WRITE_TIMEOUT; if (version_string != NULL) { - gateway.version_string = strdup(version_string); + gateway.version_string = MXS_STRDUP_A(version_string); } else { @@ -1229,7 +1297,7 @@ global_defaults() } else { - strncpy(gateway.sysname, uname_data.sysname, _SYSNAME_STR_LENGTH); + strcpy(gateway.sysname, uname_data.sysname); } /* query_classifier */ @@ -1340,9 +1408,9 @@ process_config_update(CONFIG_CONTEXT *context) { if (service->version_string) { - free(service->version_string); + MXS_FREE(service->version_string); } - service->version_string = strdup(version_string); + service->version_string = MXS_STRDUP_A(version_string); } if (user && auth) @@ -1762,12 +1830,12 @@ config_get_ifaddr(unsigned char *output) /** * Get the linux distribution info * - * @param release The allocated buffer where - * the found distribution is copied into. - * @return 1 on success, 0 on failure + * @param release The buffer where the found distribution is copied. + * Assumed to be at least _RELEASE_STR_LENGTH bytes. * + * @return 1 on success, 0 on failure */ -int +static int config_get_release_string(char* release) { const char *masks[] = @@ -1779,8 +1847,6 @@ config_get_release_string(char* release) bool have_distribution; char distribution[_RELEASE_STR_LENGTH] = ""; int fd; - int i; - char *to; have_distribution = false; @@ -1805,7 +1871,7 @@ config_get_release_string(char* release) { end = distribution + len; } - found += 20; + found += 20; // strlen("DISTRIB_DESCRIPTION=") if (*found == '"' && end[-1] == '"') { @@ -1814,10 +1880,10 @@ config_get_release_string(char* release) } *end = 0; - to = strcpy(distribution, "lsb: "); + char *to = strcpy(distribution, "lsb: "); memmove(to, found, end - found + 1 < INT_MAX ? end - found + 1 : INT_MAX); - strncpy(release, to, _RELEASE_STR_LENGTH); + strcpy(release, to); return 1; } @@ -1825,7 +1891,7 @@ config_get_release_string(char* release) } /* if not an LSB-compliant distribution */ - for (i = 0; !have_distribution && i < 4; i++) + for (int i = 0; !have_distribution && i < 4; i++) { glob_t found; char *new_to; @@ -1952,18 +2018,23 @@ unsigned long config_get_gateway_id() void config_add_param(CONFIG_CONTEXT* obj, char* key, char* value) { - CONFIG_PARAMETER* nptr = malloc(sizeof(CONFIG_PARAMETER)); + key = MXS_STRDUP(key); + value = MXS_STRDUP(value); - if (nptr == NULL) + CONFIG_PARAMETER* param = (CONFIG_PARAMETER *)MXS_MALLOC(sizeof(CONFIG_PARAMETER)); + + if (!key || !value || !param) { - MXS_ERROR("Memory allocation failed when adding configuration parameters."); + MXS_FREE(key); + MXS_FREE(value); + MXS_FREE(param); return; } - nptr->name = strdup(key); - nptr->value = strdup(value); - nptr->next = obj->parameters; - obj->parameters = nptr; + param->name = key; + param->value = value; + param->next = obj->parameters; + obj->parameters = param; } /** * Return the pointer to the global options for MaxScale. @@ -1986,17 +2057,16 @@ bool config_has_duplicate_sections(const char* config) const int table_size = 10; int errcode; PCRE2_SIZE erroffset; - HASHTABLE *hash = hashtable_alloc(table_size, simple_str_hash, strcmp); + HASHTABLE *hash = hashtable_alloc(table_size, hashtable_item_strhash, hashtable_item_strcmp); pcre2_code *re = pcre2_compile((PCRE2_SPTR) "^\\s*\\[(.+)\\]\\s*$", PCRE2_ZERO_TERMINATED, 0, &errcode, &erroffset, NULL); pcre2_match_data *mdata = NULL; int size = 1024; - char *buffer = malloc(size * sizeof(char)); + char *buffer = MXS_MALLOC(size * sizeof(char)); if (buffer && hash && re && (mdata = pcre2_match_data_create_from_pattern(re, NULL))) { - hashtable_memory_fns(hash, (HASHMEMORYFN) strdup, NULL, - (HASHMEMORYFN) free, NULL); + hashtable_memory_fns(hash, hashtable_item_strdup, NULL, hashtable_item_free, NULL); FILE* file = fopen(config, "r"); if (file) @@ -2036,15 +2106,15 @@ bool config_has_duplicate_sections(const char* config) } else { - MXS_ERROR("Failed to allocate enough memory when checking" - " for duplicate sections in configuration file."); + MXS_OOM_MESSAGE("Failed to allocate enough memory when checking" + " for duplicate sections in configuration file."); rval = true; } hashtable_free(hash); pcre2_code_free(re); pcre2_match_data_free(mdata); - free(buffer); + MXS_FREE(buffer); return rval; } @@ -2078,7 +2148,7 @@ int maxscale_getline(char** dest, int* size, FILE* file) { if (*size <= offset) { - char* tmp = (char*) realloc(destptr, *size * 2); + char* tmp = (char*) MXS_REALLOC(destptr, *size * 2); if (tmp) { destptr = tmp; @@ -2086,9 +2156,6 @@ int maxscale_getline(char** dest, int* size, FILE* file) } else { - MXS_ERROR("Failed to reallocate memory from %d" - " bytes to %d bytes when reading from file.", - *size, *size * 2); destptr[offset - 1] = '\0'; *dest = destptr; return -1; @@ -2301,23 +2368,24 @@ int create_new_service(CONFIG_CONTEXT *obj) /** Add the 5.5.5- string to the start of the version string if * the version string starts with "10.". * This mimics MariaDB 10.0 replication which adds 5.5.5- for backwards compatibility. */ - if (strncmp(version_string, "10.", 3) == 0) + if (version_string[0] != '5') { size_t len = strlen(version_string) + strlen("5.5.5-") + 1; - service->version_string = malloc(len); + service->version_string = MXS_MALLOC(len); + MXS_ABORT_IF_NULL(service->version_string); strcpy(service->version_string, "5.5.5-"); strcat(service->version_string, version_string); } else { - service->version_string = strdup(version_string); + service->version_string = MXS_STRDUP_A(version_string); } } else { if (gateway.version_string) { - service->version_string = strdup(gateway.version_string); + service->version_string = MXS_STRDUP_A(gateway.version_string); } } @@ -2707,7 +2775,8 @@ int create_new_listener(CONFIG_CONTEXT *obj, bool startnow) } else { - serviceAddProtocol(service, protocol, socket, 0, authenticator, ssl_info); + serviceAddProtocol(service, obj->object, protocol, socket, 0, + authenticator, ssl_info); if (startnow) { serviceStartProtocol(service, protocol, 0); @@ -2727,7 +2796,8 @@ int create_new_listener(CONFIG_CONTEXT *obj, bool startnow) } else { - serviceAddProtocol(service, protocol, address, atoi(port), authenticator, ssl_info); + serviceAddProtocol(service, obj->object, protocol, address, + atoi(port), authenticator, ssl_info); if (startnow) { serviceStartProtocol(service, protocol, atoi(port)); diff --git a/server/core/dbusers.c b/server/core/dbusers.c index 0ebd8f8cd..245d92f8d 100644 --- a/server/core/dbusers.c +++ b/server/core/dbusers.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -50,6 +50,7 @@ #include #include #include +#include /** Don't include the root user */ #define USERS_QUERY_NO_ROOT " AND user.user NOT IN ('root')" @@ -121,16 +122,16 @@ MaxScale authentication will proceed without including database permissions. \ See earlier error messages for user '%s' for more information." -static int add_databases(SERVICE *service, MYSQL *con); +static int add_databases(SERV_LISTENER *listener, MYSQL *con); static int add_wildcard_users(USERS *users, char* name, char* host, char* password, char* anydb, char* db, HASHTABLE* hash); static void *dbusers_keyread(int fd); static int dbusers_keywrite(int fd, void *key); static void *dbusers_valueread(int fd); static int dbusers_valuewrite(int fd, void *value); -static int get_all_users(SERVICE *service, USERS *users); -static int get_databases(SERVICE *, MYSQL *); -static int get_users(SERVICE *service, USERS *users); +static int get_all_users(SERV_LISTENER *listener, USERS *users); +static int get_databases(SERV_LISTENER *listener, MYSQL *users); +static int get_users(SERV_LISTENER *listener, USERS *users); static MYSQL *gw_mysql_init(void); static int gw_mysql_set_timeouts(MYSQL* handle); static bool host_has_singlechar_wildcard(const char *host); @@ -143,10 +144,10 @@ static int resource_add(HASHTABLE *, char *, char *); static HASHTABLE *resource_alloc(); static void *resource_fetch(HASHTABLE *, char *); static void resource_free(HASHTABLE *resource); -static int uh_cmpfun(void* v1, void* v2); -static int uh_hfun(void* key); -static void *uh_keydup(void* key); -static void uh_keyfree(void* key); +static int uh_cmpfun(const void* v1, const void* v2); +static int uh_hfun(const void* key); +static MYSQL_USER_HOST *uh_keydup(const MYSQL_USER_HOST* key); +static void uh_keyfree(MYSQL_USER_HOST* key); static int wildcard_db_grant(char* str); /** @@ -243,9 +244,9 @@ static bool host_matches_singlechar_wildcard(const char* user, const char* wild) * @return -1 on any error or the number of users inserted (0 means no users at all) */ int -load_mysql_users(SERVICE *service) +load_mysql_users(SERV_LISTENER *listener) { - return get_users(service, service->users); + return get_users(listener, listener->users); } /** @@ -256,7 +257,7 @@ load_mysql_users(SERVICE *service) * @return -1 on any error or the number of users inserted (0 means no users at all) */ int -reload_mysql_users(SERVICE *service) +reload_mysql_users(SERV_LISTENER *listener) { int i; USERS *newusers, *oldusers; @@ -267,20 +268,16 @@ reload_mysql_users(SERVICE *service) return 0; } - oldresources = service->resources; + spinlock_acquire(&listener->lock); - i = get_users(service, newusers); + oldresources = listener->resources; + i = get_users(listener, newusers); + oldusers = listener->users; + listener->users = newusers; - spinlock_acquire(&service->spin); - oldusers = service->users; + spinlock_release(&listener->lock); - service->users = newusers; - - spinlock_release(&service->spin); - - /* free the old table */ users_free(oldusers); - /* free old resources */ resource_free(oldresources); return i; @@ -295,7 +292,7 @@ reload_mysql_users(SERVICE *service) * @return -1 on any error or the number of users inserted (0 means no users at all) */ int -replace_mysql_users(SERVICE *service) +replace_mysql_users(SERV_LISTENER *listener) { int i; USERS *newusers, *oldusers; @@ -306,21 +303,22 @@ replace_mysql_users(SERVICE *service) return -1; } - oldresources = service->resources; + spinlock_acquire(&listener->lock); + oldresources = listener->resources; /* load db users ad db grants */ - i = get_users(service, newusers); + i = get_users(listener, newusers); if (i <= 0) { users_free(newusers); /* restore resources */ - service->resources = oldresources; + listener->resources = oldresources; + spinlock_release(&listener->lock); return i; } - spinlock_acquire(&service->spin); - oldusers = service->users; + oldusers = listener->users; /* digest compare */ if (oldusers != NULL && memcmp(oldusers->cksum, newusers->cksum, @@ -339,14 +337,14 @@ replace_mysql_users(SERVICE *service) /* replace the service with effective new data */ MXS_DEBUG("%lu [replace_mysql_users] users' tables replaced, checksum differs", pthread_self()); - service->users = newusers; + listener->users = newusers; } + spinlock_release(&listener->lock); + /* free old resources */ resource_free(oldresources); - spinlock_release(&service->spin); - if (i && oldusers) { /* free the old table */ @@ -433,10 +431,10 @@ int add_mysql_users_with_host_ipv4(USERS *users, const char *user, const char *h /* prepare the user@host data struct */ memset(&serv_addr, 0, sizeof(serv_addr)); - memset(&key, '\0', sizeof(key)); + memset(&key, 0, sizeof(key)); /* set user */ - key.user = strdup(user); + key.user = MXS_STRDUP(user); if (key.user == NULL) { @@ -454,7 +452,8 @@ int add_mysql_users_with_host_ipv4(USERS *users, const char *user, const char *h { if (db != NULL) { - key.resource = strdup(db); + key.resource = MXS_STRDUP(db); + MXS_ABORT_IF_NULL(key.resource); } else { @@ -463,7 +462,8 @@ int add_mysql_users_with_host_ipv4(USERS *users, const char *user, const char *h } else { - key.resource = strdup(""); + key.resource = MXS_STRDUP(""); + MXS_ABORT_IF_NULL(key.resource); } } @@ -519,8 +519,8 @@ int add_mysql_users_with_host_ipv4(USERS *users, const char *user, const char *h } } - free(key.user); - free(key.resource); + MXS_FREE(key.user); + MXS_FREE(key.resource); return ret; } @@ -534,8 +534,9 @@ int add_mysql_users_with_host_ipv4(USERS *users, const char *user, const char *h * @return -1 on any error or the number of users inserted (0 means no users at all) */ static int -add_databases(SERVICE *service, MYSQL *con) +add_databases(SERV_LISTENER *listener, MYSQL *con) { + SERVICE *service = listener->service; MYSQL_ROW row; MYSQL_RES *result = NULL; char *service_user = NULL; @@ -620,7 +621,7 @@ add_databases(SERVICE *service, MYSQL *con) /* insert key and value "" */ while ((row = mysql_fetch_row(result))) { - if (resource_add(service->resources, row[0], "")) + if (resource_add(listener->resources, row[0], "")) { MXS_DEBUG("%s: Adding database %s to the resouce hash.", service->name, row[0]); } @@ -640,8 +641,9 @@ add_databases(SERVICE *service, MYSQL *con) * @return -1 on any error or the number of users inserted (0 means no users at all) */ static int -get_databases(SERVICE *service, MYSQL *con) +get_databases(SERV_LISTENER *listener, MYSQL *con) { + SERVICE *service = listener->service; MYSQL_ROW row; MYSQL_RES *result = NULL; char *service_user = NULL; @@ -724,13 +726,13 @@ get_databases(SERVICE *service, MYSQL *con) } /* Now populate service->resources hashatable with db names */ - service->resources = resource_alloc(); + listener->resources = resource_alloc(); /* insert key and value "" */ while ((row = mysql_fetch_row(result))) { MXS_DEBUG("%s: Adding database %s to the resouce hash.", service->name, row[0]); - resource_add(service->resources, row[0], ""); + resource_add(listener->resources, row[0], ""); } mysql_free_result(result); @@ -747,8 +749,9 @@ get_databases(SERVICE *service, MYSQL *con) * @return -1 on any error or the number of users inserted */ static int -get_all_users(SERVICE *service, USERS *users) +get_all_users(SERV_LISTENER *listener, USERS *users) { + SERVICE *service = listener->service; MYSQL *con = NULL; MYSQL_ROW row; MYSQL_RES *result = NULL; @@ -782,7 +785,8 @@ get_all_users(SERVICE *service, USERS *users) } dpwd = decryptPassword(service_passwd); - final_data = (char*) malloc(sizeof(char)); + final_data = (char*) MXS_MALLOC(sizeof(char)); + MXS_ABORT_IF_NULL(final_data); *final_data = '\0'; /** @@ -797,7 +801,7 @@ get_all_users(SERVICE *service, USERS *users) goto cleanup; } - service->resources = resource_alloc(); + listener->resources = resource_alloc(); while (server != NULL) { @@ -837,7 +841,7 @@ get_all_users(SERVICE *service, USERS *users) goto cleanup; } - add_databases(service, con); + add_databases(listener, con); mysql_close(con); server = server->next; } @@ -1031,14 +1035,10 @@ get_all_users(SERVICE *service, USERS *users) goto cleanup; } - users_data = (char *) calloc(nusers, (users_data_row_len * sizeof(char)) + 1); + users_data = (char *) MXS_CALLOC(nusers, (users_data_row_len * sizeof(char)) + 1); if (users_data == NULL) { - char errbuf[STRERROR_BUFLEN]; - MXS_ERROR("Memory allocation for user data failed due to %d, %s.", - errno, - strerror_r(errno, errbuf, sizeof(errbuf))); mysql_free_result(result); mysql_close(con); @@ -1145,6 +1145,7 @@ get_all_users(SERVICE *service, USERS *users) else if (row[5]) { strncpy(dbgrant, row[5], MYSQL_DATABASE_MAXLEN); + dbgrant[MYSQL_DATABASE_MAXLEN] = 0; } } @@ -1183,17 +1184,17 @@ get_all_users(SERVICE *service, USERS *users) mysql_free_result(result); mysql_close(con); - if ((tmp = realloc(final_data, (strlen(final_data) + strlen(users_data) - + 1) * sizeof(char))) == NULL) + if ((tmp = MXS_REALLOC(final_data, (strlen(final_data) + strlen(users_data) + + 1) * sizeof(char))) == NULL) { - free(users_data); + MXS_FREE(users_data); goto cleanup; } final_data = tmp; strcat(final_data, users_data); - free(users_data); + MXS_FREE(users_data); if (service->users_from_all) { @@ -1217,8 +1218,8 @@ get_all_users(SERVICE *service, USERS *users) } cleanup: - free(dpwd); - free(final_data); + MXS_FREE(dpwd); + MXS_FREE(final_data); return total_users; } @@ -1232,8 +1233,9 @@ cleanup: * @return -1 on any error or the number of users inserted */ static int -get_users(SERVICE *service, USERS *users) +get_users(SERV_LISTENER *listener, USERS *users) { + SERVICE *service = listener->service; MYSQL *con = NULL; MYSQL_ROW row; MYSQL_RES *result = NULL; @@ -1264,7 +1266,7 @@ get_users(SERVICE *service, USERS *users) if (service->users_from_all) { - return get_all_users(service, users); + return get_all_users(listener, users); } con = gw_mysql_init(); @@ -1290,7 +1292,7 @@ get_users(SERVICE *service, USERS *users) if (service->svc_do_shutdown) { - free(dpwd); + MXS_FREE(dpwd); mysql_close(con); return -1; } @@ -1341,7 +1343,7 @@ get_users(SERVICE *service, USERS *users) if (service->svc_do_shutdown) { - free(dpwd); + MXS_FREE(dpwd); mysql_close(con); return -1; } @@ -1353,7 +1355,7 @@ get_users(SERVICE *service, USERS *users) } } - free(dpwd); + MXS_FREE(dpwd); if (server == NULL) { @@ -1494,13 +1496,10 @@ get_users(SERVICE *service, USERS *users) return -1; } - users_data = (char *) calloc(nusers, (users_data_row_len * sizeof(char)) + 1); + users_data = (char *) MXS_CALLOC(nusers, (users_data_row_len * sizeof(char)) + 1); if (users_data == NULL) { - char errbuf[STRERROR_BUFLEN]; - MXS_ERROR("Memory allocation for user data failed due to %d, %s.", - errno, strerror_r(errno, errbuf, sizeof(errbuf))); mysql_free_result(result); mysql_close(con); return -1; @@ -1509,13 +1508,13 @@ get_users(SERVICE *service, USERS *users) if (db_grants) { /* load all mysql database names */ - dbnames = get_databases(service, con); + dbnames = get_databases(listener, con); MXS_DEBUG("Loaded %d MySQL Database Names for service [%s]", dbnames, service->name); } else { - service->resources = NULL; + listener->resources = NULL; } while ((row = mysql_fetch_row(result))) @@ -1617,6 +1616,7 @@ get_users(SERVICE *service, USERS *users) else if (row[5]) { strncpy(dbgrant, row[5], MYSQL_DATABASE_MAXLEN); + dbgrant[MYSQL_DATABASE_MAXLEN] = 0; } } @@ -1669,7 +1669,7 @@ get_users(SERVICE *service, USERS *users) service->localhost_match_wildcard_host = anon_user ? 0 : 1; } - free(users_data); + MXS_FREE(users_data); mysql_free_result(result); mysql_close(con); @@ -1686,7 +1686,7 @@ mysql_users_alloc() { USERS *rval; - if ((rval = calloc(1, sizeof(USERS))) == NULL) + if ((rval = MXS_CALLOC(1, sizeof(USERS))) == NULL) { return NULL; } @@ -1694,7 +1694,7 @@ mysql_users_alloc() if ((rval->data = hashtable_alloc(USERS_HASHTABLE_DEFAULT_SIZE, uh_hfun, uh_cmpfun)) == NULL) { - free(rval); + MXS_FREE(rval); return NULL; } @@ -1704,9 +1704,9 @@ mysql_users_alloc() /* the key is handled by uh_keydup/uh_keyfree. * the value is a (char *): it's handled by strdup/free */ - hashtable_memory_fns(rval->data, (HASHMEMORYFN) uh_keydup, - (HASHMEMORYFN) strdup, (HASHMEMORYFN) uh_keyfree, - (HASHMEMORYFN) free); + hashtable_memory_fns(rval->data, + (HASHCOPYFN) uh_keydup, hashtable_item_strdup, + (HASHFREEFN) uh_keyfree, hashtable_item_free); return rval; } @@ -1761,9 +1761,9 @@ char *mysql_users_fetch(USERS *users, MYSQL_USER_HOST *key) * @return The hash key */ -static int uh_hfun(void* key) +static int uh_hfun(const void* key) { - MYSQL_USER_HOST *hu = (MYSQL_USER_HOST *) key; + const MYSQL_USER_HOST *hu = (const MYSQL_USER_HOST *) key; if (key == NULL || hu == NULL || hu->user == NULL) { @@ -1785,10 +1785,10 @@ static int uh_hfun(void* key) * @return The compare value */ -static int uh_cmpfun(void* v1, void* v2) +static int uh_cmpfun(const void* v1, const void* v2) { - MYSQL_USER_HOST *hu1 = (MYSQL_USER_HOST *) v1; - MYSQL_USER_HOST *hu2 = (MYSQL_USER_HOST *) v2; + const MYSQL_USER_HOST *hu1 = (const MYSQL_USER_HOST *) v1; + const MYSQL_USER_HOST *hu2 = (const MYSQL_USER_HOST *) v2; if (v1 == NULL || v2 == NULL) { @@ -1888,44 +1888,30 @@ static int uh_cmpfun(void* v1, void* v2) * @param key The key value, i.e. username@host ip4/ip6 data */ -static void *uh_keydup(void* key) +static MYSQL_USER_HOST *uh_keydup(const MYSQL_USER_HOST* key) { - MYSQL_USER_HOST *rval = (MYSQL_USER_HOST *) calloc(1, sizeof(MYSQL_USER_HOST)); - MYSQL_USER_HOST *current_key = (MYSQL_USER_HOST *) key; - - if (key == NULL || rval == NULL || current_key == NULL || current_key->user == NULL) + if ((key == NULL) || (key->user == NULL)) { - if (rval) - { - free(rval); - } - return NULL; } - rval->user = strdup(current_key->user); + MYSQL_USER_HOST *rval = (MYSQL_USER_HOST *) MXS_CALLOC(1, sizeof(MYSQL_USER_HOST)); + char* user = MXS_STRDUP(key->user); + char* resource = key->resource ? MXS_STRDUP(key->resource) : NULL; - if (rval->user == NULL) + if (!user || !rval || (key->resource && !resource)) { - free(rval); + MXS_FREE(rval); + MXS_FREE(user); + MXS_FREE(resource); return NULL; } - ss_dassert(strnlen(rval->hostname, MYSQL_HOST_MAXLEN + 1) <= MYSQL_HOST_MAXLEN); - strncpy(rval->hostname, current_key->hostname, MYSQL_HOST_MAXLEN); - rval->hostname[MYSQL_HOST_MAXLEN] = '\0'; - - memcpy(&rval->ipv4, ¤t_key->ipv4, sizeof(struct sockaddr_in)); - memcpy(&rval->netmask, ¤t_key->netmask, sizeof(int)); - - if (current_key->resource) - { - rval->resource = strdup(current_key->resource); - } - else - { - rval->resource = NULL; - } + rval->user = user; + rval->ipv4 = key->ipv4; + rval->netmask = key->netmask; + rval->resource = resource; + strcpy(rval->hostname, key->hostname); return (void *) rval; } @@ -1935,26 +1921,14 @@ static void *uh_keydup(void* key) * * @param key The key value, i.e. username@host ip4 data */ -static void uh_keyfree(void* key) +static void uh_keyfree(MYSQL_USER_HOST* key) { - MYSQL_USER_HOST *current_key = (MYSQL_USER_HOST *) key; - - if (key == NULL) + if (key) { - return; + MXS_FREE(key->user); + MXS_FREE(key->resource); + MXS_FREE(key); } - - if (current_key && current_key->user) - { - free(current_key->user); - } - - if (current_key && current_key->resource) - { - free(current_key->resource); - } - - free(key); } /** @@ -1979,7 +1953,7 @@ static char *mysql_format_user_entry(void *data) entry = (MYSQL_USER_HOST *) data; - mysql_user = (char *) calloc(mysql_user_len, sizeof(char)); + mysql_user = (char *) MXS_CALLOC(mysql_user_len, sizeof(char)); if (mysql_user == NULL) { @@ -2012,7 +1986,7 @@ static char *mysql_format_user_entry(void *data) } else if (entry->netmask == 32) { - strncpy(mysql_user, entry->user, MYSQL_USER_MAXLEN); + strcpy(mysql_user, entry->user); strcat(mysql_user, "@"); inet_ntop(AF_INET, &(entry->ipv4).sin_addr, mysql_user + strlen(mysql_user), INET_ADDRSTRLEN); @@ -2052,13 +2026,14 @@ resource_alloc() { HASHTABLE *resources; - if ((resources = hashtable_alloc(10, simple_str_hash, strcmp)) == NULL) + if ((resources = hashtable_alloc(10, hashtable_item_strhash, hashtable_item_strcmp)) == NULL) { return NULL; } - hashtable_memory_fns(resources, (HASHMEMORYFN) strdup, (HASHMEMORYFN) strdup, - (HASHMEMORYFN) free, (HASHMEMORYFN) free); + hashtable_memory_fns(resources, + hashtable_item_strdup, hashtable_item_strdup, + hashtable_item_free, hashtable_item_free); return resources; } @@ -2113,7 +2088,7 @@ static int normalize_hostname(const char *input_host, char *output_host) output_host[0] = 0; bytes = 0; - tmp = strdup(input_host); + tmp = MXS_STRDUP(input_host); if (tmp == NULL) { @@ -2177,7 +2152,7 @@ static int normalize_hostname(const char *input_host, char *output_host) strcpy(output_host, input_host); } - free(tmp); + MXS_FREE(tmp); return netmask; } @@ -2352,7 +2327,7 @@ dbusers_keyread(int fd) { MYSQL_USER_HOST *dbkey; - if ((dbkey = (MYSQL_USER_HOST *) malloc(sizeof(MYSQL_USER_HOST))) == NULL) + if ((dbkey = (MYSQL_USER_HOST *) MXS_MALLOC(sizeof(MYSQL_USER_HOST))) == NULL) { return NULL; } @@ -2362,54 +2337,54 @@ dbusers_keyread(int fd) int user_size; if (read(fd, &user_size, sizeof(user_size)) != sizeof(user_size)) { - free(dbkey); + MXS_FREE(dbkey); return NULL; } - if ((dbkey->user = (char *) malloc(user_size + 1)) == NULL) + if ((dbkey->user = (char *) MXS_MALLOC(user_size + 1)) == NULL) { - free(dbkey); + MXS_FREE(dbkey); return NULL; } if (read(fd, dbkey->user, user_size) != user_size) { - free(dbkey->user); - free(dbkey); + MXS_FREE(dbkey->user); + MXS_FREE(dbkey); return NULL; } dbkey->user[user_size] = 0; // NULL Terminate if (read(fd, &dbkey->ipv4, sizeof(dbkey->ipv4)) != sizeof(dbkey->ipv4)) { - free(dbkey->user); - free(dbkey); + MXS_FREE(dbkey->user); + MXS_FREE(dbkey); return NULL; } if (read(fd, &dbkey->netmask, sizeof(dbkey->netmask)) != sizeof(dbkey->netmask)) { - free(dbkey->user); - free(dbkey); + MXS_FREE(dbkey->user); + MXS_FREE(dbkey); return NULL; } int res_size; if (read(fd, &res_size, sizeof(res_size)) != sizeof(res_size)) { - free(dbkey->user); - free(dbkey); + MXS_FREE(dbkey->user); + MXS_FREE(dbkey); return NULL; } else if (res_size != -1) { - if ((dbkey->resource = (char *) malloc(res_size + 1)) == NULL) + if ((dbkey->resource = (char *) MXS_MALLOC(res_size + 1)) == NULL) { - free(dbkey->user); - free(dbkey); + MXS_FREE(dbkey->user); + MXS_FREE(dbkey); return NULL; } if (read(fd, dbkey->resource, res_size) != res_size) { - free(dbkey->resource); - free(dbkey->user); - free(dbkey); + MXS_FREE(dbkey->resource); + MXS_FREE(dbkey->user); + MXS_FREE(dbkey); return NULL; } dbkey->resource[res_size] = 0; // NULL Terminate @@ -2437,13 +2412,13 @@ dbusers_valueread(int fd) { return NULL; } - if ((value = (char *) malloc(tmp + 1)) == NULL) + if ((value = (char *) MXS_MALLOC(tmp + 1)) == NULL) { return NULL; } if (read(fd, value, tmp) != tmp) { - free(value); + MXS_FREE(value); return NULL; } value[tmp] = 0; @@ -2523,7 +2498,7 @@ static int add_wildcard_users(USERS *users, char* name, char* host, char* passwo return 0; } - if ((restr = malloc(sizeof(char) * strlen(db) * 2)) == NULL) + if ((restr = MXS_MALLOC(sizeof(char) * strlen(db) * 2)) == NULL) { return 0; } @@ -2535,7 +2510,7 @@ static int add_wildcard_users(USERS *users, char* name, char* host, char* passwo if (ptr == NULL) { - free(restr); + MXS_FREE(restr); return 0; } @@ -2553,7 +2528,7 @@ static int add_wildcard_users(USERS *users, char* name, char* host, char* passwo regerror(err, &re, errbuf, 1024); MXS_ERROR("Failed to compile regex when resolving wildcard database grants: %s", errbuf); - free(restr); + MXS_FREE(restr); return 0; } @@ -2570,7 +2545,7 @@ static int add_wildcard_users(USERS *users, char* name, char* host, char* passwo hashtable_iterator_free(iter); regfree(&re); - free(restr); + MXS_FREE(restr); return rval; } diff --git a/server/core/dcb.c b/server/core/dcb.c index 43b47ae8f..0828e2d6d 100644 --- a/server/core/dcb.c +++ b/server/core/dcb.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -59,6 +59,7 @@ * 07/02/2016 Martin Brampton Make dcb_read_SSL & dcb_create_SSL internal, * further small SSL logic changes * 31/05/2016 Martin Brampton Implement connection throttling + * 27/06/2016 Martin Brampton Implement list manager to manage DCB memory * * @endverbatim */ @@ -69,6 +70,7 @@ #include #include #include +#include #include #include #include @@ -88,24 +90,25 @@ #include #include #include +#include + +/* The list of all DCBs */ +static LIST_CONFIG DCBlist = +{LIST_TYPE_RECYCLABLE, sizeof(DCB), SPINLOCK_INIT}; + +/* A DCB with null values, used for initialization */ +static DCB dcb_initialized = DCB_INIT; -static DCB *allDCBs = NULL; /* Diagnostics need a list of DCBs */ -static DCB *lastDCB = NULL; -static DCB *wasfreeDCB = NULL; -static int freeDCBcount = 0; -static int nDCBs = 0; -static int maxDCBs = 0; static DCB *zombies = NULL; static int nzombies = 0; static int maxzombies = 0; -static SPINLOCK dcbspin = SPINLOCK_INIT; static SPINLOCK zombiespin = SPINLOCK_INIT; +static void dcb_initialize(void *dcb); static void dcb_final_free(DCB *dcb); static void dcb_call_callback(DCB *dcb, DCB_REASON reason); static int dcb_null_write(DCB *dcb, GWBUF *buf); static int dcb_null_auth(DCB *dcb, SERVER *server, SESSION *session, GWBUF *buf); -static inline int dcb_isvalid_nolock(DCB *dcb); static inline DCB * dcb_find_in_list(DCB *dcb); static inline void dcb_process_victim_queue(DCB *listofdcb); static void dcb_stop_polling_and_shutdown (DCB *dcb); @@ -176,12 +179,49 @@ dcb_get_zombies(void) return zombies; } +/* + * @brief Pre-allocate memory for a number of DCBs + * + * @param The number of DCBs to be pre-allocated + */ +bool +dcb_pre_alloc(int number) +{ + return list_pre_alloc(&DCBlist, number, dcb_initialize); +} + /** - * Allocate or recycle a new DCB. + * @brief Initialize a DCB + * + * This routine puts initial values into the fields of the DCB pointed to + * by the parameter. The parameter has to be passed as void * because the + * function can be called by the generic list manager, which does not know + * the actual type of the list entries it handles. + * + * Most fields can be initialized by the assignment of the static + * initialized DCB. The exception is the bitmask. + * + * @param *dcb Pointer to the DCB to be initialized + */ +static void +dcb_initialize(void *dcb) +{ + *(DCB *)dcb = dcb_initialized; + bitmask_init(&((DCB *)dcb)->memdata.bitmask); +} + +/** + * @brief Allocate or recycle a new DCB. * * This routine performs the generic initialisation on the DCB before returning * the newly allocated or recycled DCB. * + * Most fields will be already initialized by the list manager, through the + * call to list_find_free, passing the DCB initialization function. + * + * Remaining fields are set from the given parameters, and then the DCB is + * flagged as ready for use. + * * @param dcb_role_t The role for the new DCB * @return An available DCB or NULL if none could be allocated. */ @@ -190,156 +230,18 @@ dcb_alloc(dcb_role_t role, SERV_LISTENER *listener) { DCB *newdcb; - spinlock_acquire(&dcbspin); - if ((newdcb = dcb_find_free()) == NULL) + if ((newdcb = (DCB *)list_find_free(&DCBlist, dcb_initialize)) == NULL) { - spinlock_release(&dcbspin); return NULL; } - nDCBs++; - if (nDCBs > maxDCBs) - { - maxDCBs = nDCBs; - } - spinlock_release(&dcbspin); - newdcb->dcb_chk_top = CHK_NUM_DCB; - newdcb->dcb_chk_tail = CHK_NUM_DCB; - - newdcb->dcb_errhandle_called = false; newdcb->dcb_role = role; - spinlock_init(&newdcb->dcb_initlock); - spinlock_init(&newdcb->writeqlock); - spinlock_init(&newdcb->delayqlock); - spinlock_init(&newdcb->authlock); - spinlock_init(&newdcb->cb_lock); - spinlock_init(&newdcb->pollinlock); - spinlock_init(&newdcb->polloutlock); - newdcb->pollinbusy = 0; - newdcb->readcheck = 0; - newdcb->polloutbusy = 0; - newdcb->writecheck = 0; - newdcb->fd = DCBFD_CLOSED; - - newdcb->evq.next = NULL; - newdcb->evq.prev = NULL; - newdcb->evq.pending_events = 0; - newdcb->evq.processing = 0; - spinlock_init(&newdcb->evq.eventqlock); - - memset(&newdcb->stats, 0, sizeof(DCBSTATS)); // Zero the statistics - newdcb->state = DCB_STATE_ALLOC; - bitmask_init(&newdcb->memdata.bitmask); - newdcb->writeqlen = 0; - newdcb->high_water = 0; - newdcb->low_water = 0; - newdcb->session = NULL; - newdcb->server = NULL; - newdcb->service = NULL; - newdcb->nextpersistent = NULL; - newdcb->persistentstart = 0; - newdcb->callbacks = NULL; - newdcb->data = NULL; - newdcb->listener = listener; - newdcb->ssl_state = SSL_HANDSHAKE_UNKNOWN; + newdcb->entry_is_ready = true; - newdcb->remote = NULL; - newdcb->user = NULL; - newdcb->flags = 0; return newdcb; } -/** - * Add a new DCB to the list of all DCBs. - * - * Must be called with the general DCB lock held. - * - * A pointer, lastDCB, is held to find the end of the list, and the new DCB - * is linked to the end of the list. The pointer, wasfreeDCB, that is used to - * search for a free DCB is initialised if not already set. There cannot be - * any free DCBs until this routine has been called at least once, so - * wasfreeDCB will not be referred to before it is initialised. - * - * @param dcb The DCB to be added to the list - */ -static void -dcb_add_to_all_list(DCB *dcb) -{ - if (allDCBs == NULL) - { - allDCBs = dcb; - } - else - { - lastDCB->next = dcb; - } - lastDCB = dcb; - if (NULL == wasfreeDCB) - { - wasfreeDCB = dcb; - } -} - -/** - * Find a free DCB or allocate memory for a new one. - * - * This routine looks to see whether there are free DCB memory areas. - * If not, new memory is allocated, if possible, and the new DCB is added to - * the list of all DCBs. - * - * Must be called with the general DCB lock held. - * - * @return An available DCB or NULL if none could be allocated. - */ -static DCB * -dcb_find_free() -{ - DCB *nextdcb; - int loopcount = 0; - - if (freeDCBcount <= 0) - { - DCB *newdcb; - if ((newdcb = calloc(1, sizeof(DCB))) == NULL) - { - return NULL; - } - newdcb->next = NULL; - dcb_add_to_all_list(newdcb); - newdcb->dcb_is_in_use = true; - return newdcb; - } - /* Starting at the last place a free DCB was found, loop through the */ - /* list of DCBs searching for one that is not in use. */ - while (wasfreeDCB->dcb_is_in_use) - { - wasfreeDCB = wasfreeDCB->next; - if (NULL == wasfreeDCB) - { - loopcount++; - if (loopcount > 1) - { - /* Shouldn't need to loop round more than once */ - MXS_ERROR("Find free DCB failed to find a free DCB even" - " though the free count was positive"); - return NULL; - } - wasfreeDCB = allDCBs; - } - } - /* Dropping out of the loop means we have found a DCB that is not in use */ - freeDCBcount--; - ss_dassert(freeDCBcount >= 0); - /* Clear the old data, then reset the list forward link */ - nextdcb = wasfreeDCB->next; - memset(wasfreeDCB, 0, sizeof(DCB)); - wasfreeDCB->next = nextdcb; - wasfreeDCB->dcb_is_in_use = true; - return wasfreeDCB; -} - - /** * Provided only for consistency, simply calls dcb_close to guarantee * safe disposal of a DCB @@ -362,23 +264,39 @@ dcb_free(DCB *dcb) DCB * dcb_clone(DCB *orig) { - DCB *clonedcb; + char *remote = orig->remote; - if ((clonedcb = dcb_alloc(orig->dcb_role, orig->listener))) + if (remote) + { + remote = MXS_STRDUP(remote); + if (!remote) + { + return NULL; + } + } + + char *user = orig->user; + if (user) + { + user = MXS_STRDUP(user); + if (!user) + { + MXS_FREE(remote); + return NULL; + } + } + + DCB *clonedcb = dcb_alloc(orig->dcb_role, orig->listener); + + if (clonedcb) { clonedcb->fd = DCBFD_CLOSED; clonedcb->flags |= DCBF_CLONE; clonedcb->state = orig->state; clonedcb->data = orig->data; clonedcb->ssl_state = orig->ssl_state; - if (orig->remote) - { - clonedcb->remote = strdup(orig->remote); - } - if (orig->user) - { - clonedcb->user = strdup(orig->user); - } + clonedcb->remote = remote; + clonedcb->user = user; clonedcb->protocol = orig->protocol; clonedcb->func.write = dcb_null_write; @@ -388,6 +306,12 @@ dcb_clone(DCB *orig) clonedcb->func.close = orig->func.close; clonedcb->func.auth = dcb_null_auth; } + else + { + MXS_FREE(remote); + MXS_FREE(user); + } + return clonedcb; } @@ -453,11 +377,11 @@ void dcb_free_all_memory(DCB *dcb) { DCB_CALLBACK *cb_dcb; - ss_dassert(dcb->dcb_is_in_use); + ss_dassert(dcb->entry_is_in_use); if (dcb->protocol && (!DCB_IS_CLONE(dcb))) { - free(dcb->protocol); + MXS_FREE(dcb->protocol); } if (dcb->data && dcb->authfunc.free && !DCB_IS_CLONE(dcb)) { @@ -466,15 +390,15 @@ dcb_free_all_memory(DCB *dcb) } if (dcb->protoname) { - free(dcb->protoname); + MXS_FREE(dcb->protoname); } if (dcb->remote) { - free(dcb->remote); + MXS_FREE(dcb->remote); } if (dcb->user) { - free(dcb->user); + MXS_FREE(dcb->user); } /* Clear write and read buffers */ @@ -498,7 +422,7 @@ dcb_free_all_memory(DCB *dcb) while ((cb_dcb = dcb->callbacks) != NULL) { dcb->callbacks = cb_dcb->next; - free(cb_dcb); + MXS_FREE(cb_dcb); } spinlock_release(&dcb->cb_lock); if (dcb->ssl) @@ -508,11 +432,7 @@ dcb_free_all_memory(DCB *dcb) bitmask_free(&dcb->memdata.bitmask); /* We never free the actual DCB, it is available for reuse*/ - spinlock_acquire(&dcbspin); - dcb->dcb_is_in_use = false; - freeDCBcount++; - nDCBs--; - spinlock_release(&dcbspin); + list_free_entry(&DCBlist, (list_entry_t *)dcb); } @@ -706,7 +626,17 @@ dcb_process_victim_queue(DCB *listofdcb) { if (dcb->protocol) { - atomic_add(&dcb->service->client_count, -1); + QUEUE_ENTRY conn_waiting; + if (mxs_dequeue(dcb->service->queued_connections, &conn_waiting)) + { + DCB *waiting_dcb = (DCB *)conn_waiting.queued_object; + waiting_dcb->state = DCB_STATE_WAITING; + poll_fake_read_event(waiting_dcb); + } + else + { + atomic_add(&dcb->service->client_count, -1); + } } } else @@ -852,7 +782,7 @@ dcb_connect(SERVER *server, SESSION *session, const char *protocol) return NULL; } memcpy(&(dcb->func), funcs, sizeof(GWPROTOCOL)); - dcb->protoname = strdup(protocol); + dcb->protoname = MXS_STRDUP_A(protocol); /** * Link dcb to session. Unlink is called in dcb_final_free @@ -1804,7 +1734,7 @@ dcb_close(DCB *dcb) user = session_getUser(dcb->session); if (user && strlen(user) && !dcb->user) { - dcb->user = strdup(user); + dcb->user = MXS_STRDUP_A(user); } } /*< @@ -1872,7 +1802,7 @@ dcb_maybe_add_persistent(DCB *dcb) while ((loopcallback = dcb->callbacks) != NULL) { dcb->callbacks = loopcallback->next; - free(loopcallback); + MXS_FREE(loopcallback); } spinlock_release(&dcb->cb_lock); spinlock_acquire(&dcb->server->persistlock); @@ -1909,10 +1839,6 @@ dcb_maybe_add_persistent(DCB *dcb) void printDCB(DCB *dcb) { - if (false == dcb->dcb_is_in_use) - { - return; - } printf("DCB: %p\n", (void *)dcb); printf("\tDCB state: %s\n", gw_dcb_state2string(dcb->state)); if (dcb->remote) @@ -1935,13 +1861,13 @@ printDCB(DCB *dcb) if (statusname) { printf("\tServer status: %s\n", statusname); - free(statusname); + MXS_FREE(statusname); } char *rolename = dcb_role_name(dcb); if (rolename) { printf("\tRole: %s\n", rolename); - free(rolename); + MXS_FREE(rolename); } printf("\tStatistics:\n"); printf("\t\tNo. of Reads: %d\n", @@ -1977,16 +1903,15 @@ spin_reporter(void *dcb, char *desc, int value) */ void printAllDCBs() { - DCB *dcb; + list_entry_t *current; - spinlock_acquire(&dcbspin); - dcb = allDCBs; - while (dcb) + current = list_start_iteration(&DCBlist); + + while (current) { - printDCB(dcb); - dcb = dcb->next; + printDCB((DCB *)current); + current = list_iterate(&DCBlist, current); } - spinlock_release(&dcbspin); } /** @@ -1998,7 +1923,7 @@ void printAllDCBs() void dprintOneDCB(DCB *pdcb, DCB *dcb) { - if (false == dcb->dcb_is_in_use) + if (false == dcb->entry_is_in_use) { return; } @@ -2047,13 +1972,13 @@ dprintOneDCB(DCB *pdcb, DCB *dcb) if (statusname) { dcb_printf(pdcb, "\tServer status: %s\n", statusname); - free(statusname); + MXS_FREE(statusname); } char *rolename = dcb_role_name(dcb); if (rolename) { dcb_printf(pdcb, "\tRole: %s\n", rolename); - free(rolename); + MXS_FREE(rolename); } if (!bitmask_isallclear(&dcb->memdata.bitmask)) { @@ -2061,7 +1986,7 @@ dprintOneDCB(DCB *pdcb, DCB *dcb) if (bitmasktext) { dcb_printf(pdcb, "\tBitMask: %s\n", bitmasktext); - free(bitmasktext); + MXS_FREE(bitmasktext); } } dcb_printf(pdcb, "\tStatistics:\n"); @@ -2084,6 +2009,18 @@ dprintOneDCB(DCB *pdcb, DCB *dcb) dcb_printf(pdcb, "\t\tAdded to persistent pool: %s\n", buff); } } + +/* + * @brief Print DCB list statistics + * + * @param pdcb DCB to print results to + */ +void +dprintDCBList(DCB *pdcb) +{ + dprintListStats(pdcb, &DCBlist, "All DCBs"); +} + /** * Diagnostic to print all DCB allocated in the system * @@ -2092,22 +2029,20 @@ dprintOneDCB(DCB *pdcb, DCB *dcb) void dprintAllDCBs(DCB *pdcb) { - DCB *dcb; + list_entry_t *current; - spinlock_acquire(&dcbspin); + current = list_start_iteration(&DCBlist); #if SPINLOCK_PROFILE dcb_printf(pdcb, "DCB List Spinlock Statistics:\n"); - spinlock_stats(&dcbspin, spin_reporter, pdcb); + spinlock_stats(&DCBlist->list_lock, spin_reporter, pdcb); dcb_printf(pdcb, "Zombie Queue Lock Statistics:\n"); spinlock_stats(&zombiespin, spin_reporter, pdcb); #endif - dcb = allDCBs; - while (dcb) + while (current) { - dprintOneDCB(pdcb, dcb); - dcb = dcb->next; + dprintOneDCB(pdcb, (DCB *)current); + current = list_iterate(&DCBlist, current); } - spinlock_release(&dcbspin); } /** @@ -2119,27 +2054,24 @@ void dListDCBs(DCB *pdcb) { DCB *dcb; + list_entry_t *current; - spinlock_acquire(&dcbspin); - dcb = allDCBs; + current = list_start_iteration(&DCBlist); dcb_printf(pdcb, "Descriptor Control Blocks\n"); dcb_printf(pdcb, "------------------+----------------------------+--------------------+----------\n"); dcb_printf(pdcb, " %-16s | %-26s | %-18s | %s\n", "DCB", "State", "Service", "Remote"); dcb_printf(pdcb, "------------------+----------------------------+--------------------+----------\n"); - while (dcb) + while (current) { - if (dcb->dcb_is_in_use) - { - dcb_printf(pdcb, " %-16p | %-26s | %-18s | %s\n", - dcb, gw_dcb_state2string(dcb->state), - ((dcb->session && dcb->session->service) ? dcb->session->service->name : ""), - (dcb->remote ? dcb->remote : "")); - } - dcb = dcb->next; + dcb = (DCB *)current; + dcb_printf(pdcb, " %-16p | %-26s | %-18s | %s\n", + dcb, gw_dcb_state2string(dcb->state), + ((dcb->session && dcb->session->service) ? dcb->session->service->name : ""), + (dcb->remote ? dcb->remote : "")); + current = list_iterate(&DCBlist, current); } dcb_printf(pdcb, "------------------+----------------------------+--------------------+----------\n\n"); - spinlock_release(&dcbspin); } /** @@ -2151,17 +2083,19 @@ void dListClients(DCB *pdcb) { DCB *dcb; + list_entry_t *current; + + current = list_start_iteration(&DCBlist); - spinlock_acquire(&dcbspin); - dcb = allDCBs; dcb_printf(pdcb, "Client Connections\n"); dcb_printf(pdcb, "-----------------+------------------+----------------------+------------\n"); dcb_printf(pdcb, " %-15s | %-16s | %-20s | %s\n", "Client", "DCB", "Service", "Session"); dcb_printf(pdcb, "-----------------+------------------+----------------------+------------\n"); - while (dcb) + while (current) { - if (dcb->dcb_is_in_use && dcb->dcb_role == DCB_ROLE_CLIENT_HANDLER) + dcb = (DCB *)current; + if (dcb->dcb_role == DCB_ROLE_CLIENT_HANDLER) { dcb_printf(pdcb, " %-15s | %16p | %-20s | %10p\n", (dcb->remote ? dcb->remote : ""), @@ -2169,10 +2103,9 @@ dListClients(DCB *pdcb) dcb->session->service->name : ""), dcb->session); } - dcb = dcb->next; + current = list_iterate(&DCBlist, current); } dcb_printf(pdcb, "-----------------+------------------+----------------------+------------\n\n"); - spinlock_release(&dcbspin); } @@ -2185,10 +2118,6 @@ dListClients(DCB *pdcb) void dprintDCB(DCB *pdcb, DCB *dcb) { - if (false == dcb->dcb_is_in_use) - { - return; - } dcb_printf(pdcb, "DCB: %p\n", (void *)dcb); dcb_printf(pdcb, "\tDCB state: %s\n", gw_dcb_state2string(dcb->state)); if (dcb->session && dcb->session->service) @@ -2223,13 +2152,13 @@ dprintDCB(DCB *pdcb, DCB *dcb) if (statusname) { dcb_printf(pdcb, "\tServer status: %s\n", statusname); - free(statusname); + MXS_FREE(statusname); } char *rolename = dcb_role_name(dcb); if (rolename) { dcb_printf(pdcb, "\tRole: %s\n", rolename); - free(rolename); + MXS_FREE(rolename); } dcb_printf(pdcb, "\tStatistics:\n"); dcb_printf(pdcb, "\t\tNo. of Reads: %d\n", @@ -2498,7 +2427,8 @@ gw_write(DCB *dcb, GWBUF *writeq, bool *stop_writing) if (strncmp(str, "set autocommit", 14) == 0 && nbytes > 17) { - char* s = (char *)calloc(1, nbytes + 1); + char* s = (char *)MXS_CALLOC(1, nbytes + 1); + MXS_ABORT_IF_NULL(s); if (nbytes - 5 > len) { @@ -2517,7 +2447,7 @@ gw_write(DCB *dcb, GWBUF *writeq, bool *stop_writing) pthread_self(), w, s); - free(s); + MXS_FREE(s); } } } @@ -2578,7 +2508,7 @@ dcb_add_callback(DCB *dcb, { DCB_CALLBACK *cb, *ptr, *lastcb = NULL; - if ((ptr = (DCB_CALLBACK *)malloc(sizeof(DCB_CALLBACK))) == NULL) + if ((ptr = (DCB_CALLBACK *)MXS_MALLOC(sizeof(DCB_CALLBACK))) == NULL) { return 0; } @@ -2594,7 +2524,7 @@ dcb_add_callback(DCB *dcb, cb->userdata == userdata) { /* Callback is a duplicate, abandon it */ - free(ptr); + MXS_FREE(ptr); spinlock_release(&dcb->cb_lock); return 0; } @@ -2657,7 +2587,7 @@ dcb_remove_callback(DCB *dcb, dcb->callbacks = cb->next; } spinlock_release(&dcb->cb_lock); - free(cb); + MXS_FREE(cb); rval = 1; break; } @@ -2709,7 +2639,7 @@ dcb_call_callback(DCB *dcb, DCB_REASON reason) } /** - * Check the passed DCB to ensure it is in the list of allDCBS + * Check the passed DCB to ensure it is in the list of all DCBS * * @param dcb The DCB to check * @return 1 if the DCB is in the list, otherwise 0 @@ -2717,50 +2647,7 @@ dcb_call_callback(DCB *dcb, DCB_REASON reason) int dcb_isvalid(DCB *dcb) { - int rval = 0; - - if (dcb) - { - spinlock_acquire(&dcbspin); - rval = dcb_isvalid_nolock(dcb); - spinlock_release(&dcbspin); - } - - return rval; -} - -/** - * Find a DCB in the list of all DCB's - * - * @param dcb The DCB to find - * @return A pointer to the DCB or NULL if not in the list - */ -static inline DCB * -dcb_find_in_list (DCB *dcb) -{ - DCB *ptr = NULL; - if (dcb) - { - ptr = allDCBs; - while (ptr && (false == ptr->dcb_is_in_use || ptr != dcb)) - { - ptr = ptr->next; - } - } - return ptr; -} - -/** - * Check the passed DCB to ensure it is in the list of allDCBS. - * Requires that the DCB list is already locked before call. - * - * @param dcb The DCB to check - * @return 1 if the DCB is in the list, otherwise 0 - */ -static inline int -dcb_isvalid_nolock(DCB *dcb) -{ - return (dcb == dcb_find_in_list(dcb)); + return (int)list_is_entry_in_use(&DCBlist, (list_entry_t *)dcb); } /** @@ -2782,16 +2669,13 @@ dcb_call_foreach(struct server* server, DCB_REASON reason) case DCB_REASON_NOT_RESPONDING: { DCB *dcb; - spinlock_acquire(&dcbspin); - dcb = allDCBs; + list_entry_t *current; - while (dcb != NULL) + current = list_start_iteration(&DCBlist); + + while (current) { - if (false == dcb->dcb_is_in_use) - { - dcb = dcb->next; - continue; - } + dcb = (DCB *)current; spinlock_acquire(&dcb->dcb_initlock); if (dcb->state == DCB_STATE_POLLING && dcb->server && strcmp(dcb->server->unique_name,server->unique_name) == 0) @@ -2799,9 +2683,8 @@ dcb_call_foreach(struct server* server, DCB_REASON reason) dcb_call_callback(dcb, DCB_REASON_NOT_RESPONDING); } spinlock_release(&dcb->dcb_initlock); - dcb = dcb->next; + current = list_iterate(&DCBlist, current); } - spinlock_release(&dcbspin); break; } @@ -2819,29 +2702,23 @@ dcb_call_foreach(struct server* server, DCB_REASON reason) void dcb_hangup_foreach(struct server* server) { - MXS_DEBUG("%lu [dcb_hangup_foreach]", pthread_self()); - DCB *dcb; - spinlock_acquire(&dcbspin); - dcb = allDCBs; + list_entry_t *current; - while (dcb != NULL) + current = list_start_iteration(&DCBlist); + + while (current) { - if (false == dcb->dcb_is_in_use) - { - dcb = dcb->next; - continue; - } + dcb = (DCB *)current; spinlock_acquire(&dcb->dcb_initlock); if (dcb->state == DCB_STATE_POLLING && dcb->server && - strcmp(dcb->server->unique_name, server->unique_name) == 0) + dcb->server == server) { poll_fake_hangup_event(dcb); } spinlock_release(&dcb->dcb_initlock); - dcb = dcb->next; + current = list_iterate(&DCBlist, current); } - spinlock_release(&dcbspin); } @@ -2963,13 +2840,13 @@ dcb_count_by_usage(DCB_USAGE usage) { int rval = 0; DCB *dcb; + list_entry_t *current; - spinlock_acquire(&dcbspin); - dcb = allDCBs; - while (dcb) + current = list_start_iteration(&DCBlist); + + while (current) { - if (dcb->dcb_is_in_use) - { + dcb = (DCB *)current; switch (usage) { case DCB_USAGE_CLIENT: @@ -3007,10 +2884,8 @@ dcb_count_by_usage(DCB_USAGE usage) rval++; break; } - dcb = dcb->next; + current = list_iterate(&DCBlist, current); } - } - spinlock_release(&dcbspin); return rval; } @@ -3272,7 +3147,7 @@ dcb_accept(DCB *listener, GWPROTOCOL *protocol_funcs) if (((struct sockaddr *)&client_conn)->sa_family == AF_UNIX) { // client address - client_dcb->remote = strdup("localhost_from_socket"); + client_dcb->remote = MXS_STRDUP_A("localhost_from_socket"); // set localhost IP for user authentication (client_dcb->ipv4).sin_addr.s_addr = 0x0100007F; } @@ -3283,7 +3158,7 @@ dcb_accept(DCB *listener, GWPROTOCOL *protocol_funcs) (struct sockaddr_in *)&client_conn, sizeof(struct sockaddr_in)); /* client IPv4 in string representation */ - client_dcb->remote = (char *)calloc(INET_ADDRSTRLEN + 1, sizeof(char)); + client_dcb->remote = (char *)MXS_CALLOC(INET_ADDRSTRLEN + 1, sizeof(char)); if (client_dcb->remote != NULL) { @@ -3582,6 +3457,13 @@ dcb_listen_create_socket_unix(const char *config_bind) *tmp = '\0'; } + if (strlen(config_bind) > sizeof(local_addr.sun_path) - 1) + { + MXS_ERROR("The path %s specified for the UNIX domain socket is too long. " + "The maximum length is %lu.", config_bind, sizeof(local_addr.sun_path) - 1); + return -1; + } + // UNIX socket create if ((listener_socket = socket(AF_UNIX, SOCK_STREAM, 0)) < 0) { @@ -3608,7 +3490,7 @@ dcb_listen_create_socket_unix(const char *config_bind) memset(&local_addr, 0, sizeof(local_addr)); local_addr.sun_family = AF_UNIX; - strncpy(local_addr.sun_path, config_bind, sizeof(local_addr.sun_path) - 1); + strcpy(local_addr.sun_path, config_bind); if ((-1 == unlink(config_bind)) && (errno != ENOENT)) { @@ -3678,9 +3560,9 @@ dcb_set_socket_option(int sockfd, int level, int optname, void *optval, socklen_ char * dcb_role_name(DCB *dcb) { - char *name = NULL; + char *name = (char *)MXS_MALLOC(64); - if (NULL != (name = (char *)malloc(64))) + if (name) { name[0] = 0; if (DCB_ROLE_SERVICE_LISTENER == dcb->dcb_role) diff --git a/server/core/doxygen.c b/server/core/doxygen.c index 15aba92bd..dcdd426cb 100644 --- a/server/core/doxygen.c +++ b/server/core/doxygen.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/externcmd.c b/server/core/externcmd.c index f066d76f2..c38041f21 100644 --- a/server/core/externcmd.c +++ b/server/core/externcmd.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -12,6 +12,7 @@ */ #include +#include /** * Tokenize a string into arguments suitable for a execvp call. @@ -48,7 +49,7 @@ int tokenize_arguments(char* argstr, char** argv) else if (quoted && !escaped && *ptr == qc) /** End of quoted string */ { *ptr = '\0'; - argv[i++] = strdup(start); + argv[i++] = MXS_STRDUP_A(start); read = false; quoted = false; } @@ -59,7 +60,7 @@ int tokenize_arguments(char* argstr, char** argv) *ptr = '\0'; if (read) /** New token */ { - argv[i++] = strdup(start); + argv[i++] = MXS_STRDUP_A(start); read = false; } } @@ -81,7 +82,7 @@ int tokenize_arguments(char* argstr, char** argv) } if (read) { - argv[i++] = strdup(start); + argv[i++] = MXS_STRDUP_A(start); } argv[i] = NULL; @@ -98,8 +99,8 @@ int tokenize_arguments(char* argstr, char** argv) */ EXTERNCMD* externcmd_allocate(char* argstr) { - EXTERNCMD* cmd = (EXTERNCMD*) malloc(sizeof(EXTERNCMD)); - char** argv = (char**) malloc(sizeof(char*) * MAXSCALE_EXTCMD_ARG_MAX); + EXTERNCMD* cmd = (EXTERNCMD*) MXS_MALLOC(sizeof(EXTERNCMD)); + char** argv = (char**) MXS_MALLOC(sizeof(char*) * MAXSCALE_EXTCMD_ARG_MAX); if (argstr && cmd && argv) { @@ -131,10 +132,8 @@ EXTERNCMD* externcmd_allocate(char* argstr) } else { - MXS_ERROR("Memory allocation for external command parameters failed when " - "processing '%s'.", argstr); - free(cmd); - free(argv); + MXS_FREE(cmd); + MXS_FREE(argv); cmd = NULL; } return cmd; @@ -150,10 +149,10 @@ void externcmd_free(EXTERNCMD* cmd) { for (int i = 0; cmd->argv[i]; i++) { - free(cmd->argv[i]); + MXS_FREE(cmd->argv[i]); } - free(cmd->argv); - free(cmd); + MXS_FREE(cmd->argv); + MXS_FREE(cmd); } } @@ -210,22 +209,22 @@ bool externcmd_substitute_arg(EXTERNCMD* cmd, const char* match, const char* rep for (int i = 0; cmd->argv[i] && rval; i++) { size_t size = strlen(cmd->argv[i]); - char* dest = malloc(size); + char* dest = MXS_MALLOC(size); if (dest) { mxs_pcre2_result_t rc = mxs_pcre2_substitute(re, cmd->argv[i], replace, &dest, &size); switch (rc) { case MXS_PCRE2_ERROR: - free(dest); + MXS_FREE(dest); rval = false; break; case MXS_PCRE2_MATCH: - free(cmd->argv[i]); + MXS_FREE(cmd->argv[i]); cmd->argv[i] = dest; break; case MXS_PCRE2_NOMATCH: - free(dest); + MXS_FREE(dest); break; } } @@ -264,10 +263,15 @@ char* get_command(const char* str) size_t len = end - start; - if (len > 0 && (rval = malloc(len + 1))) + if (len > 0) { - memcpy(rval, start, len); - rval[len] = '\0'; + rval = MXS_MALLOC(len + 1); + + if (rval) + { + memcpy(rval, start, len); + rval[len] = '\0'; + } } return rval; @@ -300,7 +304,7 @@ bool externcmd_can_execute(const char* argstr) { MXS_ERROR("The executable cannot be found: %s", command); } - free(command); + MXS_FREE(command); } return rval; } diff --git a/server/core/filter.c b/server/core/filter.c index b7c97f68d..904a4aaaa 100644 --- a/server/core/filter.c +++ b/server/core/filter.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -32,6 +32,7 @@ #include #include #include +#include static SPINLOCK filter_spin = SPINLOCK_INIT; /**< Protects the list of all filters */ static FILTER_DEF *allFilters = NULL; /**< The list of all filters */ @@ -50,14 +51,20 @@ static void filter_free_parameters(FILTER_DEF *filter); FILTER_DEF * filter_alloc(char *name, char *module) { - FILTER_DEF *filter; + name = MXS_STRDUP(name); + module = MXS_STRDUP(module); - if ((filter = (FILTER_DEF *)malloc(sizeof(FILTER_DEF))) == NULL) + FILTER_DEF *filter = (FILTER_DEF *)MXS_MALLOC(sizeof(FILTER_DEF)); + + if (!name || !module || !filter) { + MXS_FREE(name); + MXS_FREE(module); + MXS_FREE(filter); return NULL; } - filter->name = strdup(name); - filter->module = strdup(module); + filter->name = name; + filter->module = module; filter->filter = NULL; filter->options = NULL; filter->obj = NULL; @@ -108,21 +115,21 @@ filter_free(FILTER_DEF *filter) spinlock_release(&filter_spin); /* Clean up session and free the memory */ - free(filter->name); - free(filter->module); + MXS_FREE(filter->name); + MXS_FREE(filter->module); if (filter->options) { for (int i = 0; filter->options[i]; i++) { - free(filter->options[i]); + MXS_FREE(filter->options[i]); } - free(filter->options); + MXS_FREE(filter->options); } filter_free_parameters(filter); - free(filter); + MXS_FREE(filter); } } @@ -288,8 +295,9 @@ filterAddOption(FILTER_DEF *filter, char *option) spinlock_acquire(&filter->spin); if (filter->options == NULL) { - filter->options = (char **)calloc(2, sizeof(char *)); - filter->options[0] = strdup(option); + filter->options = (char **)MXS_CALLOC(2, sizeof(char *)); + MXS_ABORT_IF_NULL(filter->options); + filter->options[0] = MXS_STRDUP_A(option); filter->options[1] = NULL; } else @@ -298,8 +306,11 @@ filterAddOption(FILTER_DEF *filter, char *option) { ; } - filter->options = (char **)realloc(filter->options, (i + 2) * sizeof(char *)); - filter->options[i] = strdup(option); + + filter->options = (char **)MXS_REALLOC(filter->options, (i + 2) * sizeof(char *)); + MXS_ABORT_IF_NULL(filter->options); + filter->options[i] = MXS_STRDUP_A(option); + MXS_ABORT_IF_NULL(filter->options[i]); filter->options[i + 1] = NULL; } spinlock_release(&filter->spin); @@ -318,9 +329,10 @@ filterAddParameter(FILTER_DEF *filter, char *name, char *value) int i; spinlock_acquire(&filter->spin); + FILTER_PARAMETER **parameters; if (filter->parameters == NULL) { - filter->parameters = (FILTER_PARAMETER **)calloc(2, sizeof(FILTER_PARAMETER *)); + parameters = (FILTER_PARAMETER **)MXS_CALLOC(2, sizeof(FILTER_PARAMETER *)); i = 0; } else @@ -329,13 +341,20 @@ filterAddParameter(FILTER_DEF *filter, char *name, char *value) { ; } - filter->parameters = (FILTER_PARAMETER **)realloc(filter->parameters, - (i + 2) * sizeof(FILTER_PARAMETER *)); + parameters = (FILTER_PARAMETER **)MXS_REALLOC(filter->parameters, + (i + 2) * sizeof(FILTER_PARAMETER *)); } - filter->parameters[i] = (FILTER_PARAMETER *)calloc(1, sizeof(FILTER_PARAMETER)); - filter->parameters[i]->name = strdup(name); - filter->parameters[i]->value = strdup(value); - filter->parameters[i + 1] = NULL; + FILTER_PARAMETER *parameter = MXS_CALLOC(1, sizeof(FILTER_PARAMETER)); + name = MXS_STRDUP(name); + value = MXS_STRDUP(value); + + MXS_ABORT_IF_TRUE(!parameters || !parameter || !name || !value); + + parameter->name = name; + parameter->value = value; + parameters[i] = parameter; + parameters[i + 1] = NULL; + filter->parameters = parameters; spinlock_release(&filter->spin); } @@ -349,10 +368,10 @@ static void filter_free_parameters(FILTER_DEF *filter) { for (int i = 0; filter->parameters[i]; i++) { - free(filter->parameters[i]->name); - free(filter->parameters[i]->value); + MXS_FREE(filter->parameters[i]->name); + MXS_FREE(filter->parameters[i]->value); } - free(filter->parameters); + MXS_FREE(filter->parameters); } } @@ -407,14 +426,8 @@ filterApply(FILTER_DEF *filter, SESSION *session, DOWNSTREAM *downstream) { DOWNSTREAM *me; - if ((me = (DOWNSTREAM *)calloc(1, sizeof(DOWNSTREAM))) == NULL) + if ((me = (DOWNSTREAM *)MXS_CALLOC(1, sizeof(DOWNSTREAM))) == NULL) { - char errbuf[STRERROR_BUFLEN]; - MXS_ERROR("Memory allocation for filter session failed " - "due to %d,%s.", - errno, - strerror_r(errno, errbuf, sizeof(errbuf))); - return NULL; } me->instance = filter->filter; @@ -422,7 +435,7 @@ filterApply(FILTER_DEF *filter, SESSION *session, DOWNSTREAM *downstream) if ((me->session = filter->obj->newSession(me->instance, session)) == NULL) { - free(me); + MXS_FREE(me); return NULL; } filter->obj->setDownstream(me->instance, me->session, downstream); @@ -459,7 +472,7 @@ filterUpstream(FILTER_DEF *filter, void *fsession, UPSTREAM *upstream) if (filter->obj->clientReply != NULL) { - if ((me = (UPSTREAM *)calloc(1, sizeof(UPSTREAM))) == NULL) + if ((me = (UPSTREAM *)MXS_CALLOC(1, sizeof(UPSTREAM))) == NULL) { return NULL; } diff --git a/server/core/gateway.c b/server/core/gateway.c index 168251857..4f39f91f8 100644 --- a/server/core/gateway.c +++ b/server/core/gateway.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -61,6 +61,7 @@ #include #include #include +#include #include #include @@ -84,6 +85,7 @@ #include #include #include +#include #define STRING_BUFFER_SIZE 1024 #define PIDFD_CLOSED -1 @@ -124,23 +126,24 @@ const char *progname = NULL; static struct option long_options[] = { {"config-check", no_argument, 0, 'c'}, - {"config", required_argument, 0, 'f'}, {"nodaemon", no_argument, 0, 'd'}, + {"config", required_argument, 0, 'f'}, {"log", required_argument, 0, 'l'}, {"logdir", required_argument, 0, 'L'}, - {"datadir", required_argument, 0, 'D'}, - {"configdir", required_argument, 0, 'C'}, - {"piddir", required_argument, 0, 'P'}, - {"libdir", required_argument, 0, 'B'}, {"cachedir", required_argument, 0, 'A'}, + {"libdir", required_argument, 0, 'B'}, + {"configdir", required_argument, 0, 'C'}, + {"datadir", required_argument, 0, 'D'}, + {"execdir", required_argument, 0, 'E'}, {"language", required_argument, 0, 'N'}, - {"syslog", required_argument, 0, 's'}, - {"maxlog", required_argument, 0, 'S'}, + {"piddir", required_argument, 0, 'P'}, {"user", required_argument, 0, 'U'}, - {"version", no_argument, 0, 'v'}, - {"help", no_argument, 0, '?'}, - {"version-full", no_argument, 0, 'V'}, + {"syslog", required_argument, 0, 's'}, + {"maxlog", required_argument, 0, 'S'}, {"log_augmentation", required_argument, 0, 'G'}, + {"version", no_argument, 0, 'v'}, + {"version-full", no_argument, 0, 'V'}, + {"help", no_argument, 0, '?'}, {0, 0, 0, 0} }; static bool syslog_configured = false; @@ -219,7 +222,7 @@ struct CRYPTO_dynlock_value */ static struct CRYPTO_dynlock_value *ssl_create_dynlock(const char* file, int line) { - struct CRYPTO_dynlock_value* lock = malloc(sizeof(struct CRYPTO_dynlock_value)); + struct CRYPTO_dynlock_value* lock = MXS_MALLOC(sizeof(struct CRYPTO_dynlock_value)); if (lock) { spinlock_init(&lock->lock); @@ -254,7 +257,7 @@ static void ssl_lock_dynlock(int mode, struct CRYPTO_dynlock_value * n, const ch */ static void ssl_free_dynlock(struct CRYPTO_dynlock_value * n, const char* file, int line) { - free(n); + MXS_FREE(n); } #ifdef OPENSSL_1_0 @@ -378,7 +381,7 @@ sigfatal_handler(int i) { MXS_ERROR(" %s\n", symbols[n]); } - free(symbols); + MXS_FREE(symbols); } else { @@ -605,14 +608,15 @@ static bool resolve_maxscale_conf_fname(char** cnf_full_path, { if (cnf_file_arg) { - *cnf_full_path = (char *)malloc(PATH_MAX + 1); + *cnf_full_path = (char *)MXS_MALLOC(PATH_MAX + 1); + MXS_ABORT_IF_NULL(*cnf_full_path); if (!realpath(cnf_file_arg, *cnf_full_path)) { const char* logstr = "Failed to open read access to configuration file."; int eno = errno; print_log_n_stderr(true, true, logstr, logstr, eno); - free(*cnf_full_path); + MXS_FREE(*cnf_full_path); *cnf_full_path = NULL; } } @@ -638,7 +642,7 @@ static char* check_dir_access(char* dirname, bool rd, bool wr) if (dirname == NULL) { - errstr = strdup("Directory argument is NULL"); + errstr = MXS_STRDUP_A("Directory argument is NULL"); goto retblock; } @@ -646,7 +650,7 @@ static char* check_dir_access(char* dirname, bool rd, bool wr) { snprintf(errbuf, PATH_MAX * 2 - 1, "Can't access '%s'.", dirname); errbuf[PATH_MAX * 2 - 1] = '\0'; - errstr = strdup(errbuf); + errstr = MXS_STRDUP_A(errbuf); goto retblock; } @@ -655,7 +659,7 @@ static char* check_dir_access(char* dirname, bool rd, bool wr) snprintf(errbuf, PATH_MAX * 2 - 1, "MaxScale doesn't have read permission " "to '%s'.", dirname); errbuf[PATH_MAX * 2 - 1] = '\0'; - errstr = strdup(errbuf); + errstr = MXS_STRDUP_A(errbuf); goto retblock; } @@ -664,7 +668,7 @@ static char* check_dir_access(char* dirname, bool rd, bool wr) snprintf(errbuf, PATH_MAX * 2 - 1, "MaxScale doesn't have write permission " "to '%s'.", dirname); errbuf[PATH_MAX * 2 - 1] = '\0'; - errstr = strdup(errbuf); + errstr = MXS_STRDUP_A(errbuf); goto retblock; } @@ -793,7 +797,16 @@ static char* get_expanded_pathname(char** output_path, goto return_cnf_file_buf; } - expanded_path = (char*)malloc(PATH_MAX); + expanded_path = (char*)MXS_MALLOC(PATH_MAX); + + if (!expanded_path) + { + if (output_path) + { + *output_path = NULL; + } + goto return_cnf_file_buf; + } /*< * Expand possible relative pathname to absolute path @@ -807,8 +820,11 @@ static char* get_expanded_pathname(char** output_path, snprintf(buff, sizeof(buff), "Failed to read the directory '%s'.", relative_path); print_log_n_stderr(true, true, buff, buff, eno); - free(expanded_path); - *output_path = NULL; + MXS_FREE(expanded_path); + if (output_path) + { + *output_path = NULL; + } goto return_cnf_file_buf; } @@ -820,17 +836,11 @@ static char* get_expanded_pathname(char** output_path, */ size_t pathlen = strnlen(expanded_path, PATH_MAX) + 1 + strnlen(fname, PATH_MAX) + 1; - cnf_file_buf = (char*)malloc(pathlen); + cnf_file_buf = (char*)MXS_MALLOC(pathlen); if (cnf_file_buf == NULL) { - ss_dassert(cnf_file_buf != NULL); - char errbuf[STRERROR_BUFLEN]; - - MXS_ERROR("Memory allocation failed due to %s.", - strerror_r(errno, errbuf, sizeof(errbuf))); - - free(expanded_path); + MXS_FREE(expanded_path); expanded_path = NULL; goto return_cnf_file_buf; } @@ -838,8 +848,8 @@ static char* get_expanded_pathname(char** output_path, if (!file_is_readable(cnf_file_buf)) { - free(expanded_path); - free(cnf_file_buf); + MXS_FREE(expanded_path); + MXS_FREE(cnf_file_buf); expanded_path = NULL; cnf_file_buf = NULL; goto return_cnf_file_buf; @@ -853,7 +863,7 @@ static char* get_expanded_pathname(char** output_path, */ if (!file_is_readable(expanded_path)) { - free(expanded_path); + MXS_FREE(expanded_path); expanded_path = NULL; goto return_cnf_file_buf; } @@ -861,7 +871,7 @@ static char* get_expanded_pathname(char** output_path, if (output_path == NULL) { - free(expanded_path); + MXS_FREE(expanded_path); } else { @@ -877,31 +887,31 @@ static void usage(void) { fprintf(stderr, "\nUsage : %s [OPTION]...\n\n" - " -c, --config-check Validate configuration file and exit\n" - " -d, --nodaemon enable running in terminal process (default:disabled)\n" - " -f, --config=FILE relative or absolute pathname of MaxScale configuration file\n" - " (default:/etc/maxscale.cnf)\n" + " -c, --config-check Validate configuration file and exit\n" + " -d, --nodaemon enable running in terminal process (default:disabled)\n" + " -f, --config=FILE relative or absolute pathname of MaxScale configuration file\n" + " (default:/etc/maxscale.cnf)\n" " -l, --log=[file|shm|stdout] log to file, shared memory or stdout (default: file)\n" - " -L, --logdir=PATH path to log file directory (default: /var/log/maxscale)\n" - " -A, --cachedir=PATH path to cache directory (default: /var/cache/maxscale)\n" - " -B, --libdir=PATH path to module directory (default: /usr/lib64/maxscale)\n" - " -C, --configdir=PATH path to configuration file directory (default: /etc/)\n" - " -D, --datadir=PATH path to data directory, stored embedded mysql tables\n" - " (default: /var/cache/maxscale)\n" - " -E, --execdir=PATH path to the maxscale and other executable files\n" - " (default: /usr/bin)\n" - " -N, --language=PATH path to errmsg.sys file (default: /var/lib/maxscale)\n" - " -P, --piddir=PATH path to PID file directory (default: /var/run/maxscale)\n" - " -U, --user=USER run MaxScale as another user.\n" - " The user ID and group ID of this user are used to run MaxScale.\n" - " -s, --syslog=[yes|no] log messages to syslog (default:yes)\n" - " -S, --maxlog=[yes|no] log messages to MaxScale log (default: yes)\n" - " -G, --log_augmentation=0|1 augment messages with the name of the function where\n" - " the message was logged (default: 0). Primarily for \n" - " development purposes.\n" - " -v, --version print version info and exit\n" - " -V, --version-full print full version info and exit\n" - " -?, --help show this help\n" + " -L, --logdir=PATH path to log file directory (default: /var/log/maxscale)\n" + " -A, --cachedir=PATH path to cache directory (default: /var/cache/maxscale)\n" + " -B, --libdir=PATH path to module directory (default: /usr/lib64/maxscale)\n" + " -C, --configdir=PATH path to configuration file directory (default: /etc/)\n" + " -D, --datadir=PATH path to data directory, stored embedded mysql tables\n" + " (default: /var/cache/maxscale)\n" + " -E, --execdir=PATH path to the maxscale and other executable files\n" + " (default: /usr/bin)\n" + " -N, --language=PATH path to errmsg.sys file (default: /var/lib/maxscale)\n" + " -P, --piddir=PATH path to PID file directory (default: /var/run/maxscale)\n" + " -U, --user=USER run MaxScale as another user.\n" + " The user ID and group ID of this user are used to run MaxScale.\n" + " -s, --syslog=[yes|no] log messages to syslog (default:yes)\n" + " -S, --maxlog=[yes|no] log messages to MaxScale log (default: yes)\n" + " -G, --log_augmentation=0|1 augment messages with the name of the function where\n" + " the message was logged (default: 0). Primarily for \n" + " development purposes.\n" + " -v, --version print version info and exit\n" + " -V, --version-full print full version info and exit\n" + " -?, --help show this help\n" , progname); } @@ -1327,7 +1337,7 @@ int main(int argc, char **argv) case 'D': snprintf(datadir, PATH_MAX, "%s", optarg); datadir[PATH_MAX] = '\0'; - set_datadir(strdup(optarg)); + set_datadir(MXS_STRDUP_A(optarg)); datadir_defined = true; break; case 'C': @@ -1552,10 +1562,8 @@ int main(int argc, char **argv) OPENSSL_add_all_algorithms_noconf(); int numlocks = CRYPTO_num_locks(); - if ((ssl_locks = malloc(sizeof(SPINLOCK) * (numlocks + 1))) == NULL) + if ((ssl_locks = MXS_MALLOC(sizeof(SPINLOCK) * (numlocks + 1))) == NULL) { - char* logerr = "Memory allocation failed"; - print_log_n_stderr(true, true, logerr, logerr, eno); rc = MAXSCALE_INTERNALERROR; goto return_main; } @@ -1877,7 +1885,7 @@ int main(int argc, char **argv) * configured as the main thread will also poll. */ n_threads = config_threadcount(); - threads = calloc(n_threads, sizeof(THREAD)); + threads = (THREAD*)MXS_CALLOC(n_threads, sizeof(THREAD)); /*< * Start server threads. */ @@ -1951,11 +1959,11 @@ return_main: if (threads) { - free(threads); + MXS_FREE(threads); } if (cnf_file_path) { - free(cnf_file_path); + MXS_FREE(cnf_file_path); } return rc; @@ -2269,13 +2277,13 @@ bool handle_path_arg(char** dest, char* path, char* arg, bool rd, bool wr) if ((errstr = check_dir_access(pathbuffer, rd, wr)) == NULL) { - *dest = strdup(pathbuffer); + *dest = MXS_STRDUP_A(pathbuffer); rval = true; } else { print_log_n_stderr(true, true, errstr, errstr, 0); - free(errstr); + MXS_FREE(errstr); errstr = NULL; } } diff --git a/server/core/gw_ssl.c b/server/core/gw_ssl.c index 812e8466d..483b138dc 100644 --- a/server/core/gw_ssl.c +++ b/server/core/gw_ssl.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/gw_utils.c b/server/core/gw_utils.c index ccf2a12b2..29f8d96de 100644 --- a/server/core/gw_utils.c +++ b/server/core/gw_utils.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -177,12 +177,11 @@ bool gw_daemonize(void) int parse_bindconfig(const char *config, struct sockaddr_in *addr) { - char *port, buf[1024 + 1]; - short pnum; - struct hostent *hp; + char buf[strlen(config) + 1]; + strcpy(buf, config); - strncpy(buf, config, 1024); - port = strrchr(buf, ':'); + char *port = strrchr(buf, ':'); + short pnum; if (port) { *port = 0; @@ -202,7 +201,9 @@ parse_bindconfig(const char *config, struct sockaddr_in *addr) { if (!inet_aton(buf, &addr->sin_addr)) { - if ((hp = gethostbyname(buf)) != NULL) + struct hostent *hp = gethostbyname(buf); + + if (hp) { bcopy(hp->h_addr, &(addr->sin_addr.s_addr), hp->h_length); } diff --git a/server/core/gwbitmask.c b/server/core/gwbitmask.c index fd8f331aa..baab9ad97 100644 --- a/server/core/gwbitmask.c +++ b/server/core/gwbitmask.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -14,6 +14,7 @@ #include #include #include +#include /** * @file gwbitmask.c Implementation of bitmask operations for the gateway @@ -73,7 +74,7 @@ bitmask_init(GWBITMASK *bitmask) { bitmask->length = BIT_LENGTH_INITIAL; bitmask->size = bitmask->length / 8; - if ((bitmask->bits = calloc(bitmask->size, 1)) == NULL) + if ((bitmask->bits = MXS_CALLOC(bitmask->size, 1)) == NULL) { bitmask->length = bitmask->size = 0; } @@ -90,7 +91,7 @@ bitmask_free(GWBITMASK *bitmask) { if (bitmask->length) { - free(bitmask->bits); + MXS_FREE(bitmask->bits); bitmask->length = bitmask->size = 0; } } @@ -108,24 +109,30 @@ bitmask_free(GWBITMASK *bitmask) void bitmask_set(GWBITMASK *bitmask, int bit) { - unsigned char *ptr = bitmask->bits; + unsigned char *ptr; spinlock_acquire(&bitmask->lock); if (bit >= 8) { while (bit >= bitmask->length) { - bitmask->bits = realloc(bitmask->bits, - (bitmask->size + (BIT_LENGTH_INC / 8))); + bitmask->bits = MXS_REALLOC(bitmask->bits, (bitmask->size + (BIT_LENGTH_INC / 8))); + MXS_ABORT_IF_NULL(bitmask->bits); memset(bitmask->bits + (bitmask->size), 0, BIT_LENGTH_INC / 8); bitmask->length += BIT_LENGTH_INC; bitmask->size += (BIT_LENGTH_INC / 8); } + ptr = bitmask->bits; ptr += (bit / 8); bit = bit % 8; } + else + { + ptr = bitmask->bits; + } + *ptr |= bitmapset[bit]; spinlock_release(&bitmask->lock); } @@ -282,9 +289,9 @@ bitmask_copy(GWBITMASK *dest, GWBITMASK *src) spinlock_acquire(&dest->lock); if (dest->length) { - free(dest->bits); + MXS_FREE(dest->bits); } - if ((dest->bits = malloc(src->size)) == NULL) + if ((dest->bits = MXS_MALLOC(src->size)) == NULL) { dest->length = 0; } @@ -320,7 +327,7 @@ bitmask_render_readable(GWBITMASK *bitmask) spinlock_acquire(&bitmask->lock); if (999 < bitmask->length) { - result = malloc(sizeof(toobig)); + result = MXS_MALLOC(sizeof(toobig)); if (result) { strcpy(result, toobig); @@ -331,7 +338,7 @@ bitmask_render_readable(GWBITMASK *bitmask) count_set = bitmask_count_bits_set(bitmask); if (count_set) { - result = malloc(1 + (4 * count_set)); + result = MXS_MALLOC(1 + (4 * count_set)); if (result) { result[0] = 0; @@ -348,7 +355,7 @@ bitmask_render_readable(GWBITMASK *bitmask) } else { - result = malloc(sizeof(empty)); + result = MXS_MALLOC(sizeof(empty)); if (result) { strcpy(result, empty); diff --git a/server/core/gwdirs.c b/server/core/gwdirs.c index b4d6d833b..2a9aef8c8 100644 --- a/server/core/gwdirs.c +++ b/server/core/gwdirs.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -12,6 +12,7 @@ */ #include +#include #include /** @@ -20,7 +21,7 @@ */ void set_configdir(char* str) { - free(configdir); + MXS_FREE(configdir); clean_up_pathname(str); configdir = str; } @@ -31,7 +32,7 @@ void set_configdir(char* str) */ void set_logdir(char* str) { - free(logdir); + MXS_FREE(logdir); clean_up_pathname(str); logdir = str; } @@ -42,7 +43,7 @@ void set_logdir(char* str) */ void set_langdir(char* str) { - free(langdir); + MXS_FREE(langdir); clean_up_pathname(str); langdir = str; } @@ -53,7 +54,7 @@ void set_langdir(char* str) */ void set_piddir(char* str) { - free(piddir); + MXS_FREE(piddir); clean_up_pathname(str); piddir = str; } @@ -64,7 +65,7 @@ void set_piddir(char* str) */ void set_cachedir(char* param) { - free(cachedir); + MXS_FREE(cachedir); clean_up_pathname(param); cachedir = param; } @@ -75,7 +76,7 @@ void set_cachedir(char* param) */ void set_datadir(char* param) { - free(maxscaledatadir); + MXS_FREE(maxscaledatadir); clean_up_pathname(param); maxscaledatadir = param; } @@ -86,7 +87,7 @@ void set_datadir(char* param) */ void set_process_datadir(char* param) { - free(processdatadir); + MXS_FREE(processdatadir); clean_up_pathname(param); processdatadir = param; } @@ -97,7 +98,7 @@ void set_process_datadir(char* param) */ void set_libdir(char* param) { - free(libdir); + MXS_FREE(libdir); clean_up_pathname(param); libdir = param; } @@ -109,7 +110,7 @@ void set_libdir(char* param) */ void set_execdir(char* param) { - free(execdir); + MXS_FREE(execdir); clean_up_pathname(param); execdir = param; } diff --git a/server/core/hashtable.c b/server/core/hashtable.c index 06e38a29b..650b2ee06 100644 --- a/server/core/hashtable.c +++ b/server/core/hashtable.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -17,6 +17,8 @@ #include #include #include +#include +#include #include /** @@ -65,11 +67,11 @@ static void hashtable_write_lock(HASHTABLE *table); static void hashtable_write_unlock(HASHTABLE *table); static HASHTABLE *hashtable_alloc_real(HASHTABLE* target, int size, - int (*hashfn)(), - int (*cmpfn)()); + HASHHASHFN hashfn, + HASHCMPFN cmpfn); /** - * Special null function used as default memory allfunctions in the hashtable + * Special identity function used as default key/value copy function in the hashtable * implementation. This avoids having to special case the code that manipulates * the keys and values * @@ -77,9 +79,21 @@ static HASHTABLE *hashtable_alloc_real(HASHTABLE* target, * @return Return the value we were called with */ static void * +identityfn(const void *data) +{ + return (void*)data; +} + +/** + * Special null function used as default free function in the hashtable + * implementation. This avoids having to special case the code that manipulates + * the keys and values + * + * @param data The data pointer + */ +static void nullfn(void *data) { - return data; } /** @@ -95,15 +109,15 @@ nullfn(void *data) * @return The hashtable table */ HASHTABLE * -hashtable_alloc(int size, int (*hashfn)(), int (*cmpfn)()) +hashtable_alloc(int size, HASHHASHFN hashfn, HASHCMPFN cmpfn) { return hashtable_alloc_real(NULL, size, hashfn, cmpfn); } HASHTABLE* hashtable_alloc_flat(HASHTABLE* target, int size, - int (*hashfn)(), - int (*cmpfn)()) + HASHHASHFN hashfn, + HASHCMPFN cmpfn) { return hashtable_alloc_real(target, size, hashfn, cmpfn); } @@ -111,14 +125,14 @@ HASHTABLE* hashtable_alloc_flat(HASHTABLE* target, static HASHTABLE * hashtable_alloc_real(HASHTABLE* target, int size, - int (*hashfn)(), - int (*cmpfn)()) + HASHHASHFN hashfn, + HASHCMPFN cmpfn) { HASHTABLE *rval; if (target == NULL) { - if ((rval = malloc(sizeof(HASHTABLE))) == NULL) + if ((rval = MXS_MALLOC(sizeof(HASHTABLE))) == NULL) { return NULL; } @@ -137,17 +151,17 @@ hashtable_alloc_real(HASHTABLE* target, rval->hashsize = size > 0 ? size : 1; rval->hashfn = hashfn; rval->cmpfn = cmpfn; - rval->kcopyfn = nullfn; - rval->vcopyfn = nullfn; + rval->kcopyfn = identityfn; + rval->vcopyfn = identityfn; rval->kfreefn = nullfn; rval->vfreefn = nullfn; rval->n_readers = 0; rval->writelock = 0; rval->n_elements = 0; spinlock_init(&rval->spin); - if ((rval->entries = (HASHENTRIES **)calloc(rval->hashsize, sizeof(HASHENTRIES *))) == NULL) + if ((rval->entries = (HASHENTRIES **)MXS_CALLOC(rval->hashsize, sizeof(HASHENTRIES *))) == NULL) { - free(rval); + MXS_FREE(rval); return NULL; } memset(rval->entries, 0, rval->hashsize * sizeof(HASHENTRIES *)); @@ -180,16 +194,16 @@ hashtable_free(HASHTABLE *table) ptr = entry->next; table->kfreefn(entry->key); table->vfreefn(entry->value); - free(entry); + MXS_FREE(entry); entry = ptr; } } - free(table->entries); + MXS_FREE(table->entries); hashtable_write_unlock(table); if (!table->ht_isflat) { - free(table); + MXS_FREE(table); } } @@ -205,11 +219,11 @@ hashtable_free(HASHTABLE *table) * @param vfreefn The free function for the value */ void -hashtable_memory_fns(HASHTABLE *table, - HASHMEMORYFN kcopyfn, - HASHMEMORYFN vcopyfn, - HASHMEMORYFN kfreefn, - HASHMEMORYFN vfreefn) +hashtable_memory_fns(HASHTABLE *table, + HASHCOPYFN kcopyfn, + HASHCOPYFN vcopyfn, + HASHFREEFN kfreefn, + HASHFREEFN vfreefn) { if (kcopyfn != NULL) { @@ -270,7 +284,7 @@ hashtable_add(HASHTABLE *table, void *key, void *value) } else { - HASHENTRIES *ptr = (HASHENTRIES *)malloc(sizeof(HASHENTRIES)); + HASHENTRIES *ptr = (HASHENTRIES *)MXS_MALLOC(sizeof(HASHENTRIES)); if (ptr == NULL) { hashtable_write_unlock(table); @@ -283,7 +297,7 @@ hashtable_add(HASHTABLE *table, void *key, void *value) /* check succesfull key copy */ if (ptr->key == NULL) { - free(ptr); + MXS_FREE(ptr); hashtable_write_unlock(table); return 0; @@ -297,7 +311,7 @@ hashtable_add(HASHTABLE *table, void *key, void *value) { /* remove the key ! */ table->kfreefn(ptr->key); - free(ptr); + MXS_FREE(ptr); /* value not copied, return */ hashtable_write_unlock(table); @@ -363,7 +377,7 @@ hashtable_delete(HASHTABLE *table, void *key) entry->key = NULL; entry->value = NULL; } - free(entry); + MXS_FREE(entry); } else { @@ -380,7 +394,7 @@ hashtable_delete(HASHTABLE *table, void *key) ptr->next = entry->next; table->kfreefn(entry->key); table->vfreefn(entry->value); - free(entry); + MXS_FREE(entry); } table->n_elements--; assert(table->n_elements >= 0); @@ -633,9 +647,9 @@ hashtable_write_unlock(HASHTABLE *table) HASHITERATOR * hashtable_iterator(HASHTABLE *table) { - HASHITERATOR *rval; + HASHITERATOR *rval = (HASHITERATOR *)MXS_MALLOC(sizeof(HASHITERATOR)); - if ((rval = (HASHITERATOR *)malloc(sizeof(HASHITERATOR))) != NULL) + if (rval) { rval->table = table; rval->chain = 0; @@ -697,7 +711,7 @@ hashtable_next(HASHITERATOR *iter) void hashtable_iterator_free(HASHITERATOR *iter) { - free(iter); + MXS_FREE(iter); } /** @@ -826,3 +840,76 @@ int hashtable_size(HASHTABLE *table) spinlock_release(&table->spin); return rval; } + +/** + * Frees memory assumed to have been allocated using one of the MaxScale + * allocation functions. Intended to be used together with a hashtable + * copy function that merely makes a straight memory copy of the key/value, + * e.g. hashtable_item_strdup. + * @param data The memory to be freed. + */ +void hashtable_item_free(void *data) +{ + MXS_FREE(data); +} + +/** + * Convenience function intended for use as the comparison function of a hashtable, + * when the key is a NULL terminated string. Behaves as strcasecmp. + * @param str1 Pointer to string. + * @param str2 Pointer to string. + * @return Same as strcasecmp. + */ +int hashtable_item_strcasecmp(const void *str1, const void *str2) +{ + return strcasecmp((const char*)str1, (const char*)str2); +} + +/** + * Convenience function intended for use as the comparison function of a hashtable, + * when the key is a NULL terminated string. Behaves as strcmp. + * @param str1 Pointer to string. + * @param str2 Pointer to string. + * @return Same as strcmp. + */ +int hashtable_item_strcmp(const void *str1, const void *str2) +{ + return strcmp((const char*)str1, (const char*)str2); +} + +/** + * Convenience function intended for use as the copy function of a hashtable, + * when the key/value is a NULL terminated string. + * @param data A pointer to a NULL terminated string. + * @return A copy of the provided string or NULL if memory + * allocation fails. + */ +void* hashtable_item_strdup(const void* data) +{ + return MXS_STRDUP((const char*)data); +} + +/** + * Convenience function intended for use as the hash function of a hashtable, + * when the key is a NULL terminated string. + * @param data A pointer to a NULL terminated string. + * @return A hash of the string. + */ +int hashtable_item_strhash(const void* data) +{ + int hash = 0; + + if (data) + { + const char* key = (const char*)data; + + int c; + + while ((c = *key++)) + { + hash = c + (hash << 6) + (hash << 16) - hash; + } + } + + return hash; +} diff --git a/server/core/hint.c b/server/core/hint.c index 8c1a3d907..7cfb01f70 100644 --- a/server/core/hint.c +++ b/server/core/hint.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -14,6 +14,7 @@ #include #include #include +#include /** * @file hint.c generic support routines for hints. @@ -44,14 +45,14 @@ hint_dup(HINT *hint) ptr1 = hint; while (ptr1) { - if ((ptr2 = (HINT *)malloc(sizeof(HINT))) == NULL) + if ((ptr2 = (HINT *)MXS_MALLOC(sizeof(HINT))) == NULL) { return nlhead; } ptr2->type = ptr1->type; if (ptr1->data) { - ptr2->data = strdup(ptr1->data); + ptr2->data = MXS_STRDUP_A(ptr1->data); } else { @@ -59,7 +60,7 @@ hint_dup(HINT *hint) } if (ptr1->value) { - ptr2->value = strdup(ptr1->value); + ptr2->value = MXS_STRDUP_A(ptr1->value); } else { @@ -94,7 +95,7 @@ hint_create_route(HINT *head, HINT_TYPE type, char *data) { HINT *hint; - if ((hint = (HINT *)malloc(sizeof(HINT))) == NULL) + if ((hint = (HINT *)MXS_MALLOC(sizeof(HINT))) == NULL) { return head; } @@ -102,7 +103,7 @@ hint_create_route(HINT *head, HINT_TYPE type, char *data) hint->type = type; if (data) { - hint->data = strdup(data); + hint->data = MXS_STRDUP_A(data); } else { @@ -125,14 +126,14 @@ hint_create_parameter(HINT *head, char *pname, char *value) { HINT *hint; - if ((hint = (HINT *)malloc(sizeof(HINT))) == NULL) + if ((hint = (HINT *)MXS_MALLOC(sizeof(HINT))) == NULL) { return head; } hint->next = head; hint->type = HINT_PARAMETER; - hint->data = strdup(pname); - hint->value = strdup(value); + hint->data = MXS_STRDUP_A(pname); + hint->value = MXS_STRDUP_A(value); return hint; } @@ -146,13 +147,13 @@ hint_free(HINT *hint) { if (hint->data) { - free(hint->data); + MXS_FREE(hint->data); } if (hint->value) { - free(hint->value); + MXS_FREE(hint->value); } - free(hint); + MXS_FREE(hint); } bool hint_exists(HINT** p_hint, diff --git a/server/core/housekeeper.c b/server/core/housekeeper.c index 1dff3ad7b..4dae44983 100644 --- a/server/core/housekeeper.c +++ b/server/core/housekeeper.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -12,6 +12,7 @@ */ #include #include +#include #include #include #include @@ -88,13 +89,13 @@ hktask_add(const char *name, void (*taskfn)(void *), void *data, int frequency) { HKTASK *task, *ptr; - if ((task = (HKTASK *)malloc(sizeof(HKTASK))) == NULL) + if ((task = (HKTASK *)MXS_MALLOC(sizeof(HKTASK))) == NULL) { return 0; } - if ((task->name = strdup(name)) == NULL) + if ((task->name = MXS_STRDUP(name)) == NULL) { - free(task); + MXS_FREE(task); return 0; } task->task = taskfn; @@ -110,8 +111,8 @@ hktask_add(const char *name, void (*taskfn)(void *), void *data, int frequency) if (strcmp(ptr->name, name) == 0) { spinlock_release(&tasklock); - free(task->name); - free(task); + MXS_FREE(task->name); + MXS_FREE(task); return 0; } ptr = ptr->next; @@ -121,8 +122,8 @@ hktask_add(const char *name, void (*taskfn)(void *), void *data, int frequency) if (strcmp(ptr->name, name) == 0) { spinlock_release(&tasklock); - free(task->name); - free(task); + MXS_FREE(task->name); + MXS_FREE(task); return 0; } ptr->next = task; @@ -154,13 +155,13 @@ hktask_oneshot(const char *name, void (*taskfn)(void *), void *data, int when) { HKTASK *task, *ptr; - if ((task = (HKTASK *)malloc(sizeof(HKTASK))) == NULL) + if ((task = (HKTASK *)MXS_MALLOC(sizeof(HKTASK))) == NULL) { return 0; } - if ((task->name = strdup(name)) == NULL) + if ((task->name = MXS_STRDUP(name)) == NULL) { - free(task); + MXS_FREE(task); return 0; } task->task = taskfn; @@ -219,8 +220,8 @@ hktask_remove(const char *name) if (ptr) { - free(ptr->name); - free(ptr); + MXS_FREE(ptr->name); + MXS_FREE(ptr); return 1; } else diff --git a/server/core/listener.c b/server/core/listener.c index 04a05527e..71b7e35c7 100644 --- a/server/core/listener.c +++ b/server/core/listener.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -34,6 +34,8 @@ #include #include #include +#include +#include static RSA *rsa_512 = NULL; static RSA *rsa_1024 = NULL; @@ -51,21 +53,81 @@ static RSA *tmp_rsa_callback(SSL *s, int is_export, int keylength); * @return New listener object or NULL if unable to allocate */ SERV_LISTENER * -listener_alloc(char *protocol, char *address, unsigned short port, char *authenticator, SSL_LISTENER *ssl) +listener_alloc(struct service* service, char* name, char *protocol, char *address, + unsigned short port, char *authenticator, SSL_LISTENER *ssl) { - SERV_LISTENER *proto = NULL; - if ((proto = (SERV_LISTENER *)malloc(sizeof(SERV_LISTENER))) != NULL) + if (address) { - proto->listener = NULL; - proto->protocol = strdup(protocol); - proto->address = address ? strdup(address) : NULL; - proto->port = port; - proto->authenticator = authenticator ? strdup(authenticator) : NULL; - proto->ssl = ssl; + address = MXS_STRDUP(address); + if (!address) + { + return NULL; + } } + + if (authenticator) + { + authenticator = MXS_STRDUP(authenticator); + if (!authenticator) + { + MXS_FREE(address); + return NULL; + } + } + + protocol = MXS_STRDUP(protocol); + name = MXS_STRDUP(name); + SERV_LISTENER *proto = (SERV_LISTENER*)MXS_MALLOC(sizeof(SERV_LISTENER)); + + if (!protocol || !proto || !name) + { + MXS_FREE(protocol); + MXS_FREE(proto); + MXS_FREE(name); + return NULL; + } + + proto->name = name; + proto->listener = NULL; + proto->service = service; + proto->protocol = protocol; + proto->address = address; + proto->port = port; + proto->authenticator = authenticator; + proto->ssl = ssl; + proto->users = NULL; + proto->resources = NULL; + proto->next = NULL; + spinlock_init(&proto->lock); + return proto; } +/** + * @brief Free a listener + * + * @param listener Listener to free + */ +void listener_free(SERV_LISTENER* listener) +{ + if (listener) + { + if (listener->resources) + { + hashtable_free(listener->resources); + } + if (listener->users) + { + users_free(listener->users); + } + + MXS_FREE(listener->address); + MXS_FREE(listener->authenticator); + MXS_FREE(listener->protocol); + MXS_FREE(listener); + } +} + /** * Set the maximum SSL/TLS version the listener will support * @param ssl_listener Listener data to configure @@ -111,14 +173,14 @@ listener_set_ssl_version(SSL_LISTENER *ssl_listener, char* version) void listener_set_certificates(SSL_LISTENER *ssl_listener, char* cert, char* key, char* ca_cert) { - free(ssl_listener->ssl_cert); - ssl_listener->ssl_cert = cert ? strdup(cert) : NULL; + MXS_FREE(ssl_listener->ssl_cert); + ssl_listener->ssl_cert = cert ? MXS_STRDUP_A(cert) : NULL; - free(ssl_listener->ssl_key); - ssl_listener->ssl_key = key ? strdup(key) : NULL; + MXS_FREE(ssl_listener->ssl_key); + ssl_listener->ssl_key = key ? MXS_STRDUP_A(key) : NULL; - free(ssl_listener->ssl_ca_cert); - ssl_listener->ssl_ca_cert = ca_cert ? strdup(ca_cert) : NULL; + MXS_FREE(ssl_listener->ssl_ca_cert); + ssl_listener->ssl_ca_cert = ca_cert ? MXS_STRDUP_A(ca_cert) : NULL; } /** diff --git a/server/core/listmanager.c b/server/core/listmanager.c new file mode 100644 index 000000000..63c987300 --- /dev/null +++ b/server/core/listmanager.c @@ -0,0 +1,451 @@ +/* + * Copyright (c) 2016 MariaDB Corporation Ab + * + * Use of this software is governed by the Business Source License included + * in the LICENSE.TXT file and at www.mariadb.com/bsl. + * + * Change Date: 2019-07-01 + * + * On the date above, in accordance with the Business Source License, use + * of this software will be governed by version 2 or later of the General + * Public License. + */ + +/** + * @file listmanager.c - Logic for list handling + * + * MaxScale contains a number of linked lists. This code attempts to provide + * standard functions for handling them. Initially, the main work has been + * on recyclable lists - lists of entries that use dynamically allocated + * memory but are reused rather than freed. Some functions are not fully + * tested - see comments. + * + * @verbatim + * Revision History + * + * Date Who Description + * 20/04/16 Martin Brampton Initial implementation + * + * @endverbatim + */ +#include +#include +#include +#include +#include +#include +#include +#include + +/** + * Initialise a list configuration. + * + * @param list_config Pointer to the configuration of the list to be initialised + * @param type_of_list Type of list to be initialised. + * @param entry_size The size of each list entry (typically from sizeof). + * + * @note This is only required if a list is configured at execution time + * rather than being declared and statically initialized. + */ +void +list_initialise(LIST_CONFIG *list_config, list_type_t type_of_list, size_t entry_size) +{ + list_config->list_type = type_of_list; + list_config->all_entries = NULL; + list_config->last_entry = NULL; + list_config->last_free = NULL; + list_config->count = 0; + list_config->maximum = 0; + list_config->freecount = 0; + list_config->num_malloc = 0; + list_config->entry_size = entry_size; + spinlock_init(&list_config->list_lock); +} + +/** + * Allocate memory for some initial list entries + * + * The caller must check the return value. If it is false, this most likely + * indicates a memory allocation failure, and the caller should act + * accordingly. (It could indicate that the requested number of entries + * was less than 1, but this is fairly unlikely). + * + * @param list_config Pointer to the configuration of the list to be initialised + * @param num_entries Number of list entries to be allocated + * @return true if memory was allocated, false if not (or num_entries < 1) + * + */ +bool +list_pre_alloc(LIST_CONFIG *list_config, int num_entries, void (*init_struct)(void *)) +{ + uint8_t *entry_space; + bool result; + + spinlock_acquire(&list_config->list_lock); + if (num_entries <= 0) + { + MXS_ERROR("Attempt to preallocate space for recyclable list asked for no entries"); + result = false; + } + else if ((entry_space = (uint8_t *)MXS_CALLOC(num_entries, list_config->entry_size)) == NULL) + { + result = false; + } + else + { + list_entry_t *first_new_entry = (list_entry_t *)entry_space; + list_entry_t *previous = first_new_entry; + + for (int i = 1; i <= num_entries; i++) + { + if (init_struct) + { + init_struct((void *)previous); + } + previous->list_entry_chk_top = CHK_NUM_MANAGED_LIST; + previous->list_entry_chk_tail = CHK_NUM_MANAGED_LIST; + list_entry_t *next_entry = (list_entry_t *)((uint8_t *)previous + list_config->entry_size); + if (i < num_entries) + { + previous->next = next_entry; + previous = next_entry; + } + else + { + previous->next = NULL; + } + } + list_config->freecount += num_entries; + list_add_to_end(list_config, first_new_entry); + list_config->last_entry = previous; + list_config->last_free = first_new_entry; + result = true; + } + spinlock_release(&list_config->list_lock); + return result; +} + +/** + * @brief Find a free list entry or allocate memory for a new one. + * + * This routine looks to see whether there are free entries. + * If not, new memory is allocated, if possible, and the new entry is added to + * the list of all entries. + * + * @param Pointer to the list configuration structure + * @return An available entry or NULL if none could be found or created. + */ +list_entry_t * +list_find_free(LIST_CONFIG *list_config, void (*init_struct)(void *)) +{ + list_entry_t *available_entry; + + spinlock_acquire(&list_config->list_lock); + if (list_config->freecount <= 0) + { + /* No free entries, need to allocate a new one */ + if ((available_entry = MXS_CALLOC(1, list_config->entry_size)) == NULL) + { + spinlock_release(&list_config->list_lock); + return NULL; + } + list_config->num_malloc++; + + if (init_struct) + { + init_struct((void *)available_entry); + } + available_entry->list_entry_chk_top = CHK_NUM_MANAGED_LIST; + available_entry->list_entry_chk_tail = CHK_NUM_MANAGED_LIST; + available_entry->next = NULL; + list_add_to_end(list_config, available_entry); + } + /* Starting at the last place a free DCB was found, loop through the */ + /* list of DCBs searching for one that is not in use. */ + else + { + list_entry_t *next_in_list; + int loopcount = 0; + + while (list_config->last_free->entry_is_in_use) + { + list_config->last_free = list_config->last_free->next; + if (NULL == list_config->last_free) + { + loopcount++; + ss_dassert(loopcount == 1); + if (loopcount > 1) + { + /* Shouldn't need to loop round more than once */ + MXS_ERROR("Find free list entry failed to find when count positive"); + spinlock_release(&list_config->list_lock); + return NULL; + } + list_config->last_free = list_config->all_entries; + } + } + list_config->freecount--; + available_entry = list_config->last_free; + /* Clear the old data, then reset the list forward link */ + next_in_list = available_entry->next; + if (init_struct) + { + init_struct((void *)available_entry); + } + else + { + memset(available_entry, 0, list_config->entry_size); + } + available_entry->list_entry_chk_top = CHK_NUM_MANAGED_LIST; + available_entry->list_entry_chk_tail = CHK_NUM_MANAGED_LIST; + available_entry->next = next_in_list; + } + list_config->count++; + if (list_config->count > list_config->maximum) + { + list_config->maximum = list_config->count; + } + available_entry->entry_is_in_use = true; + spinlock_release(&list_config->list_lock); + return available_entry; +} + +/** + * @brief Display information about a recyclable list + * + * Should be called from a module that handles one specific list, passing + * the name for the list as well as the print DCB and the list config. + * + * @param Pointer to the print DCB + * @param List configuration pointer, the list to be displayed + * @param Name for the list + */ +void +dprintListStats(DCB *pdcb, LIST_CONFIG *list_config, const char *listname) +{ + dcb_printf(pdcb, "Recyclable list statistics\n"); + dcb_printf(pdcb, "--------------------------\n"); + dcb_printf(pdcb, "Name of list: %s\n", listname); + dcb_printf(pdcb, "Size of entries: %zu\n", list_config->entry_size); + dcb_printf(pdcb, "Currently in use: %d\n", list_config->count); + dcb_printf(pdcb, "Maximum ever used at once: %d\n", list_config->maximum); + dcb_printf(pdcb, "Currently free for reuse: %d\n", list_config->freecount); + dcb_printf(pdcb, "Total in use + free: %d\n", + list_config->freecount + list_config->count); + dcb_printf(pdcb, "Number of memory allocations: %d\n", list_config->num_malloc); +} + +/** + * @brief Dispose of a list entry by making it available for reuse. + * + * Within spinlock control, the entry is marked not in use and the + * counts are adjusted. + * + * @param Pointer to the list configuration structure + * @param List entry pointer, the item to be "freed" + */ +void +list_free_entry(LIST_CONFIG *list_config, list_entry_t *to_be_freed) +{ + spinlock_acquire(&list_config->list_lock); + to_be_freed->entry_is_in_use = false; + list_config->freecount++; + list_config->count--; + spinlock_release(&list_config->list_lock); +} + +/** + * @brief Find out whether a pointer points to a valid list entry + * + * Search the list for the given entry, under spinlock control. + * + * @param Pointer to the list configuration structure + * @param Pointer to be searched for + * @return True if the pointer is in the list and it is in use + */ +bool +list_is_entry_in_use(LIST_CONFIG *list_config, list_entry_t *to_be_found) +{ + list_entry_t *entry; + + spinlock_acquire(&list_config->list_lock); + entry = list_config->all_entries; + while (entry && to_be_found != entry) + { + entry = entry->next; + } + spinlock_release(&list_config->list_lock); + + return (entry && entry->entry_is_in_use); +} + +/** + * @brief Invoke a callback for every active member of list + * + * The list is locked, list entries that are in use are successively + * submitted to the callback function. The list entry is supplied to the + * callback function as the first parameters, followed by whatever other + * parameters have been passed to list_map. The process will continue so + * long as the callback function returns true, and will terminate either + * at the end of the list or when the callback function returns false. + * + * Code to be developed. + * + * @param Pointer to the list configuration structure + * @param Pointer to the callback function + */ +void +list_map(LIST_CONFIG *list_config, bool (*callback)(void *, ...)) +{ + +} + +/** + * @brief Start to iterate over a list + * + * The list is locked, and the first entry returned to the caller + * + * @param Pointer to the list configuration structure + * @return Pointer to the first entry in the list + */ +list_entry_t * +list_start_iteration(LIST_CONFIG *list_config) +{ + spinlock_acquire(&list_config->list_lock); + return list_config->all_entries; +} + +/** + * @brief Iterate over a list from a given point + * + * The list is assumed locked through list_start_iteration having been + * called. The next entry that is currently in use is returned to the caller. + * If the end of the list is reached, the spinlock is freed. + * + * @param Pointer to the list configuration structure + * @param Pointer to the entry to move forward from + * @return The next item in the list, or NULL if reached the end + */ +list_entry_t * +list_iterate(LIST_CONFIG *list_config, list_entry_t *current_entry) +{ + list_entry_t *next_entry = current_entry->next; + while (next_entry && !(next_entry->entry_is_in_use && next_entry->entry_is_ready)) + { + next_entry = next_entry->next; + } + if (NULL == next_entry) + { + spinlock_release(&list_config->list_lock); + } + return next_entry; +} + +/** + * @brief Terminate list iteration before reaching the end + * + * The list is assumed locked through list_start_iteration having been + * called, unless the last item is NULL, in which case it is assumed that + * the iteration had already reached the end of the list. If this is not + * the case, then the spinlock is released. + * + * @param Pointer to the list configuration structure + * @param Pointer to the entry last reached in the iteration + */ +void +list_terminate_iteration_early(LIST_CONFIG *list_config, list_entry_t *current_entry) +{ + if (current_entry) + { + spinlock_release(&list_config->list_lock); + } +} + +/** + * Add a new item to the end of a list. + * + * Must be called with the list lock held. + * + * A pointer, last_entry, is held to find the end of the list, and the new entry + * is linked to the end of the list. The pointer, last_free, that is used to + * search for a free entry is initialised if not already set. There cannot be + * any free entries until this routine has been called at least once. + * + * @param list_config The configuration of the list. + * @param new_entry The new entry to be added to the end of the list + * + * @note UNTESTED for simple or doubly linked lists, currently used + * internally for recyclable lists. + */ +void +list_add_to_end(LIST_CONFIG *list_config, list_entry_t *new_entry) +{ + if (NULL == list_config->all_entries) + { + list_config->all_entries = new_entry; + if (LIST_TYPE_DOUBLE == list_config->list_type) + { + new_entry->previous = NULL; + } + } + else + { + list_config->last_entry->next = new_entry; + if (LIST_TYPE_DOUBLE == list_config->list_type) + { + new_entry->previous = list_config->last_entry; + } + } + list_config->last_entry = new_entry; + if (NULL == list_config->last_free) + { + list_config->last_free = new_entry; + } +} + +/** + * @brief the list entry removed from the start of the list + * + * Must be called with the list lock held. + * + * @return The first list entry or NULL if the list is empty. + * + * @note UNTESTED! Intended for use on simple or doubly linked lists. + */ +list_entry_t * +list_remove_first(LIST_CONFIG *list_config) +{ + list_entry_t *first_in_list = NULL; + if (list_config->all_entries) + { + first_in_list = list_config->all_entries; + list_config->all_entries = first_in_list->next; + } + return first_in_list; +} + +/** + * @brief Return the list entry removed from the end of the list + * + * Must be called with the list lock held. + * + * @return The last list entry or NULL if the list is empty. + * + * @note UNTESTED! Intended for use only with doubly linked lists. + */ +list_entry_t * +list_remove_last(LIST_CONFIG *list_config) +{ + list_entry_t *last_in_list = NULL; + if (list_config->list_type != LIST_TYPE_DOUBLE) + { + MXS_ERROR("Attempt to remove the last entry in a list that is not doubly linked"); + return NULL; + } + if (list_config->all_entries) + { + last_in_list = list_config->last_entry; + list_config->last_entry = last_in_list->previous; + } + return last_in_list; +} diff --git a/server/core/load_utils.c b/server/core/load_utils.c index 30fa2a698..36ce7200c 100644 --- a/server/core/load_utils.c +++ b/server/core/load_utils.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -45,6 +45,7 @@ #include #include #include +#include static MODULES *registered = NULL; @@ -81,14 +82,14 @@ WriteMemoryCallback(void *contents, size_t size, size_t nmemb, void *userp) size_t realsize = size * nmemb; struct MemoryStruct *mem = (struct MemoryStruct *)userp; - mem->data = realloc(mem->data, mem->size + realsize + 1); - if (mem->data == NULL) + void *data = MXS_REALLOC(mem->data, mem->size + realsize + 1); + + if (data == NULL) { - /* out of memory! */ - MXS_ERROR("Error in module_feedback_send(), not enough memory for realloc"); return 0; } + mem->data = data; memcpy(&(mem->data[mem->size]), contents, realsize); mem->size += realsize; mem->data[mem->size] = 0; @@ -325,16 +326,25 @@ register_module(const char *module, void *modobj, MODULE_INFO *mod_info) { - MODULES *mod; + module = MXS_STRDUP(module); + type = MXS_STRDUP(type); + version = MXS_STRDUP(version); - if ((mod = malloc(sizeof(MODULES))) == NULL) + MODULES *mod = (MODULES *)MXS_MALLOC(sizeof(MODULES)); + + if (!module || !type || !version || !mod) { + MXS_FREE((void*)module); + MXS_FREE((void*)type); + MXS_FREE(version); + MXS_FREE(mod); return; } - mod->module = strdup(module); - mod->type = strdup(type); + + mod->module = (char*)module; + mod->type = (char*)type; mod->handle = dlhandle; - mod->version = strdup(version); + mod->version = version; mod->modobj = modobj; mod->next = registered; mod->info = mod_info; @@ -382,10 +392,10 @@ unregister_module(const char *module) * memory related to it can be freed */ dlclose(mod->handle); - free(mod->module); - free(mod->type); - free(mod->version); - free(mod); + MXS_FREE(mod->module); + MXS_FREE(mod->type); + MXS_FREE(mod->version); + MXS_FREE(mod); } /** @@ -506,7 +516,7 @@ moduleRowCallback(RESULTSET *set, void *data) } if (ptr == NULL) { - free(data); + MXS_FREE(data); return NULL; } (*rowno)++; @@ -543,14 +553,14 @@ moduleGetList() RESULTSET *set; int *data; - if ((data = (int *)malloc(sizeof(int))) == NULL) + if ((data = (int *)MXS_MALLOC(sizeof(int))) == NULL) { return NULL; } *data = 0; if ((set = resultset_create(moduleRowCallback, data)) == NULL) { - free(data); + MXS_FREE(data); return NULL; } resultset_add_column(set, "Module Name", 18, COL_TYPE_VARCHAR); @@ -807,7 +817,8 @@ do_http_post(GWBUF *buffer, void *cfg) FEEDBACK_CONF *feedback_config = (FEEDBACK_CONF *) cfg; /* allocate first memory chunck for httpd servr reply */ - chunk.data = malloc(1); /* will be grown as needed by the realloc above */ + chunk.data = MXS_MALLOC(1); /* will be grown as needed by the realloc above */ + MXS_ABORT_IF_NULL(chunk.data); chunk.size = 0; /* no data at this point */ /* Initializing curl library for data send via HTTP */ @@ -901,7 +912,7 @@ cleanup: if (chunk.data) { - free(chunk.data); + MXS_FREE(chunk.data); } if (curl) diff --git a/server/core/log_manager.cc b/server/core/log_manager.cc index 097ba88c5..313d50eb4 100644 --- a/server/core/log_manager.cc +++ b/server/core/log_manager.cc @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -17,6 +17,7 @@ #include #include +#include #include #include #include @@ -24,10 +25,13 @@ #include #include +#include #include +#include #include #include #include +#include #define MAX_PREFIXLEN 250 #define MAX_SUFFIXLEN 250 @@ -95,20 +99,24 @@ static simple_mutex_t msg_mutex; * Default augmentation. */ static int DEFAULT_LOG_AUGMENTATION = 0; +// A message that is logged 10 times in 1 second will be suppressed for 10 seconds. +static MXS_LOG_THROTTLING DEFAULT_LOG_THROTTLING = { 10, 1000, 10000 }; static struct { - int augmentation; // Can change during the lifetime of log_manager. - bool do_highprecision; // Can change during the lifetime of log_manager. - bool do_syslog; // Can change during the lifetime of log_manager. - bool do_maxlog; // Can change during the lifetime of log_manager. - bool use_stdout; // Can NOT changed during the lifetime of log_manager. + int augmentation; // Can change during the lifetime of log_manager. + bool do_highprecision; // Can change during the lifetime of log_manager. + bool do_syslog; // Can change during the lifetime of log_manager. + bool do_maxlog; // Can change during the lifetime of log_manager. + MXS_LOG_THROTTLING throttling; // Can change during the lifetime of log_manager. + bool use_stdout; // Can NOT change during the lifetime of log_manager. } log_config = { DEFAULT_LOG_AUGMENTATION, // augmentation false, // do_highprecision true, // do_syslog true, // do_maxlog + DEFAULT_LOG_THROTTLING, // throttling false // use_stdout }; @@ -140,7 +148,7 @@ ssize_t mxs_log_session_count[LOG_DEBUG + 1] = {0}; * Path to directory in which all files are stored to shared memory * by the OS. */ -const char* shm_pathname_prefix = "/dev/shm/"; +static const char SHM_PATHNAME_PREFIX[] = "/dev/shm/"; /** Forward declarations */ @@ -158,6 +166,7 @@ static logmanager_t* lm; static bool flushall_flag; static bool flushall_started_flag; static bool flushall_done_flag; +static HASHTABLE* message_stats; /** This is used to detect if the initialization of the log manager has failed * and that it isn't initialized again after a failure has occurred. */ @@ -183,6 +192,144 @@ struct filewriter #endif }; +typedef struct lm_message_key +{ + const char* filename; /** The filename where the error was reported. Must be a + statically allocated buffer, e.g. __FILE__ */ + int linenumber; /** The linenumber where the error was reported. */ +} LM_MESSAGE_KEY; + +typedef struct lm_message_stats +{ + SPINLOCK lock; + uint64_t first_ms; /** The time when the error was logged the first time in this window. */ + uint64_t last_ms; /** The time when the error was logged the last time. */ + size_t count; /** How many times the error has been reported within this window. */ +} LM_MESSAGE_STATS; + +static const int LM_MESSAGE_HASH_SIZE = 293; /** A prime, and roughly a quarter of current + number of MXS_{ERROR|WARNING|NOTICE} calls. */ + +/** + * Returns the current time. + * + * @return Current monotonic raw time in milliseconds. + */ +static uint64_t time_monotonic_raw_ms() +{ + struct timespec now; + clock_gettime(CLOCK_MONOTONIC_RAW, &now); + + return now.tv_sec * 1000 + now.tv_nsec / 1000000; +} + +/** + * Hash-function for lm_message_key. + * + * This is an implementation of the Jenkin's one-at-a-time hash function. + * https://en.wikipedia.org/wiki/Jenkins_hash_function + * + * @param v A pointer to a LM_MESSAGE_KEY + * @return Hash for the contents of the LM_MESSAGE_KEY structure. + */ +int lm_message_key_hash(const void *v) +{ + const LM_MESSAGE_KEY* key = (const LM_MESSAGE_KEY*)v; + + uint64_t key1 = (uint64_t)key->filename; + uint16_t key2 = (uint16_t)key->linenumber; // The first 48 bits are likely to be 0. + + uint32_t hash = 0; + size_t i; + + for (i = 0; i < sizeof(key1); ++i) + { + hash += (key1 >> i * 8) & 0xff; + hash += (hash << 10); + hash ^= (hash >> 6); + } + + for (i = 0; i < sizeof(key2); ++i) + { + hash += (key1 >> i * 8) & 0xff; + hash += (hash << 10); + hash ^= (hash >> 6); + } + + hash += (hash << 3); + hash ^= (hash >> 11); + hash += (hash << 15); + return hash; +} + +/** + * Compares two LM_MESSAGE_KEY structures + * + * @param v1 Pointer to a LM_MESSAGE_KEY struct. + * @param v2 Pointer to a LM_MESSAGE_KEY struct. + * @return 0 if the structures are equal. + * -1 if v1 is less than v2 + * 1 if v1 is greater than v2 + */ +static int lm_message_key_cmp(const void* v1, const void* v2) +{ + const LM_MESSAGE_KEY* key1 = (const LM_MESSAGE_KEY*)v1; + const LM_MESSAGE_KEY* key2 = (const LM_MESSAGE_KEY*)v2; + + int cmp = (int64_t)key1->filename - (int64_t)key2->filename; + + if (cmp == 0) + { + cmp = key1->linenumber - key2->linenumber; + } + + return cmp == 0 ? 0 : (cmp < 0 ? -1 : 1); +} + +/** + * Clone a LM_MESSAGE_KEY + * + * @param v Pointer to a LM_MESSAGE_KEY structure + * @return A copy of v or NULL if memory allocation fails. + */ +static void* lm_message_key_clone(const void* v) +{ + const LM_MESSAGE_KEY* src = (const LM_MESSAGE_KEY*)v; + LM_MESSAGE_KEY* dst = (LM_MESSAGE_KEY*)MXS_MALLOC(sizeof(LM_MESSAGE_KEY)); + + if (dst) + { + dst->filename = src->filename; + dst->linenumber = src->linenumber; + } + + return dst; +} + +/** + * Clone a LM_MESSAGE_STATS + * + * @param v Pointer to a LM_MESSAGE_STATS structure + * @return A copy of v or NULL if memory allocation fails. + */ +static void* lm_message_stats_clone(const void* v) +{ + const LM_MESSAGE_STATS* src = (const LM_MESSAGE_STATS*)v; + LM_MESSAGE_STATS* dst = (LM_MESSAGE_STATS*)MXS_MALLOC(sizeof(LM_MESSAGE_STATS)); + + if (dst) + { + // Somewhat questionable to copy a lock, but in this context we + // know that src is stack allocated and will not be used after this. + dst->lock = src->lock; + dst->first_ms = src->first_ms; + dst->last_ms = src->last_ms; + dst->count = src->count; + } + + return dst; +} + /** * Log client's string is copied to block-sized log buffer, which is passed * to file writer thread. @@ -215,7 +362,6 @@ struct logfile skygw_chk_t lf_chk_top; #endif flat_obj_state_t lf_state; - bool lf_init_started; bool lf_store_shmem; logmanager_t* lf_lmgr; /** fwr_logmes is for messages from log clients */ @@ -224,18 +370,14 @@ struct logfile char* lf_linkpath; /**< path to symlink file. */ const char* lf_name_prefix; const char* lf_name_suffix; - int lf_name_seqno; char* lf_full_file_name; /**< complete log file name */ char* lf_full_link_name; /**< complete symlink name */ - int lf_nfiles_max; - size_t lf_file_size; /** list of block-sized log buffers */ mlist_t lf_blockbuf_list; size_t lf_buf_size; bool lf_flushflag; - bool lf_rotateflag; + bool lf_rotateflag; int lf_spinlock; /**< lf_flushflag & lf_rotateflag */ - int lf_npending_writes; #if defined(SS_DEBUG) skygw_chk_t lf_chk_tail; #endif @@ -276,19 +418,13 @@ struct logmanager }; /** - * Type definition for string part. It is used in forming the log file name - * from string parts provided by the client of log manager, as arguments. + * Error logging the the log-manager itself. + * + * For obvious reasons, the log cannot use its own functions for reporting errors. */ - -typedef struct strpart -{ - const char* sp_string; - struct strpart* sp_next; -} strpart_t; - +#define LOG_ERROR(format, ...) do { fprintf(stderr, format, ##__VA_ARGS__); } while (false) /** Static function declarations */ -static bool logfiles_init(logmanager_t* lmgr); static bool logfile_init(logfile_t* logfile, logmanager_t* logmanager, bool store_shmem); @@ -297,12 +433,18 @@ static void logfile_free_memory(logfile_t* lf); static void logfile_flush(logfile_t* lf); static void logfile_rotate(logfile_t* lf); static bool logfile_build_name(logfile_t* lf); -static bool logfile_open_file(filewriter_t* fw, logfile_t* lf); -static char* form_full_file_name(strpart_t* parts, logfile_t* lf, int seqnoidx); +static bool logfile_open_file(filewriter_t* fw, + logfile_t* lf, + skygw_open_mode_t mode, + bool write_header); +static char* form_full_file_name(const char* directory, + const char* prefix, + const char* suffix); static bool filewriter_init(logmanager_t* logmanager, - filewriter_t* fw); -static void filewriter_done(filewriter_t* filewriter); + filewriter_t* fw, + bool write_header); +static void filewriter_done(filewriter_t* filewriter, bool write_footer); static bool fnames_conf_init(fnames_conf_t* fn, const char* logdir); static void fnames_conf_done(fnames_conf_t* fn); static void fnames_conf_free_memory(fnames_conf_t* fn); @@ -312,7 +454,8 @@ static bool logmanager_register(bool writep); static void logmanager_unregister(void); static bool logmanager_init_nomutex(const char* ident, const char* logdir, - mxs_log_target_t target); + mxs_log_target_t target, + bool write_header); static void logmanager_done_nomutex(void); static int logmanager_write_log(int priority, @@ -331,25 +474,23 @@ static void blockbuf_register(blockbuf_t* bb); static void blockbuf_unregister(blockbuf_t* bb); static char* add_slash(char* str); -static bool check_file_and_path(char* filename, - bool* writable, - bool do_log); +static bool check_file_and_path(const char* filename, bool* writable); -static bool file_is_symlink(char* filename); -static int find_last_seqno(strpart_t* parts, int seqno, int seqnoidx); +static bool file_is_symlink(const char* filename); void flushall_logfiles(bool flush); bool thr_flushall_check(); static bool logmanager_init_nomutex(const char* ident, const char* logdir, - mxs_log_target_t target) + mxs_log_target_t target, + bool write_header) { fnames_conf_t* fn; filewriter_t* fw; int err; bool succ = false; - lm = (logmanager_t *)calloc(1, sizeof(logmanager_t)); + lm = (logmanager_t *)MXS_CALLOC(1, sizeof(logmanager_t)); if (lm == NULL) { @@ -397,8 +538,8 @@ static bool logmanager_init_nomutex(const char* ident, goto return_succ; } - /** Initialize logfiles */ - if (!logfiles_init(lm)) + /** Initialize logfile */ + if (!logfile_init(&lm->lm_logfile, lm, (lm->lm_target == MXS_LOG_TARGET_SHMEM))) { err = 1; goto return_succ; @@ -413,7 +554,7 @@ static bool logmanager_init_nomutex(const char* ident, * Initialize filewriter data and open the log file * for each log file type. */ - if (!filewriter_init(lm, fw)) + if (!filewriter_init(lm, fw, write_header)) { err = 1; goto return_succ; @@ -443,7 +584,7 @@ return_succ: { /** This releases memory of all created objects */ logmanager_done_nomutex(); - fprintf(stderr, "*\n* Error : Initializing log manager failed.\n*\n"); + LOG_ERROR("MaxScale Log: Error, initialization failed.\n"); } return succ; } @@ -469,7 +610,29 @@ bool mxs_log_init(const char* ident, const char* logdir, mxs_log_target_t target if (!lm) { - succ = logmanager_init_nomutex(ident, logdir, target); + ss_dassert(!message_stats); + + message_stats = hashtable_alloc(LM_MESSAGE_HASH_SIZE, + lm_message_key_hash, + lm_message_key_cmp); + if (message_stats) + { + // As entries are added to the hashtable they will be cloned, + // so stack allocated keys and values are ok. + hashtable_memory_fns(message_stats, + lm_message_key_clone, + lm_message_stats_clone, + hashtable_item_free, + hashtable_item_free); + + succ = logmanager_init_nomutex(ident, logdir, target, log_config.do_maxlog); + + if (!succ) + { + hashtable_free(message_stats); + message_stats = NULL; + } + } } else { @@ -508,7 +671,7 @@ static void logmanager_done_nomutex(void) } /** Free filewriter memory. */ - filewriter_done(fwr); + filewriter_done(fwr, log_config.do_maxlog); lf = logmanager_get_logfile(lm); /** Release logfile memory */ @@ -522,8 +685,11 @@ static void logmanager_done_nomutex(void) skygw_message_done(lm->lm_logmes); /** Set global pointer NULL to prevent access to freed data. */ - free(lm); + MXS_FREE(lm); lm = NULL; + + hashtable_free(message_stats); + message_stats = NULL; } /** @@ -584,8 +750,6 @@ static logfile_t* logmanager_get_logfile(logmanager_t* lmgr) * @param priority Syslog priority * @param flush indicates whether log string must be written to disk * immediately - * @param rotate if set, closes currently open log file and opens a - * new one * @param prefix_len length of prefix to be stripped away when syslogging * @param str_len length of formatted string (including terminating NULL). * @param str string to be written to log @@ -672,7 +836,7 @@ static int logmanager_write_log(int priority, int tokval; simple_mutex_lock(&msg_mutex, true); - copy = strdup(str); + copy = MXS_STRDUP_A(str); tok = strtok(copy, "|"); tok = strtok(NULL, "|"); @@ -686,7 +850,7 @@ static int logmanager_write_log(int priority, } prevval = tokval; } - free(copy); + MXS_FREE(copy); simple_mutex_unlock(&msg_mutex); } #endif @@ -698,7 +862,7 @@ static int logmanager_write_log(int priority, } else { - wp = (char*)malloc(sizeof(char) * (timestamp_len - sizeof(char) + cmplen + str_len + 1)); + wp = (char*)MXS_MALLOC(sizeof(char) * (timestamp_len - sizeof(char) + cmplen + str_len + 1)); } if (wp == NULL) @@ -783,7 +947,7 @@ static int logmanager_write_log(int priority, } else { - free(wp); + MXS_FREE(wp); } return err; @@ -1112,7 +1276,7 @@ static blockbuf_t* blockbuf_init() { blockbuf_t* bb; - if ((bb = (blockbuf_t *) calloc(1, sizeof (blockbuf_t)))) + if ((bb = (blockbuf_t *) MXS_CALLOC(1, sizeof (blockbuf_t)))) { #if defined(SS_DEBUG) bb->bb_chk_top = CHK_NUM_BLOCKBUF; @@ -1128,10 +1292,6 @@ static blockbuf_t* blockbuf_init() #endif CHK_BLOCKBUF(bb); } - else - { - fprintf(stderr, "Error: Memory allocation failed when initializing log manager block buffers."); - } return bb; } @@ -1235,7 +1395,7 @@ static bool logmanager_register(bool writep) // If someone is logging before the log manager has been inited, // or after the log manager has been finished, the messages are // written to stdout. - succ = logmanager_init_nomutex(NULL, NULL, MXS_LOG_TARGET_DEFAULT); + succ = logmanager_init_nomutex(NULL, NULL, MXS_LOG_TARGET_DEFAULT, true); } } /** if logmanager existed or was succesfully restarted, increase link */ @@ -1315,7 +1475,7 @@ static bool fnames_conf_init(fnames_conf_t* fn, const char* logdir) dir = "/tmp"; } - fn->fn_logpath = strdup(dir); + fn->fn_logpath = MXS_STRDUP_A(dir); if (fn->fn_logpath) { @@ -1328,38 +1488,6 @@ static bool fnames_conf_init(fnames_conf_t* fn, const char* logdir) } -/** - * @node Calls logfile initializer for each logfile. - * - * - * Parameters: - * @param lm Log manager pointer - * - * @return true if succeed, otherwise false. - * - * - * @details If logfile is supposed to be located to shared memory - * it is specified here. In the case of shared memory file, a soft - * link is created to log directory. - * - * Motivation is performance. LOGFILE_DEBUG, for example, has so much data - * that writing it to disk slows execution down remarkably. - * - */ -static bool logfiles_init(logmanager_t* lm) -{ - bool store_shmem = (lm->lm_target == MXS_LOG_TARGET_SHMEM); - - bool succ = logfile_init(&lm->lm_logfile, lm, store_shmem); - - if (!succ) - { - fprintf(stderr, "*\n* Error : Initializing log files failed.\n"); - } - - return succ; -} - static void logfile_flush(logfile_t* lf) { CHK_LOGFILE(lf); @@ -1399,76 +1527,74 @@ static void logfile_rotate(logfile_t* lf) */ static bool logfile_build_name(logfile_t* lf) { - bool namecreatefail; - bool nameconflicts; - bool store_shmem; - bool writable; - bool succ; - strpart_t spart[3]; /*< string parts of which the file is composed of */ + bool succ = false; if (log_config.use_stdout) { // TODO: Refactor so that lf_full_file_name can be NULL in this case. - lf->lf_full_file_name = strdup("stdout"); + lf->lf_full_file_name = MXS_STRDUP_A("stdout"); succ = true; // TODO: Refactor to get rid of the gotos. goto return_succ; } + /** - * sparts is an array but next pointers are used to walk through - * the list of string parts. + * Create name for log file. */ - spart[0].sp_next = &spart[1]; - spart[1].sp_next = &spart[2]; - spart[2].sp_next = NULL; + lf->lf_full_file_name = form_full_file_name(lf->lf_filepath, + lf->lf_name_prefix, + lf->lf_name_suffix); - spart[1].sp_string = lf->lf_name_prefix; - spart[2].sp_string = lf->lf_name_suffix; - - store_shmem = lf->lf_store_shmem; - - do + if (lf->lf_store_shmem) { - namecreatefail = false; - nameconflicts = false; - - spart[0].sp_string = lf->lf_filepath; /** - * Create name for log file. Seqno is added between prefix & - * suffix (index == 2) + * Create name for link file */ - lf->lf_full_file_name = form_full_file_name(spart, lf, 2); + lf->lf_full_link_name = form_full_file_name(lf->lf_linkpath, + lf->lf_name_prefix, + lf->lf_name_suffix); + } + /** + * At least one of the files couldn't be created. Either + * memory allocation failed or the filename is too long. + */ + if (!lf->lf_full_file_name || (lf->lf_store_shmem && !lf->lf_full_link_name)) + { + goto return_succ; + } - if (store_shmem) + /** + * If file exists but is different type, create fails. + */ + bool writable; + if (check_file_and_path(lf->lf_full_file_name, &writable)) + { + /** Found similarly named file which isn't writable */ + if (!writable || file_is_symlink(lf->lf_full_file_name)) { - spart[0].sp_string = lf->lf_linkpath; - /** - * Create name for link file - */ - lf->lf_full_link_name = form_full_file_name(spart, lf, 2); + goto return_succ; } + } + else + { /** - * At least one of the files couldn't be created. Increase - * sequence number and retry until succeeds. + * Opening the file failed for some other reason than + * existing non-writable file. Shut down. */ - if (lf->lf_full_file_name == NULL || - (store_shmem && lf->lf_full_link_name == NULL)) + if (!writable) { - namecreatefail = true; - goto file_create_fail; + goto return_succ; } + } - /** - * If file exists but is different type, create fails and - * new, increased sequence number is added to file name. - */ - if (check_file_and_path(lf->lf_full_file_name, &writable, true)) + if (lf->lf_store_shmem) + { + if (check_file_and_path(lf->lf_full_link_name, &writable)) { - /** Found similarly named file which isn't writable */ - if (!writable || file_is_symlink(lf->lf_full_file_name)) + /** Found similarly named link which isn't writable */ + if (!writable) { - nameconflicts = true; - goto file_create_fail; + goto return_succ; } } else @@ -1479,52 +1605,10 @@ static bool logfile_build_name(logfile_t* lf) */ if (!writable) { - succ = false; goto return_succ; } } - - if (store_shmem) - { - if (check_file_and_path(lf->lf_full_link_name, &writable, true)) - { - /** Found similarly named link which isn't writable */ - if (!writable) - { - nameconflicts = true; - } - } - else - { - /** - * Opening the file failed for some other reason than - * existing non-writable file. Shut down. - */ - if (!writable) - { - succ = false; - goto return_succ; - } - } - } - file_create_fail: - if (namecreatefail || nameconflicts) - { - lf->lf_name_seqno += 1; - - if (lf->lf_full_file_name != NULL) - { - free(lf->lf_full_file_name); - lf->lf_full_file_name = NULL; - } - if (lf->lf_full_link_name != NULL) - { - free(lf->lf_full_link_name); - lf->lf_full_link_name = NULL; - } - } } - while (namecreatefail || nameconflicts); succ = true; @@ -1532,19 +1616,60 @@ return_succ: return succ; } +static bool logfile_write_header(skygw_file_t* file) +{ + CHK_FILE(file); + + bool written = true; + time_t t = time(NULL); + struct tm tm; + localtime_r(&t, &tm); + + const char PREFIX[] = "MariaDB MaxScale "; // sizeof(PREFIX) includes the NULL. + char time_string[32]; // 26 would be enough, according to "man asctime". + asctime_r(&tm, time_string); + + size_t size = sizeof(PREFIX) + strlen(file->sf_fname) + 2 * sizeof(' ') + strlen(time_string); + + char header[size + 2]; // For the 2 newlines. + sprintf(header, "\n\n%s%s %s", PREFIX, file->sf_fname, time_string); + + char line[sizeof(header) - 1]; + memset(line, '-', sizeof(line) - 1); + line[sizeof(line) - 1] = '\n'; + + size_t header_items = fwrite(header, sizeof(header) - 1, 1, file->sf_file); + size_t line_items = fwrite(line, sizeof(line), 1, file->sf_file); + + if ((header_items != 1) || (line_items != 1)) + { + char errbuf[STRERROR_BUFLEN]; + LOG_ERROR("MaxScale Log: Writing header failed due to %d, %s\n", + errno, strerror_r(errno, errbuf, sizeof(errbuf))); + written = false; + } + + return written; +} + /** * Opens a log file and writes header to the beginning of it. File name, FILE*, * and file descriptor are stored to skygw_file_t struct which is stored in * filewriter strcuture passed as parameter. * - * @param fw filewriter pointer - * @param lf logfile pointer + * @param fw filewriter pointer + * @param lf logfile pointer + * @param mode Wheather the file should be truncated or appended to. + * @param write_header Wheather a file header should be written. * * @return true if succeed; the resulting skygw_file_t is written in filewriter, * false if failed. * */ -static bool logfile_open_file(filewriter_t* fw, logfile_t* lf) +static bool logfile_open_file(filewriter_t* fw, + logfile_t* lf, + skygw_open_mode_t mode, + bool write_header) { bool rv = true; @@ -1553,20 +1678,22 @@ static bool logfile_open_file(filewriter_t* fw, logfile_t* lf) fw->fwr_file = skygw_file_alloc(lf->lf_full_file_name); fw->fwr_file->sf_file = stdout; } - else if (lf->lf_store_shmem) - { - /** Create symlink pointing to log file */ - fw->fwr_file = skygw_file_init(lf->lf_full_file_name, lf->lf_full_link_name); - } else { - /** Create normal disk-resident log file */ - fw->fwr_file = skygw_file_init(lf->lf_full_file_name, NULL); + const char* full_link_name = lf->lf_store_shmem ? lf->lf_full_link_name : NULL; + + /** Create symlink pointing to log file */ + fw->fwr_file = skygw_file_init(lf->lf_full_file_name, full_link_name, mode); + + if (fw->fwr_file && write_header) + { + logfile_write_header(fw->fwr_file); + } } if (fw->fwr_file == NULL) { - fprintf(stderr, "Error : opening logfile %s failed.\n", lf->lf_full_file_name); + LOG_ERROR("MaxScale Log: Error, opening log file %s failed.\n", lf->lf_full_file_name); rv = false; } @@ -1575,114 +1702,38 @@ static bool logfile_open_file(filewriter_t* fw, logfile_t* lf) /** - * @node Combine all name parts from left to right. + * @brief Form the complete file path. * - * Parameters: - * @param parts - * - * @param seqno specifies the the sequence number which will be added as a part - * of full file name. seqno == -1 indicates that sequence number won't be used. - * - * @param seqnoidx Specifies the seqno position in the 'array' of name parts. - * If seqno == -1 seqnoidx will be set -1 as well. + * @param directory The directory component containing a trailing '/'. + * @param prefix The prefix part of the filename. + * @param suffix The suffix part of the filename. * * @return Pointer to filename, of NULL if failed. * */ -static char* form_full_file_name(strpart_t* parts, logfile_t* lf, int seqnoidx) +static char* form_full_file_name(const char* directory, + const char* prefix, + const char* suffix) { - int i; - int seqno; - size_t s; - size_t fnlen; - char* filename = NULL; - char* seqnostr = NULL; - strpart_t* p; + char* filename = NULL; - if (lf->lf_name_seqno != -1) + size_t len = strlen(directory) + strlen(prefix) + strlen(suffix) + 1; + + if (len <= NAME_MAX) { - int file_sn; - int link_sn = 0; - const char* tmp = parts[0].sp_string; + filename = (char*)MXS_CALLOC(1, len); - file_sn = find_last_seqno(parts, lf->lf_name_seqno, seqnoidx); - - if (lf->lf_linkpath != NULL) + if (filename) { - tmp = parts[0].sp_string; - parts[0].sp_string = lf->lf_linkpath; - link_sn = find_last_seqno(parts, lf->lf_name_seqno, seqnoidx); - parts[0].sp_string = tmp; + strcat(filename, directory); + strcat(filename, prefix); + strcat(filename, suffix); } - lf->lf_name_seqno = MAX(file_sn, link_sn); - - seqno = lf->lf_name_seqno; - s = UINTLEN(seqno); - seqnostr = (char *)malloc((int)s + 1); } else { - /** - * These magic numbers are needed to indicate this and - * in subroutines that sequence number is not used. - */ - s = 0; - seqnoidx = -1; - seqno = lf->lf_name_seqno; - } - - if (parts == NULL || parts->sp_string == NULL) - { - goto return_filename; - } - /** - * Length of path, file name, separating slash, sequence number and - * terminating character. - */ - fnlen = sizeof('/') + s + sizeof('\0'); - p = parts; - /** Calculate the combined length of all parts put together */ - while (p->sp_string != NULL) - { - fnlen += strnlen(p->sp_string, NAME_MAX); - - if (p->sp_next == NULL) - { - break; - } - p = p->sp_next; - } - - if (fnlen > NAME_MAX) - { - fprintf(stderr, "Error : Too long file name= %d.\n", (int)fnlen); - goto return_filename; - } - filename = (char*)calloc(1, fnlen); - - if (seqnostr != NULL) - { - snprintf(seqnostr, s + 1, "%d", seqno); - } - - for (i = 0, p = parts; p->sp_string != NULL; i++, p = p->sp_next) - { - if (seqnostr != NULL && i == seqnoidx) - { - strcat(filename, seqnostr); /*< add sequence number */ - } - strcat(filename, p->sp_string); - - if (p->sp_next == NULL) - { - break; - } - } - -return_filename: - if (seqnostr != NULL) - { - free(seqnostr); + LOG_ERROR("MaxScale Log: Error, filename too long %s%s%s.\n", + directory, prefix, suffix); } return filename; @@ -1711,9 +1762,10 @@ static char* add_slash(char* str) /** Add slash if missing */ if (p[plen - 1] != '/') { - str = (char *)malloc(plen + 2); + str = (char *)MXS_MALLOC(plen + 2); + MXS_ABORT_IF_NULL(str); snprintf(str, plen + 2, "%s/", p); - free(p); + MXS_FREE(p); } return str; } @@ -1724,10 +1776,9 @@ static char* add_slash(char* str) * check if they are accessible and writable. * * Parameters: - * @param filename file to be checked - * + * @param filename File to be checked * @param writable flag indicating whether file was found writable or not - * if writable is NULL, check is skipped. + * if writable is NULL, check is skipped. * * @return true & writable if file exists and it is writable, * true & not writable if file exists but it can't be written, @@ -1738,7 +1789,7 @@ static char* add_slash(char* str) * TODO: recall what was the reason for not succeeding with simply * calling access, and fstat. vraa 26.11.13 */ -static bool check_file_and_path(char* filename, bool* writable, bool do_log) +static bool check_file_and_path(const char* filename, bool* writable) { bool exists; @@ -1767,25 +1818,17 @@ static bool check_file_and_path(char* filename, bool* writable, bool do_log) } else { - - if (do_log && file_is_symlink(filename)) + if (file_is_symlink(filename)) { char errbuf[STRERROR_BUFLEN]; - fprintf(stderr, - "*\n* Error : Can't access " - "file pointed to by %s due " - "to %s.\n", - filename, - strerror_r(errno, errbuf, sizeof(errbuf))); + LOG_ERROR("MaxScale Log: Error, Can't access file pointed to by %s due to %d, %s.\n", + filename, errno, strerror_r(errno, errbuf, sizeof(errbuf))); } - else if (do_log) + else { char errbuf[STRERROR_BUFLEN]; - fprintf(stderr, - "*\n* Error : Can't access %s due " - "to %s.\n", - filename, - strerror_r(errno, errbuf, sizeof(errbuf))); + LOG_ERROR("MaxScale Log: Error, Can't access %s due to %d, %s.\n", + filename, errno, strerror_r(errno, errbuf, sizeof(errbuf))); } if (writable) @@ -1809,7 +1852,7 @@ static bool check_file_and_path(char* filename, bool* writable, bool do_log) -static bool file_is_symlink(char* filename) +static bool file_is_symlink(const char* filename) { int rc; bool succ = false; @@ -1854,8 +1897,6 @@ static bool logfile_init(logfile_t* logfile, logfile->lf_logmes = logmanager->lm_logmes; logfile->lf_name_prefix = LOGFILE_NAME_PREFIX; logfile->lf_name_suffix = LOGFILE_NAME_SUFFIX; - logfile->lf_npending_writes = 0; - logfile->lf_name_seqno = 1; logfile->lf_lmgr = logmanager; logfile->lf_flushflag = false; logfile->lf_rotateflag = false; @@ -1869,34 +1910,34 @@ static bool logfile_init(logfile_t* logfile, */ if (store_shmem) { - char* c; - pid_t pid = getpid(); - int len = strlen(shm_pathname_prefix) - + strlen("maxscale.") + - get_decimal_len((size_t)pid) + 1; + char* dir; + int len = sizeof(SHM_PATHNAME_PREFIX) + sizeof(LOGFILE_NAME_PREFIX); - c = (char *)calloc(len, sizeof(char)); + dir = (char *)MXS_CALLOC(len, sizeof(char)); - if (c == NULL) + if (dir == NULL) { succ = false; goto return_with_succ; } - sprintf(c, "%smaxscale.%d", shm_pathname_prefix, pid); - logfile->lf_filepath = c; + sprintf(dir, "%s%s", SHM_PATHNAME_PREFIX, LOGFILE_NAME_PREFIX); + logfile->lf_filepath = dir; - if (mkdir(c, S_IRWXU | S_IRWXG) != 0 && - errno != EEXIST) + if (mkdir(dir, S_IRWXU | S_IRWXG) != 0 && (errno != EEXIST)) { + char errbuf[STRERROR_BUFLEN]; + LOG_ERROR("MaxScale Log: Error, creating directory %s failed due to %d, %s.\n", + dir, errno, strerror_r(errno, errbuf, sizeof(errbuf))); + succ = false; goto return_with_succ; } - logfile->lf_linkpath = strdup(fn->fn_logpath); + logfile->lf_linkpath = MXS_STRDUP_A(fn->fn_logpath); logfile->lf_linkpath = add_slash(logfile->lf_linkpath); } else { - logfile->lf_filepath = strdup(fn->fn_logpath); + logfile->lf_filepath = MXS_STRDUP_A(fn->fn_logpath); } logfile->lf_filepath = add_slash(logfile->lf_filepath); @@ -1910,13 +1951,11 @@ static bool logfile_init(logfile_t* logfile, */ if (mlist_init(&logfile->lf_blockbuf_list, NULL, - strdup("logfile block buffer list"), + MXS_STRDUP_A("logfile block buffer list"), blockbuf_node_done, MAXNBLOCKBUFS) == NULL) { - ss_dfprintf(stderr, - "*\n* Error : Initializing buffers for log files " - "failed."); + LOG_ERROR("MaxScale Log: Error, Initializing buffers for log files failed.\n"); logfile_free_memory(logfile); goto return_with_succ; } @@ -1959,7 +1998,6 @@ static void logfile_done(logfile_t* lf) { case RUN: CHK_LOGFILE(lf); - ss_dassert(lf->lf_npending_writes == 0); /** fallthrough */ case INIT: /** Test if list is initialized before freeing it */ @@ -1981,32 +2019,33 @@ static void logfile_free_memory(logfile_t* lf) { if (lf->lf_filepath != NULL) { - free(lf->lf_filepath); + MXS_FREE(lf->lf_filepath); } if (lf->lf_linkpath != NULL) { - free(lf->lf_linkpath); + MXS_FREE(lf->lf_linkpath); } if (lf->lf_full_link_name != NULL) { - free(lf->lf_full_link_name); + MXS_FREE(lf->lf_full_link_name); } if (lf->lf_full_file_name != NULL) { - free(lf->lf_full_file_name); + MXS_FREE(lf->lf_full_file_name); } } /** * @node Initialize filewriter data and open the log file for each log file type. * - * @param logmanager Log manager struct - * @param fw File writer struct + * @param logmanager Log manager struct + * @param fw File writer struct + * @param write_header Wheather a file header should be written. * * @return true if succeed, false if failed * */ -static bool filewriter_init(logmanager_t* logmanager, filewriter_t* fw) +static bool filewriter_init(logmanager_t* logmanager, filewriter_t* fw, bool write_header) { bool succ = false; @@ -2027,7 +2066,7 @@ static bool filewriter_init(logmanager_t* logmanager, filewriter_t* fw) logfile_t* lf = logmanager_get_logfile(logmanager); - if (logfile_open_file(fw, lf)) + if (logfile_open_file(fw, lf, SKYGW_OPEN_APPEND, write_header)) { fw->fwr_state = RUN; CHK_FILEWRITER(fw); @@ -2035,18 +2074,52 @@ static bool filewriter_init(logmanager_t* logmanager, filewriter_t* fw) } else { - fprintf(stderr, - "Error : opening log file %s failed. Exiting " - "MaxScale\n", - lf->lf_full_file_name); - filewriter_done(fw); + filewriter_done(fw, write_header); } ss_dassert(fw->fwr_state == RUN || fw->fwr_state == DONE); return succ; } -static void filewriter_done(filewriter_t* fw) +static bool logfile_write_footer(skygw_file_t* file, const char* suffix) +{ + CHK_FILE(file); + + bool written = true; + time_t t = time(NULL); + struct tm tm; + localtime_r(&t, &tm); + + const char FORMAT[] = "%04d-%02d-%02d %02d:%02d:%02d"; + char time_string[20]; // 19 chars + NULL. + + sprintf(time_string, FORMAT, + tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday, tm.tm_hour, tm.tm_min, tm.tm_sec); + + size_t size = sizeof(time_string) + 3 * sizeof(' ') + strlen(suffix) + sizeof('\n'); + + char header[size]; + sprintf(header, "%s %s\n", time_string, suffix); + + char line[sizeof(header) - 1]; + memset(line, '-', sizeof(line) - 1); + line[sizeof(line) - 1] = '\n'; + + size_t header_items = fwrite(header, sizeof(header) - 1, 1, file->sf_file); + size_t line_items = fwrite(line, sizeof(line), 1, file->sf_file); + + if ((header_items != 1) || (line_items != 1)) + { + char errbuf[STRERROR_BUFLEN]; + LOG_ERROR("MaxScale Log: Writing footer failed due to %d, %s\n", + errno, strerror_r(errno, errbuf, sizeof(errbuf))); + written = false; + } + + return written; +} + +static void filewriter_done(filewriter_t* fw, bool write_footer) { switch (fw->fwr_state) { @@ -2061,7 +2134,12 @@ static void filewriter_done(filewriter_t* fw) } else { - skygw_file_close(fw->fwr_file, true); + if (write_footer) + { + logfile_write_footer(fw->fwr_file, "MariaDB MaxScale is shut down."); + } + + skygw_file_close(fw->fwr_file); } fw->fwr_state = DONE; case DONE: @@ -2077,7 +2155,6 @@ static bool thr_flush_file(logmanager_t *lm, filewriter_t *fwr) * Get file pointer of current logfile. */ bool do_flushall = thr_flushall_check(); - skygw_file_t *file = fwr->fwr_file; logfile_t *lf = &lm->lm_logfile; /** @@ -2089,42 +2166,34 @@ static bool thr_flush_file(logmanager_t *lm, filewriter_t *fwr) lf->lf_flushflag = false; lf->lf_rotateflag = false; release_lock(&lf->lf_spinlock); - /** - * Log rotation : - * Close current, and open a new file for the log. - */ - if (rotate_logfile) + + // fwr->fwr_file may be NULL if an earlier log-rotation failed. + if (rotate_logfile || !fwr->fwr_file) { - bool succ; - - lf->lf_name_seqno += 1; /*< new sequence number */ - - if (!(succ = logfile_build_name(lf))) + /** + * Log rotation: Close file, and reopen in truncate mode. + */ + if (!log_config.use_stdout) { - lf->lf_name_seqno -= 1; /*< restore */ - } - else if ((succ = logfile_open_file(fwr, lf))) - { - if (log_config.use_stdout) + if (log_config.do_maxlog) { - skygw_file_free(file); + logfile_write_footer(fwr->fwr_file, "File closed due to log rotation."); } - else + + skygw_file_close(fwr->fwr_file); + fwr->fwr_file = NULL; + + if (!logfile_open_file(fwr, lf, SKYGW_OPEN_TRUNCATE, log_config.do_maxlog)) { - skygw_file_close(file, false); /*< close old file */ + LOG_ERROR("MaxScale Log: Error, could not re-open log file %s.\n", + lf->lf_full_file_name); } } - if (!succ) - { - MXS_ERROR("Log rotation failed. " - "Creating replacement file %s " - "failed. Continuing " - "logging to existing file.", - lf->lf_full_file_name); - } return true; } + + skygw_file_t *file = fwr->fwr_file; /** * get logfile's block buffer list */ @@ -2169,12 +2238,9 @@ static bool thr_flush_file(logmanager_t *lm, filewriter_t *fwr) { // TODO: Log this to syslog. char errbuf[STRERROR_BUFLEN]; - fprintf(stderr, - "Error : Writing to the log-file %s failed due to (%d, %s). " - "Disabling writing to the log.", - lf->lf_full_file_name, - err, - strerror_r(err, errbuf, sizeof(errbuf))); + LOG_ERROR("MaxScale Log: Error, writing to the log-file %s failed due to %d, %s. " + "Disabling writing to the log.\n", + lf->lf_full_file_name, err, strerror_r(err, errbuf, sizeof(errbuf))); mxs_log_set_maxlog_enabled(false); } @@ -2346,67 +2412,7 @@ static void fnames_conf_done(fnames_conf_t* fn) static void fnames_conf_free_memory(fnames_conf_t* fn) { - free(fn->fn_logpath); -} - -/** - * Find the file with biggest sequence number from given directory and return - * the sequence number. - * - * @param parts string parts of which the file name is composed of - * @param seqno the sequence number to start with, if seqno is -1 just return - * - * @return the biggest sequence number used - */ -static int find_last_seqno(strpart_t* parts, - int seqno, - int seqnoidx) -{ - strpart_t* p; - char* snstr; - int snstrlen; - - if (seqno == -1) - { - return seqno; - } - snstrlen = UINTLEN(INT_MAX); - snstr = (char *)calloc(1, snstrlen); - p = parts; - - while (true) - { - int i; - char filename[NAME_MAX] = {0}; - /** Form name with next seqno */ - snprintf(snstr, snstrlen, "%d", seqno + 1); - - for (i = 0, p = parts; p->sp_string != NULL; i++, p = p->sp_next) - { - if (snstr != NULL && i == seqnoidx) - { - strncat(filename, snstr, NAME_MAX - 1); /*< add sequence number */ - } - strncat(filename, p->sp_string, NAME_MAX - 1); - - if (p->sp_next == NULL) - { - break; - } - } - - if (check_file_and_path(filename, NULL, false)) - { - seqno++; - } - else - { - break; - } - } - free(snstr); - - return seqno; + MXS_FREE(fn->fn_logpath); } bool thr_flushall_check() @@ -2465,6 +2471,45 @@ void mxs_log_set_maxlog_enabled(bool enabled) MXS_NOTICE("maxlog logging is %s.", enabled ? "enabled" : "disabled"); } +/** + * Set the log throttling parameters. + * + * @param throttling The throttling parameters. + */ +void mxs_log_set_throttling(const MXS_LOG_THROTTLING* throttling) +{ + // No locking; it does not have any real impact, even if the struct + // is used right when its values are modified. + log_config.throttling = *throttling; + + if ((log_config.throttling.count == 0) || + (log_config.throttling.window_ms == 0) || + (log_config.throttling.suppress_ms == 0)) + { + MXS_NOTICE("Log throttling has been disabled."); + } + else + { + MXS_NOTICE("A message that is logged %lu times in %lu milliseconds, " + "will be suppressed for %lu milliseconds.", + log_config.throttling.count, + log_config.throttling.window_ms, + log_config.throttling.suppress_ms); + } +} + +/** + * Get the log throttling parameters. + * + * @param throttling The throttling parameters. + */ +void mxs_log_get_throttling(MXS_LOG_THROTTLING* throttling) +{ + // No locking; this is used only from maxadmin and an inconsistent set + // may be returned only if mxs_log_set_throttling() is called via an + // other instance of maxadmin at the very same moment. + *throttling = log_config.throttling; +} /** * Explicitly ensure that all pending log messages are flushed. @@ -2493,7 +2538,7 @@ int mxs_log_flush() } else { - ss_dfprintf(stderr, "Can't register to logmanager, flushing failed.\n"); + LOG_ERROR("MaxScale Log: Error, can't register to logmanager, flushing failed.\n"); } return err; @@ -2569,7 +2614,7 @@ int mxs_log_rotate() } else { - ss_dfprintf(stderr, "Can't register to logmanager, rotating failed.\n"); + LOG_ERROR("MaxScale Log: Error, Can't register to logmanager, rotating failed.\n"); } return err; @@ -2735,10 +2780,122 @@ static enum log_flush priority_to_flush(int priority) } } +typedef enum message_suppression +{ + MESSAGE_NOT_SUPPRESSED, // Message is not suppressed. + MESSAGE_SUPPRESSED, // Message is suppressed for the first time (for this round) + MESSAGE_STILL_SUPPRESSED // Message is still suppressed (for this round) +} message_suppression_t; + +static message_suppression_t message_status(const char* file, int line) +{ + message_suppression_t rv = MESSAGE_NOT_SUPPRESSED; + + // Copy the config to prevent the values from changing while we are using + // them. It does not matter if they are changed just when we are copying + // them, but we want to use one set of values throughout the function. + MXS_LOG_THROTTLING t = log_config.throttling; + + if ((t.count != 0) && (t.window_ms != 0) && (t.suppress_ms != 0)) + { + LM_MESSAGE_KEY key = { file, line }; + LM_MESSAGE_STATS *value; + + errno = 0; + + // Since the hashtable can not be externally locked, the lookup/creation + // must be done in a loop to ensure that everything works even if two + // threads logs the same message at the very same time. + do + { + value = (LM_MESSAGE_STATS*) hashtable_fetch(message_stats, &key); + + if (!value) + { + LM_MESSAGE_STATS stats; + spinlock_init(&stats.lock); + stats.first_ms = time_monotonic_raw_ms(); + stats.last_ms = 0; + stats.count = 0; + + // hashtable_add may fail for two reasons; the key exists + // already or it runs out of memory. In the latter case + // errno is set. + hashtable_add(message_stats, &key, &stats); + } + } + while (!value && (errno == 0)); + + if (value) + { + uint64_t now_ms = time_monotonic_raw_ms(); + + spinlock_acquire(&value->lock); + + ++value->count; + + // Less that t.window_ms milliseconds since the message was logged + // the last time. May have to be throttled. + if (value->count < t.count) + { + // t.count times has not been reached, still ok to log. + } + else if (value->count == t.count) + { + // t.count times has been reached. Was it within the window? + if (now_ms - value->first_ms < t.window_ms) + { + // Within the window, suppress the message. + rv = MESSAGE_SUPPRESSED; + } + else + { + // Not within the window, reset the situation. + + // The flooding situation is analyzed window by window. + // That means that if there in each of two consequtive + // windows are not enough messages for throttling to take + // effect, but there would be if the window was placed at a + // slightly different position (e.g. starting in the middle + // of the first and ending in the middle of the second) it + // will go undetected and no throttling will be made. + // However, if that's the case, it was a spike so the + // flooding will stop anyway. + + value->first_ms = now_ms; + value->count = 1; + } + } + else + { + // In suppression mode. + if (now_ms - value->first_ms < (t.window_ms + t.suppress_ms)) + { + // Still in the suppression window. + rv = MESSAGE_STILL_SUPPRESSED; + } + else + { + // We have exited the suppression window, reset the situation. + value->first_ms = now_ms; + value->count = 1; + } + } + + value->last_ms = now_ms; + + spinlock_release(&value->lock); + } + } + + return rv; +} + /** * Log a message of a particular priority. * * @param priority One of the syslog constants: LOG_ERR, LOG_WARNING, ... + * @param modname The name of the module. * @param file The name of the file where the message was logged. * @param line The line where the message was logged. * @param function The function where the message was logged. @@ -2746,6 +2903,7 @@ static enum log_flush priority_to_flush(int priority) * @param ... Optional arguments according to the format. */ int mxs_log_message(int priority, + const char* modname, const char* file, int line, const char* function, const char* format, ...) { @@ -2757,27 +2915,55 @@ int mxs_log_message(int priority, { if (MXS_LOG_PRIORITY_IS_ENABLED(priority)) { - va_list valist; + message_suppression_t status = MESSAGE_NOT_SUPPRESSED; - /** - * Find out the length of log string (to be formatted str). - */ - va_start(valist, format); - int message_len = vsnprintf(NULL, 0, format, valist); - va_end(valist); - - if (message_len >= 0) + // We only throttle errors and warnings. Info and debug messages + // are never on during normal operation, so if they are enabled, + // we are presumably debugging something. Notice messages are + // assumed to be logged for a reason and always in a context where + // flooding cannot be caused. + if ((priority == LOG_ERR) || (priority == LOG_WARNING)) { - log_prefix_t prefix = priority_to_prefix(priority); + status = message_status(file, line); + } - static const char FORMAT_FUNCTION[] = "(%s): "; + if (status != MESSAGE_STILL_SUPPRESSED) + { + va_list valist; - // Other thread might change log_config.augmentation. - int augmentation = log_config.augmentation; - int augmentation_len = 0; + int modname_len = modname ? strlen(modname) + 3 : 0; // +3 due to "[...] " - switch (augmentation) + static const char SUPPRESSION[] = + " (subsequent similar messages suppressed for %lu milliseconds)"; + int suppression_len = 0; + size_t suppress_ms = log_config.throttling.suppress_ms; + + if (status == MESSAGE_SUPPRESSED) { + suppression_len += sizeof(SUPPRESSION) - 1; // Remove trailing NULL + suppression_len += 3; // Remove the %lu + suppression_len += UINTLEN(suppress_ms); + } + + /** + * Find out the length of log string (to be formatted str). + */ + va_start(valist, format); + int message_len = vsnprintf(NULL, 0, format, valist); + va_end(valist); + + if (message_len >= 0) + { + log_prefix_t prefix = priority_to_prefix(priority); + + static const char FORMAT_FUNCTION[] = "(%s): "; + + // Other thread might change log_config.augmentation. + int augmentation = log_config.augmentation; + int augmentation_len = 0; + + switch (augmentation) + { case MXS_LOG_AUGMENT_WITH_FUNCTION: augmentation_len = sizeof(FORMAT_FUNCTION) - 1; // Remove trailing 0 augmentation_len -= 2; // Remove the %s @@ -2786,50 +2972,72 @@ int mxs_log_message(int priority, default: break; - } + } - int buffer_len = prefix.len + augmentation_len + message_len + 1; // Trailing NULL + int buffer_len = 0; + buffer_len += prefix.len; + buffer_len += modname_len; + buffer_len += augmentation_len; + buffer_len += message_len; + buffer_len += suppression_len; + buffer_len += 1; // Trailing NULL - if (buffer_len > MAX_LOGSTRLEN) - { - message_len -= (buffer_len - MAX_LOGSTRLEN); - buffer_len = MAX_LOGSTRLEN; - - assert(prefix.len + augmentation_len + message_len + 1 == buffer_len); - } - - char buffer[buffer_len]; - - char *prefix_text = buffer; - char *augmentation_text = buffer + prefix.len; - char *message_text = buffer + prefix.len + augmentation_len; - - strcpy(prefix_text, prefix.text); - - if (augmentation_len) - { - int len = 0; - - switch (augmentation) + if (buffer_len > MAX_LOGSTRLEN) { + message_len -= (buffer_len - MAX_LOGSTRLEN); + buffer_len = MAX_LOGSTRLEN; + + ss_dassert(prefix.len + modname_len + + augmentation_len + message_len + suppression_len + 1 == buffer_len); + } + + char buffer[buffer_len]; + + char *prefix_text = buffer; + char *modname_text = prefix_text + prefix.len; + char *augmentation_text = modname_text + modname_len; + char *message_text = augmentation_text + augmentation_len; + char *suppression_text = message_text + message_len; + + strcpy(prefix_text, prefix.text); + + if (modname_len) + { + strcpy(modname_text, "["); + strcat(modname_text, modname); + strcat(modname_text, "] "); + } + + if (augmentation_len) + { + int len = 0; + + switch (augmentation) + { case MXS_LOG_AUGMENT_WITH_FUNCTION: len = sprintf(augmentation_text, FORMAT_FUNCTION, function); break; default: assert(!true); + } + + assert(len == augmentation_len); } - assert(len == augmentation_len); + va_start(valist, format); + vsnprintf(message_text, message_len + 1, format, valist); + va_end(valist); + + if (suppression_len) + { + sprintf(suppression_text, SUPPRESSION, suppress_ms); + } + + enum log_flush flush = priority_to_flush(priority); + + err = log_write(priority, file, line, function, prefix.len, buffer_len, buffer, flush); } - - va_start(valist, format); - vsnprintf(message_text, message_len + 1, format, valist); - va_end(valist); - - enum log_flush flush = priority_to_flush(priority); - - err = log_write(priority, file, line, function, prefix.len, buffer_len, buffer, flush); } } } diff --git a/server/core/maxkeys.c b/server/core/maxkeys.c index 8d7501e7e..5103f908c 100644 --- a/server/core/maxkeys.c +++ b/server/core/maxkeys.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -22,33 +22,82 @@ * * @endverbatim */ +#include #include #include #include #include #include +struct option options[] = +{ + { + "help", + no_argument, + NULL, + 'h' + }, + { NULL, 0, NULL, 0 } +}; + +void print_usage(const char* executable, const char* directory) +{ + printf("usage: %s [-h|--help] [directory]\n" + "\n" + "This utility writes into the file .secrets, in the specified directory, the\n" + "AES encryption key and init vector that are used by the utility maxpasswd,\n" + "when encrypting passwords used in the MariaDB MaxScale configuration file.\n" + "\n" + "Note that re-creating the .secrets file will invalidate all existing\n" + "passwords used in the configuration file.\n" + "\n" + " -h, --help: Display this help.\n" + "\n" + "directory : The directory where the .secrets file should be created.\n" + "\n" + "If a specific directory is not provided, the file is created in\n" + "%s.\n", + executable, directory); +} + int main(int argc, char **argv) { - const char *keyfile; - int rval = 0; + const char* directory = get_datadir(); - if (argc < 2) + int c; + while ((c = getopt_long(argc, argv, "h", options, NULL)) != -1) { - keyfile = get_datadir(); - fprintf(stderr, "Generating .secrets file in %s ...\n", keyfile); + switch (c) + { + case 'h': + print_usage(argv[0], directory); + exit(EXIT_SUCCESS); + break; + + default: + print_usage(argv[0], directory); + exit(EXIT_FAILURE); + break; + } + } + + int rval = EXIT_SUCCESS; + + if (optind == argc) + { + fprintf(stderr, "Generating .secrets file in %s.\n", directory); } else { - keyfile = argv[1]; + directory = argv[optind]; } mxs_log_init(NULL, NULL, MXS_LOG_TARGET_DEFAULT); - if (secrets_writeKeys(keyfile)) + if (secrets_writeKeys(directory) != 0) { - fprintf(stderr, "Failed to encode the password\n"); - rval = 1; + fprintf(stderr, "Failed to create the .secrets file.\n"); + rval = EXIT_FAILURE; } mxs_log_flush_sync(); diff --git a/server/core/maxpasswd.c b/server/core/maxpasswd.c index 412fcd67f..0e1650b76 100644 --- a/server/core/maxpasswd.c +++ b/server/core/maxpasswd.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -23,57 +23,159 @@ * @endverbatim */ #include +#include +#include #include #include -#include -/** - * Encrypt a password for storing in the MaxScale.cnf file - * - * @param argc Argument count - * @param argv Argument vector - */ -int -main(int argc, char **argv) -{ - char *enc; - char *pw; - char *home; - int rval = 0; - if (argc != 3) +struct option options[] = +{ { - fprintf(stderr, "Usage: %s \n", argv[0]); - return 1; + "help", + no_argument, + NULL, + 'h' + }, + { NULL, 0, NULL, 0 } +}; + +void print_usage(const char* executable, const char* directory) +{ + printf("usage: %s [-h|--help] [path] password\n" + "\n" + "This utility creates an encrypted password using a .secrets file\n" + "that has earlier been created using maxkeys.\n" + "\n" + "-h, --help: Display this help.\n" + "\n" + "path : The directory where the .secrets file is located, or the path\n" + " of the .secrets file itself. Note that the name of the file\n" + " must be .secrets.\n" + "\n" + "If a path is not provided, the .secrets file is looked for in the\n" + "directory %s.\n", + executable, directory); +} + +bool path_is_ok(const char* path) +{ + // TODO: The check here is superfluous, in the sense that encryptPassword + // TODO: checks the same. However, so as not to get annoying notice output + // TODO: on success, notices are turned off and the absence of the file is + // TODO: reported as a notice. Thus, without this check maxpasswd would + // TODO: simply fail, without telling the cause. + + bool rv = false; + + struct stat st; + + if (stat(path, &st) == 0) + { + if (S_ISDIR(st.st_mode)) + { + const char TAIL[] = "/.secrets"; + + char file[strlen(path) + sizeof(TAIL)]; + strcpy(file, path); + strcat(file, TAIL); + + rv = path_is_ok(file); + } + else + { + // encryptPassword reports of any errors. + rv = true; + } + } + else + { + fprintf(stderr, "error: Could not access %s: %s.\n", path, strerror(errno)); } - mxs_log_init(NULL, NULL, MXS_LOG_TARGET_DEFAULT); + return rv; +} + +int main(int argc, char **argv) +{ + const char* path = get_datadir(); + + int c; + while ((c = getopt_long(argc, argv, "h", options, NULL)) != -1) + { + switch (c) + { + case 'h': + print_usage(argv[0], path); + exit(EXIT_SUCCESS); + break; + + default: + print_usage(argv[0], path); + exit(EXIT_FAILURE); + break; + } + } + + char* password; + + switch (argc - optind) + { + case 2: + // Two args provided. + path = argv[optind]; + if (!path_is_ok(path)) + { + exit(EXIT_FAILURE); + } + password = argv[optind + 1]; + break; + + case 1: + // One arg provided. + password = argv[optind]; + break; + + default: + print_usage(argv[0], path); + exit(EXIT_FAILURE); + break; + } + + // We'll ignore errors and steam ahead. + (void)mxs_log_init(NULL, NULL, MXS_LOG_TARGET_DEFAULT); mxs_log_set_priority_enabled(LOG_NOTICE, false); mxs_log_set_priority_enabled(LOG_INFO, false); mxs_log_set_priority_enabled(LOG_DEBUG, false); - pw = calloc(81, sizeof(char)); + size_t len = strlen(password); - if (pw == NULL) + if (len > MXS_PASSWORD_MAXLEN) { - fprintf(stderr, "Error: cannot allocate enough memory."); - return 1; + fprintf(stderr, + "warning: The provided password is %lu characters long. " + "Only the first %d will be used.\n", len, MXS_PASSWORD_MAXLEN); } - strncpy(pw, argv[2], 80); + char used_password[MXS_PASSWORD_MAXLEN + 1]; + strncpy(used_password, password, MXS_PASSWORD_MAXLEN); + used_password[MXS_PASSWORD_MAXLEN] = 0; - if ((enc = encryptPassword(argv[1], pw)) != NULL) + int rval = EXIT_SUCCESS; + + char* enc = encryptPassword(path, used_password); + if (enc) { printf("%s\n", enc); } else { - fprintf(stderr, "Failed to encode the password\n"); - rval = 1; + fprintf(stderr, "error: Failed to encode the password.\n"); + rval = EXIT_FAILURE; } - free(pw); mxs_log_flush_sync(); mxs_log_finish(); + return rval; } diff --git a/server/core/maxscale_pcre2.c b/server/core/maxscale_pcre2.c index 128cd765b..c13eb323b 100644 --- a/server/core/maxscale_pcre2.c +++ b/server/core/maxscale_pcre2.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -25,6 +25,7 @@ */ #include +#include /** * Utility wrapper for PCRE2 library function call pcre2_substitute. @@ -56,7 +57,7 @@ mxs_pcre2_result_t mxs_pcre2_substitute(pcre2_code *re, const char *subject, con (PCRE2_SPTR) replace, PCRE2_ZERO_TERMINATED, (PCRE2_UCHAR*) *dest, size)) == PCRE2_ERROR_NOMEMORY) { - char *tmp = realloc(*dest, *size * 2); + char *tmp = MXS_REALLOC(*dest, *size * 2); if (tmp == NULL) { break; diff --git a/server/core/memlog.c b/server/core/memlog.c index 15e15c1d0..20ed56ce5 100644 --- a/server/core/memlog.c +++ b/server/core/memlog.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -27,6 +27,8 @@ #include #include #include +#include +#include static MEMLOG *memlogs = NULL; static SPINLOCK memlock = SPINLOCK_INIT; @@ -43,14 +45,18 @@ static SPINLOCK memlock = SPINLOCK_INIT; MEMLOG * memlog_create(char *name, MEMLOGTYPE type, int size) { - MEMLOG *log; + name = MXS_STRDUP(name); - if ((log = (MEMLOG *)malloc(sizeof(MEMLOG))) == NULL) + MEMLOG *log = (MEMLOG *)MXS_MALLOC(sizeof(MEMLOG)); + + if (!name || !log) { + MXS_FREE(name); + MXS_FREE(log); return NULL; } - log->name = strdup(name); + log->name = name; spinlock_init(&log->lock); log->type = type; log->offset = 0; @@ -59,21 +65,22 @@ memlog_create(char *name, MEMLOGTYPE type, int size) switch (type) { case ML_INT: - log->values = malloc(sizeof(int) * size); + log->values = MXS_MALLOC(sizeof(int) * size); break; case ML_LONG: - log->values = malloc(sizeof(long) * size); + log->values = MXS_MALLOC(sizeof(long) * size); break; case ML_LONGLONG: - log->values = malloc(sizeof(long long) * size); + log->values = MXS_MALLOC(sizeof(long long) * size); break; case ML_STRING: - log->values = malloc(sizeof(char *) * size); + log->values = MXS_MALLOC(sizeof(char *) * size); break; } if (log->values == NULL) { - free(log); + MXS_FREE(name); + MXS_FREE(log); return NULL; } spinlock_acquire(&memlock); @@ -98,7 +105,7 @@ memlog_destroy(MEMLOG *log) { memlog_flush(log); } - free(log->values); + MXS_FREE(log->values); spinlock_acquire(&memlock); if (memlogs == log) @@ -118,8 +125,8 @@ memlog_destroy(MEMLOG *log) } } spinlock_release(&memlock); - free(log->name); - free(log); + MXS_FREE(log->name); + MXS_FREE(log); } /** diff --git a/server/core/misc.c b/server/core/misc.c index 0eecbb1d6..b43b362e6 100644 --- a/server/core/misc.c +++ b/server/core/misc.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/mlist.c b/server/core/mlist.c index e45b099e6..0242bcfff 100644 --- a/server/core/mlist.c +++ b/server/core/mlist.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -12,6 +12,7 @@ */ #include +#include static void mlist_free_memory(mlist_t* ml, char* name); static mlist_node_t* mlist_node_init(void* data, mlist_cursor_t* cursor); @@ -79,7 +80,7 @@ mlist_t* mlist_init(mlist_t* listp, mlist_cursor_t** cursor, char* name, /** listp is not NULL if caller wants flat list */ if (listp == NULL) { - list = (mlist_t*) calloc(1, sizeof (mlist_t)); + list = (mlist_t*) MXS_CALLOC(1, sizeof (mlist_t)); } else { @@ -91,7 +92,6 @@ mlist_t* mlist_init(mlist_t* listp, mlist_cursor_t** cursor, char* name, if (list == NULL) { - fprintf(stderr, "* Allocating memory for mlist failed\n"); mlist_free_memory(list, name); goto return_list; } @@ -161,7 +161,7 @@ static void mlist_free_memory(mlist_t* ml, char* name) /** name */ if (name != NULL) { - free(name); + MXS_FREE(name); } if (ml != NULL) { @@ -177,7 +177,7 @@ static void mlist_free_memory(mlist_t* ml, char* name) /** list structure */ if (!ml->mlist_flat) { - free(ml); + MXS_FREE(ml); } } } @@ -197,9 +197,9 @@ void mlist_node_done(mlist_node_t* n) { (n->mlnode_list->mlist_datadel(n->mlnode_data)); } - free(n->mlnode_data); + MXS_FREE(n->mlnode_data); } - free(n); + MXS_FREE(n); } /** @@ -254,7 +254,8 @@ static mlist_node_t* mlist_node_init(void* data, mlist_cursor_t* cursor) { mlist_node_t* node; - node = (mlist_node_t*) calloc(1, sizeof (mlist_node_t)); + node = (mlist_node_t*) MXS_CALLOC(1, sizeof (mlist_node_t)); + MXS_ABORT_IF_NULL(node); node->mlnode_chk_top = CHK_NUM_MLIST_NODE; node->mlnode_chk_tail = CHK_NUM_MLIST_NODE; node->mlnode_data = data; @@ -358,7 +359,7 @@ mlist_cursor_t* mlist_cursor_init(mlist_t* list) /** acquire shared lock to the list */ simple_mutex_lock(&list->mlist_mutex, true); - c = (mlist_cursor_t *) calloc(1, sizeof (mlist_cursor_t)); + c = (mlist_cursor_t *) MXS_CALLOC(1, sizeof (mlist_cursor_t)); if (c == NULL) { diff --git a/server/core/modutil.c b/server/core/modutil.c index 42e0c6673..77c8bd7f7 100644 --- a/server/core/modutil.c +++ b/server/core/modutil.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -283,7 +284,9 @@ modutil_get_SQL(GWBUF *buf) length += (*ptr++ << 8); length += (*ptr++ << 16); - if ((rval = (char *) malloc(length + 1))) + rval = (char *) MXS_MALLOC(length + 1); + + if (rval) { dptr = rval; ptr += 2; // Skip sequence id and COM_QUERY byte @@ -332,7 +335,7 @@ modutil_get_query(GWBUF *buf) { case MYSQL_COM_QUIT: len = strlen("[Quit msg]") + 1; - if ((query_str = (char *)malloc(len + 1)) == NULL) + if ((query_str = (char *)MXS_MALLOC(len + 1)) == NULL) { goto retblock; } @@ -342,8 +345,12 @@ modutil_get_query(GWBUF *buf) case MYSQL_COM_QUERY: len = MYSQL_GET_PACKET_LEN(packet) - 1; /*< distract 1 for packet type byte */ - if (len < 1 || len > ~(size_t)0 - 1 || (query_str = (char *)malloc(len + 1)) == NULL) + if (len < 1 || len > ~(size_t)0 - 1 || (query_str = (char *)MXS_MALLOC(len + 1)) == NULL) { + if (len >= 1 && len <= ~(size_t)0 - 1) + { + ss_dassert(!query_str); + } goto retblock; } memcpy(query_str, &packet[5], len); @@ -352,8 +359,12 @@ modutil_get_query(GWBUF *buf) default: len = strlen(STRPACKETTYPE(packet_type)) + 1; - if (len < 1 || len > ~(size_t)0 - 1 || (query_str = (char *)malloc(len + 1)) == NULL) + if (len < 1 || len > ~(size_t)0 - 1 || (query_str = (char *)MXS_MALLOC(len + 1)) == NULL) { + if (len >= 1 && len <= ~(size_t)0 - 1) + { + ss_dassert(!query_str); + } goto retblock; } memcpy(query_str, STRPACKETTYPE(packet_type), len); @@ -777,7 +788,7 @@ static void modutil_reply_routing_error(DCB* backend_dcb, CHK_DCB(backend_dcb); buf = modutil_create_mysql_err_msg(1, 0, error, state, errstr); - free(errstr); + MXS_FREE(errstr); if (buf == NULL) { @@ -1117,56 +1128,61 @@ mxs_pcre2_result_t modutil_mysql_wildcard_match(const char* pattern, const char* bool err = false; PCRE2_SIZE matchsize = strlen(string) + 1; PCRE2_SIZE tempsize = matchsize; - char* matchstr = (char*) malloc(matchsize); - char* tempstr = (char*) malloc(tempsize); + char* matchstr = (char*) MXS_MALLOC(matchsize); + char* tempstr = (char*) MXS_MALLOC(tempsize); - pcre2_match_data *mdata_percent = pcre2_match_data_create_from_pattern(re_percent, NULL); - pcre2_match_data *mdata_single = pcre2_match_data_create_from_pattern(re_single, NULL); - pcre2_match_data *mdata_escape = pcre2_match_data_create_from_pattern(re_escape, NULL); - - if (matchstr && tempstr && mdata_percent && mdata_single && mdata_escape) + if (matchstr && tempstr) { - if (mxs_pcre2_substitute(re_escape, pattern, sub_escape, - &matchstr, &matchsize) == MXS_PCRE2_ERROR || - mxs_pcre2_substitute(re_single, matchstr, sub_single, - &tempstr, &tempsize) == MXS_PCRE2_ERROR || - mxs_pcre2_substitute(re_percent, tempstr, sub_percent, - &matchstr, &matchsize) == MXS_PCRE2_ERROR) + pcre2_match_data *mdata_percent = pcre2_match_data_create_from_pattern(re_percent, NULL); + pcre2_match_data *mdata_single = pcre2_match_data_create_from_pattern(re_single, NULL); + pcre2_match_data *mdata_escape = pcre2_match_data_create_from_pattern(re_escape, NULL); + + if (mdata_percent && mdata_single && mdata_escape) + { + if (mxs_pcre2_substitute(re_escape, pattern, sub_escape, + &matchstr, &matchsize) == MXS_PCRE2_ERROR || + mxs_pcre2_substitute(re_single, matchstr, sub_single, + &tempstr, &tempsize) == MXS_PCRE2_ERROR || + mxs_pcre2_substitute(re_percent, tempstr, sub_percent, + &matchstr, &matchsize) == MXS_PCRE2_ERROR) + { + err = true; + } + + if (!err) + { + int errcode; + rval = mxs_pcre2_simple_match(matchstr, string, PCRE2_CASELESS, &errcode); + if (rval == MXS_PCRE2_ERROR) + { + if (errcode != 0) + { + PCRE2_UCHAR errbuf[STRERROR_BUFLEN]; + pcre2_get_error_message(errcode, errbuf, sizeof(errbuf)); + MXS_ERROR("Failed to match pattern: %s", errbuf); + } + err = true; + } + } + } + else { err = true; } - if (!err) + if (err) { - int errcode; - rval = mxs_pcre2_simple_match(matchstr, string, PCRE2_CASELESS, &errcode); - if (rval == MXS_PCRE2_ERROR) - { - if (errcode != 0) - { - PCRE2_UCHAR errbuf[STRERROR_BUFLEN]; - pcre2_get_error_message(errcode, errbuf, sizeof(errbuf)); - MXS_ERROR("Failed to match pattern: %s", errbuf); - } - err = true; - } + MXS_ERROR("Fatal error when matching wildcard patterns."); } - } - else - { - err = true; + + pcre2_match_data_free(mdata_percent); + pcre2_match_data_free(mdata_single); + pcre2_match_data_free(mdata_escape); } - if (err) - { - MXS_ERROR("Fatal error when matching wildcard patterns."); - } + MXS_FREE(matchstr); + MXS_FREE(tempstr); - pcre2_match_data_free(mdata_percent); - pcre2_match_data_free(mdata_single); - pcre2_match_data_free(mdata_escape); - free(matchstr); - free(tempstr); return rval; } @@ -1205,17 +1221,17 @@ char* modutil_get_canonical(GWBUF* querybuf) if (replace_values((const char**)&dest, &destsize, &src, &srcsize)) { querystr = squeeze_whitespace(src); - free(dest); + MXS_FREE(dest); } else { - free(src); - free(dest); + MXS_FREE(src); + MXS_FREE(dest); } } else { - free(src); + MXS_FREE(src); } } } diff --git a/server/core/monitor.c b/server/core/monitor.c index 7f229c2bd..6397e7e29 100644 --- a/server/core/monitor.c +++ b/server/core/monitor.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -40,6 +40,7 @@ #include #include #include +#include /* * Create declarations of the enum for monitor events and also the array of @@ -71,21 +72,26 @@ static void monitor_servers_free(MONITOR_SERVERS *servers); MONITOR * monitor_alloc(char *name, char *module) { - MONITOR *mon; + name = MXS_STRDUP(name); - if ((mon = (MONITOR *)malloc(sizeof(MONITOR))) == NULL) + MONITOR *mon = (MONITOR *)MXS_MALLOC(sizeof(MONITOR)); + + if (!name || !mon) { + MXS_FREE(name); + MXS_FREE(mon); return NULL; } if ((mon->module = load_module(module, MODULE_MONITOR)) == NULL) { MXS_ERROR("Unable to load monitor module '%s'.", name); - free(mon); + MXS_FREE(name); + MXS_FREE(mon); return NULL; } mon->state = MONITOR_STATE_ALLOC; - mon->name = strdup(name); + mon->name = name; mon->handle = NULL; mon->databases = NULL; mon->password = NULL; @@ -138,8 +144,8 @@ monitor_free(MONITOR *mon) spinlock_release(&monLock); free_config_parameter(mon->parameters); monitor_servers_free(mon->databases); - free(mon->name); - free(mon); + MXS_FREE(mon->name); + MXS_FREE(mon); } @@ -240,12 +246,9 @@ monitorStopAll() void monitorAddServer(MONITOR *mon, SERVER *server) { - MONITOR_SERVERS *ptr, *db; + MONITOR_SERVERS *db = (MONITOR_SERVERS *)MXS_MALLOC(sizeof(MONITOR_SERVERS)); + MXS_ABORT_IF_NULL(db); - if ((db = (MONITOR_SERVERS *)malloc(sizeof(MONITOR_SERVERS))) == NULL) - { - return; - } db->server = server; db->con = NULL; db->next = NULL; @@ -264,7 +267,7 @@ monitorAddServer(MONITOR *mon, SERVER *server) } else { - ptr = mon->databases; + MONITOR_SERVERS *ptr = mon->databases; while (ptr->next != NULL) { ptr = ptr->next; @@ -288,7 +291,7 @@ static void monitor_servers_free(MONITOR_SERVERS *servers) { mysql_close(tofree->con); } - free(tofree); + MXS_FREE(tofree); } } @@ -303,8 +306,8 @@ static void monitor_servers_free(MONITOR_SERVERS *servers) void monitorAddUser(MONITOR *mon, char *user, char *passwd) { - mon->user = strdup(user); - mon->password = strdup(passwd); + mon->user = MXS_STRDUP_A(user); + mon->password = MXS_STRDUP_A(passwd); } /** @@ -486,7 +489,7 @@ monitorRowCallback(RESULTSET *set, void *data) if (ptr == NULL) { spinlock_release(&monLock); - free(data); + MXS_FREE(data); return NULL; } (*rowno)++; @@ -509,14 +512,14 @@ monitorGetList() RESULTSET *set; int *data; - if ((data = (int *)malloc(sizeof(int))) == NULL) + if ((data = (int *)MXS_MALLOC(sizeof(int))) == NULL) { return NULL; } *data = 0; if ((set = resultset_create(monitorRowCallback, data)) == NULL) { - free(data); + MXS_FREE(data); return NULL; } resultset_add_column(set, "Monitor", 20, COL_TYPE_VARCHAR); @@ -613,7 +616,7 @@ bool check_monitor_permissions(MONITOR* monitor, const char* query) mysql_close(mysql); } - free(dpasswd); + MXS_FREE(dpasswd); return rval; } @@ -925,14 +928,15 @@ monitor_launch_script(MONITOR* mon, MONITOR_SERVERS* ptr, char* script) int mon_parse_event_string(bool* events, size_t count, char* given_string) { - char *tok, *saved, *string = strdup(given_string); + char *tok, *saved, *string = MXS_STRDUP(given_string); + MXS_ABORT_IF_NULL(string); monitor_event_t event; tok = strtok_r(string, ",| ", &saved); if (tok == NULL) { - free(string); + MXS_FREE(string); return -1; } @@ -942,7 +946,7 @@ mon_parse_event_string(bool* events, size_t count, char* given_string) if (event == UNDEFINED_MONITOR_EVENT) { MXS_ERROR("Invalid event name %s", tok); - free(string); + MXS_FREE(string); return -1; } if (event < count) @@ -952,7 +956,7 @@ mon_parse_event_string(bool* events, size_t count, char* given_string) } } - free(string); + MXS_FREE(string); return 0; } @@ -1006,7 +1010,7 @@ mon_connect_to_db(MONITOR* mon, MONITOR_SERVERS *database) } } - free(dpwd); + MXS_FREE(dpwd); } else { @@ -1044,6 +1048,6 @@ void mon_log_state_change(MONITOR_SERVERS *ptr) MXS_NOTICE("Server changed state: %s[%s:%u]: %s. [%s] -> [%s]", ptr->server->unique_name, ptr->server->name, ptr->server->port, mon_get_event_name(ptr), prev, next); - free(prev); - free(next); + MXS_FREE(prev); + MXS_FREE(next); } diff --git a/server/core/mysql_binlog.c b/server/core/mysql_binlog.c index 22da763a4..342c7c34c 100644 --- a/server/core/mysql_binlog.c +++ b/server/core/mysql_binlog.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/mysql_utils.c b/server/core/mysql_utils.c index 137c73898..7671bbf35 100644 --- a/server/core/mysql_utils.c +++ b/server/core/mysql_utils.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -23,8 +23,10 @@ #include #include #include +#include #include #include +#include /** * @brief Calculate the length of a length-encoded integer in bytes @@ -115,7 +117,7 @@ uint64_t leint_consume(uint8_t ** c) char* lestr_consume_dup(uint8_t** c) { uint64_t slen = leint_consume(c); - char *str = malloc((slen + 1) * sizeof(char)); + char *str = MXS_MALLOC((slen + 1) * sizeof(char)); if (str) { @@ -161,15 +163,9 @@ MYSQL *mxs_mysql_real_connect(MYSQL *con, SERVER *server, const char *user, cons if (listener) { -#ifdef CONNECTOR_C_SSL_AND_OPENSSL_INTERFERENCE_SORTED_OUT - // TODO: No conclusive evidence yet, but tentatively it seems that when OpenSSL is - // TODO: used explicitly (backend SSL) and in conjunction with Connector-C, the - // TODO: latter SSL becomes unstable. So for the time being the monitors and - // TODO: services (fetch users) do not use SSL when connecting to the backend. - + GATEWAY_CONF* config = config_get_global_options(); // mysql_ssl_set always returns true. mysql_ssl_set(con, listener->ssl_key, listener->ssl_cert, listener->ssl_ca_cert, NULL, NULL); -#endif } return mysql_real_connect(con, server->name, user, passwd, NULL, server->port, NULL, 0); diff --git a/server/core/poll.c b/server/core/poll.c index a2078f387..18361ca58 100644 --- a/server/core/poll.c +++ b/server/core/poll.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -69,6 +70,7 @@ int max_poll_sleep; * 07/07/15 Martin Brampton Simplified add and remove DCB, improve error handling. * 23/08/15 Martin Brampton Added test so only DCB with a session link can be added to the poll list * 07/02/16 Martin Brampton Added a small piece of SSL logic to EPOLLIN + * 15/06/16 Martin Brampton Changed ts_stats_add to inline ts_stats_increment * * @endverbatim */ @@ -213,14 +215,16 @@ poll_init() } if ((epoll_fd = epoll_create(MAX_EVENTS)) == -1) { - perror("epoll_create"); + char errbuf[STRERROR_BUFLEN]; + MXS_ERROR("FATAL: Could not create epoll instance: %s", strerror_r(errno, errbuf, sizeof(errbuf))); exit(-1); } memset(&pollStats, 0, sizeof(pollStats)); memset(&queueStats, 0, sizeof(queueStats)); bitmask_init(&poll_mask); n_threads = config_threadcount(); - if ((thread_data = (THREAD_DATA *)malloc(n_threads * sizeof(THREAD_DATA))) != NULL) + thread_data = (THREAD_DATA *)MXS_MALLOC(n_threads * sizeof(THREAD_DATA)); + if (thread_data); { for (i = 0; i < n_threads; i++) { @@ -239,7 +243,7 @@ poll_init() (pollStats.n_nothreads = ts_stats_alloc()) == NULL || (pollStats.blockingpolls = ts_stats_alloc()) == NULL) { - perror("Fatal error: Memory allocation failed."); + MXS_OOM_MESSAGE("FATAL: Could not allocate statistics data."); exit(-1); } @@ -249,12 +253,14 @@ poll_init() hktask_add("Load Average", poll_loadav, NULL, POLL_LOAD_FREQ); n_avg_samples = 15 * 60 / POLL_LOAD_FREQ; - avg_samples = (double *)malloc(sizeof(double) * n_avg_samples); + avg_samples = (double *)MXS_MALLOC(sizeof(double) * n_avg_samples); + MXS_ABORT_IF_NULL(avg_samples); for (i = 0; i < n_avg_samples; i++) { avg_samples[i] = 0.0; } - evqp_samples = (int *)malloc(sizeof(int) * n_avg_samples); + evqp_samples = (int *)MXS_MALLOC(sizeof(int) * n_avg_samples); + MXS_ABORT_IF_NULL(evqp_samples); for (i = 0; i < n_avg_samples; i++) { evqp_samples[i] = 0.0; @@ -556,8 +562,6 @@ poll_waitevents(void *arg) intptr_t thread_id = (intptr_t)arg; int poll_spins = 0; - ts_stats_set_thread_id(thread_id); - /** Add this thread to the bitmask of running polling threads */ bitmask_set(&poll_mask, thread_id); if (thread_data) @@ -585,7 +589,7 @@ poll_waitevents(void *arg) thread_data[thread_id].state = THREAD_POLLING; } - ts_stats_add(pollStats.n_polls, 1); + ts_stats_increment(pollStats.n_polls, thread_id); if ((nfds = epoll_wait(epoll_fd, events, MAX_EVENTS, 0)) == -1) { atomic_add(&n_waiting, -1); @@ -608,7 +612,7 @@ poll_waitevents(void *arg) */ else if (nfds == 0 && pollStats.evq_pending == 0 && poll_spins++ > number_poll_spins) { - ts_stats_add(pollStats.blockingpolls, 1); + ts_stats_increment(pollStats.blockingpolls, thread_id); nfds = epoll_wait(epoll_fd, events, MAX_EVENTS, @@ -626,7 +630,7 @@ poll_waitevents(void *arg) if (n_waiting == 0) { - ts_stats_add(pollStats.n_nothreads, 1); + ts_stats_increment(pollStats.n_nothreads, thread_id); } #if MUTEX_EPOLL simple_mutex_unlock(&epoll_wait_mutex); @@ -637,13 +641,13 @@ poll_waitevents(void *arg) timeout_bias = 1; if (poll_spins <= number_poll_spins + 1) { - ts_stats_add(pollStats.n_nbpollev, 1); + ts_stats_increment(pollStats.n_nbpollev, thread_id); } poll_spins = 0; MXS_DEBUG("%lu [poll_waitevents] epoll_wait found %d fds", pthread_self(), nfds); - ts_stats_add(pollStats.n_pollev, 1); + ts_stats_increment(pollStats.n_pollev, thread_id); if (thread_data) { thread_data[thread_id].n_fds = nfds; @@ -928,7 +932,7 @@ process_pollq(int thread_id) if (eno == 0) { - ts_stats_add(pollStats.n_write, 1); + ts_stats_increment(pollStats.n_write, thread_id); /** Read session id to thread's local storage */ dcb_get_ses_log_info(dcb, &mxs_log_tls.li_sesid, @@ -954,13 +958,13 @@ process_pollq(int thread_id) } if (ev & EPOLLIN) { - if (dcb->state == DCB_STATE_LISTENING) + if (dcb->state == DCB_STATE_LISTENING || dcb->state == DCB_STATE_WAITING) { MXS_DEBUG("%lu [poll_waitevents] " "Accept in fd %d", pthread_self(), dcb->fd); - ts_stats_add(pollStats.n_accept, 1); + ts_stats_increment(pollStats.n_accept, thread_id); dcb_get_ses_log_info(dcb, &mxs_log_tls.li_sesid, &mxs_log_tls.li_enabled_priorities); @@ -977,7 +981,7 @@ process_pollq(int thread_id) pthread_self(), dcb, dcb->fd); - ts_stats_add(pollStats.n_read, 1); + ts_stats_increment(pollStats.n_read, thread_id); /** Read session id to thread's local storage */ dcb_get_ses_log_info(dcb, &mxs_log_tls.li_sesid, @@ -1027,7 +1031,7 @@ process_pollq(int thread_id) eno, strerror_r(eno, errbuf, sizeof(errbuf))); } - ts_stats_add(pollStats.n_error, 1); + ts_stats_increment(pollStats.n_error, thread_id); /** Read session id to thread's local storage */ dcb_get_ses_log_info(dcb, &mxs_log_tls.li_sesid, @@ -1052,7 +1056,7 @@ process_pollq(int thread_id) dcb->fd, eno, strerror_r(eno, errbuf, sizeof(errbuf))); - ts_stats_add(pollStats.n_hup, 1); + ts_stats_increment(pollStats.n_hup, thread_id); spinlock_acquire(&dcb->dcb_initlock); if ((dcb->flags & DCBF_HUNG) == 0) { @@ -1088,7 +1092,7 @@ process_pollq(int thread_id) dcb->fd, eno, strerror_r(eno, errbuf, sizeof(errbuf))); - ts_stats_add(pollStats.n_hup, 1); + ts_stats_increment(pollStats.n_hup, thread_id); spinlock_acquire(&dcb->dcb_initlock); if ((dcb->flags & DCBF_HUNG) == 0) { @@ -1309,7 +1313,7 @@ event_to_string(uint32_t event) { char *str; - str = malloc(22); // 22 is max returned string length + str = MXS_MALLOC(22); // 22 is max returned string length if (str == NULL) { return NULL; @@ -1479,7 +1483,7 @@ dShowThreads(DCB *dcb) if (from_heap) { - free(event_string); + MXS_FREE(event_string); } } } @@ -1780,8 +1784,8 @@ dShowEventQ(DCB *pdcb) dcb->evq.processing ? "Processing" : "Pending", (tmp1 = event_to_string(dcb->evq.processing_events)), (tmp2 = event_to_string(dcb->evq.pending_events))); - free(tmp1); - free(tmp2); + MXS_FREE(tmp1); + MXS_FREE(tmp2); dcb = dcb->evq.next; } while (dcb != eventq); @@ -1870,7 +1874,7 @@ eventTimesRowCallback(RESULTSET *set, void *data) if (*rowno >= N_QUEUE_TIMES) { - free(data); + MXS_FREE(data); return NULL; } row = resultset_make_row(set); @@ -1911,14 +1915,14 @@ eventTimesGetList() RESULTSET *set; int *data; - if ((data = (int *)malloc(sizeof(int))) == NULL) + if ((data = (int *)MXS_MALLOC(sizeof(int))) == NULL) { return NULL; } *data = 0; if ((set = resultset_create(eventTimesRowCallback, data)) == NULL) { - free(data); + MXS_FREE(data); return NULL; } resultset_add_column(set, "Duration", 20, COL_TYPE_VARCHAR); diff --git a/server/core/query_classifier.c b/server/core/query_classifier.c index 91dc1df3d..442dd9e9b 100644 --- a/server/core/query_classifier.c +++ b/server/core/query_classifier.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -15,6 +15,7 @@ #include #include #include +#include //#define QC_TRACE_ENABLED #undef QC_TRACE_ENABLED @@ -229,13 +230,14 @@ char* qc_get_qtype_str(qc_query_type_t qtype) if (qtype_str == NULL) { - qtype_str = strdup(STRQTYPE(t)); + qtype_str = MXS_STRDUP_A(STRQTYPE(t)); } else { size_t len = strlen(STRQTYPE(t)); /** reallocate space for delimiter, new string and termination */ - qtype_str = (char *) realloc(qtype_str, strlen(qtype_str) + 1 + len + 1); + qtype_str = (char *) MXS_REALLOC(qtype_str, strlen(qtype_str) + 1 + len + 1); + MXS_ABORT_IF_NULL(qtype_str); snprintf(qtype_str + strlen(qtype_str), 1 + len + 1, "|%s", STRQTYPE(t)); } @@ -603,7 +605,7 @@ char* qc_types_to_string(uint32_t types) ++len; // Then make one allocation and build the string. - char* s = (char*) malloc(len); + char* s = (char*) MXS_MALLOC(len); if (s) { diff --git a/server/core/queuemanager.c b/server/core/queuemanager.c index 04adab7ea..ae615b1d7 100644 --- a/server/core/queuemanager.c +++ b/server/core/queuemanager.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -26,10 +26,18 @@ * @endverbatim */ #include +#include +#include #include +#include #include #include #include +#include + +#if defined(SS_DEBUG) +int debug_check_fail = 0; +#endif /* SS_DEBUG */ /** * @brief Allocate a new queue @@ -44,25 +52,23 @@ QUEUE_CONFIG *mxs_queue_alloc(int limit, int timeout) { - QUEUE_CONFIG *new_queue = NULL; - if (limit > CONNECTION_QUEUE_LIMIT) - { - MXS_ERROR("Limit configured for connection queue exceeds system maximum"); - limit = CONNECTION_QUEUE_LIMIT; - } - new_queue = (QUEUE_CONFIG *)calloc(1, sizeof(QUEUE_CONFIG)); + QUEUE_CONFIG *new_queue = (QUEUE_CONFIG *)MXS_CALLOC(1, sizeof(QUEUE_CONFIG)); if (new_queue) { - new_queue->queue_size = CONNECTION_QUEUE_LIMIT; - new_queue->queue_limit = limit; - new_queue->timeout = timeout; - spinlock_init(&new_queue->queue_lock); + new_queue->queue_array = MXS_CALLOC(limit+1, sizeof(QUEUE_ENTRY)); + if (new_queue->queue_array) + { + new_queue->queue_limit = limit; + new_queue->timeout = timeout; + spinlock_init(&new_queue->queue_lock); +#if defined(SS_DEBUG) + new_queue->sequence_number = 0; +#endif /* SS_DEBUG */ + return new_queue; + } + MXS_FREE(new_queue); } - else - { - MXS_ERROR("Failed to allocate memory for new queue in mxs_queue_alloc"); - } - return new_queue; + return NULL; } /** @@ -75,13 +81,18 @@ QUEUE_CONFIG */ void mxs_queue_free(QUEUE_CONFIG *queue_config) { - free(queue_config); + if (queue_config) + { + MXS_FREE(queue_config->queue_array); + MXS_FREE(queue_config); + } } /** * @brief Add an item to a queue * - * Add a new item to a FIFO queue + * Add a new item to a FIFO queue. If the queue config is null, this function + * will behave as if the queue is full. * * @param queue_config The configuration and anchor structure for the queue * @param new_entry The new entry, to be added @@ -94,16 +105,20 @@ bool mxs_enqueue(QUEUE_CONFIG *queue_config, void *new_entry) if (queue_config) { spinlock_acquire(&queue_config->queue_lock); - if (queue_config->queue_limit < queue_config->queue_size - && mxs_queue_count(queue_config) < queue_config->queue_size) + if (mxs_queue_count(queue_config) < queue_config->queue_limit) { queue_config->queue_array[queue_config->end].queued_object = new_entry; queue_config->queue_array[queue_config->end].heartbeat = hkheartbeat; +#if defined(SS_DEBUG) + queue_config->queue_array[queue_config->end].sequence_check = queue_config->sequence_number; + queue_config->sequence_number++; +#endif /* SS_DEBUG */ queue_config->end++; - if (queue_config->end >= queue_config->queue_size) + if (queue_config->end > queue_config->queue_limit) { queue_config->end = 0; } + queue_config->has_entries = true; result = true; } else @@ -118,25 +133,86 @@ bool mxs_enqueue(QUEUE_CONFIG *queue_config, void *new_entry) /** * @brief Remove an item from a queue * - * Remove an item from a FIFO queue + * Remove an item from a FIFO queue. If the queue config is NULL, the function + * will behave as if for an empty queue. * * @param queue_config The configuration and anchor structure for the queue - * @return QUEUE_ENTRY A structure defining the removed entry + * @param result A queue entry structure that will receive the result + * @return bool indicating whether an item was successfully dequeued */ -QUEUE_ENTRY *mxs_dequeue(QUEUE_CONFIG *queue_config) +bool mxs_dequeue(QUEUE_CONFIG *queue_config, QUEUE_ENTRY *result) { - int count; - QUEUE_ENTRY *result = NULL; + QUEUE_ENTRY *found = NULL; - spinlock_acquire(&queue_config->queue_lock); - if (NULL == result && (count = mxs_queue_count(queue_config)) > 0) + if (queue_config && queue_config->has_entries) { - result = &(queue_config->queue_array[queue_config->start++]); - if (queue_config->start >= queue_config->queue_size) + spinlock_acquire(&queue_config->queue_lock); + if (mxs_queue_count(queue_config) > 0) { - queue_config->start = 0; + found = &(queue_config->queue_array[queue_config->start]); +#if defined(SS_DEBUG) + ss_dassert((queue_config->sequence_number) == (found->sequence_check + mxs_queue_count(queue_config))); + if ((queue_config->sequence_number) != (found->sequence_check + mxs_queue_count(queue_config))) + { + debug_check_fail++; + } +#endif /* SS_DEBUG */ + result->heartbeat = found->heartbeat; + result->queued_object = found->queued_object; + if (++queue_config->start > queue_config->queue_limit) + { + queue_config->start = 0; + } + queue_config->has_entries = (mxs_queue_count(queue_config) > 0); } + spinlock_release(&queue_config->queue_lock); } - spinlock_release(&queue_config->queue_lock); - return result; + return (found != NULL); +} + +/** + * @brief Remove an item from a queue if it has passed the timeout limit + * + * Remove an item from a FIFO queue if expired. If the queue config is NULL, + * the function will behave as for an empty queue. + * + * @param queue_config The configuration and anchor structure for the queue + * @param result A queue entry structure that will receive the result + * @return bool indicating whether an item was successfully dequeued + */ +bool mxs_dequeue_if_expired(QUEUE_CONFIG *queue_config, QUEUE_ENTRY *result) +{ + QUEUE_ENTRY *found = NULL; + + if (queue_config && queue_config->has_entries) + { + spinlock_acquire(&queue_config->queue_lock); + if (mxs_queue_count(queue_config) > 0) + { + found = &(queue_config->queue_array[queue_config->start]); + if (found->heartbeat > hkheartbeat - queue_config->timeout) + { + found = NULL; + } + else + { +#if defined(SS_DEBUG) + ss_dassert((queue_config->sequence_number) == (found->sequence_check + mxs_queue_count(queue_config))); + if ((queue_config->sequence_number) != (found->sequence_check + mxs_queue_count(queue_config))) + { + debug_check_fail++; + } +#endif /* SS_DEBUG */ + result->heartbeat = found->heartbeat; + result->queued_object = found->queued_object; + if (++queue_config->start > queue_config->queue_limit) + { + queue_config->start = 0; + } + queue_config->has_entries = (mxs_queue_count(queue_config) > 0); + } + } + spinlock_release(&queue_config->queue_lock); + } + return (found != NULL); } diff --git a/server/core/random_jkiss.c b/server/core/random_jkiss.c index cb52f350e..83b3d5e19 100644 --- a/server/core/random_jkiss.c +++ b/server/core/random_jkiss.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/resultset.c b/server/core/resultset.c index d18b30afd..4213cd07f 100644 --- a/server/core/resultset.c +++ b/server/core/resultset.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -25,6 +25,7 @@ #include #include +#include #include #include #include @@ -46,9 +47,9 @@ static int mysql_send_row(DCB *, RESULT_ROW *, int); RESULTSET * resultset_create(RESULT_ROW_CB func, void *data) { - RESULTSET *rval; + RESULTSET *rval = (RESULTSET *)MXS_MALLOC(sizeof(RESULTSET)); - if ((rval = (RESULTSET *)malloc(sizeof(RESULTSET))) != NULL) + if (rval) { rval->n_cols = 0; rval->column = NULL; @@ -79,7 +80,7 @@ resultset_free(RESULTSET *resultset) resultset_column_free(col); col = next; } - free(resultset); + MXS_FREE(resultset); } } @@ -96,17 +97,17 @@ resultset_free(RESULTSET *resultset) int resultset_add_column(RESULTSET *set, char *name, int len, RESULT_COL_TYPE type) { - RESULT_COLUMN *newcol, *ptr; + name = MXS_STRDUP(name); + RESULT_COLUMN *newcol = (RESULT_COLUMN *)MXS_MALLOC(sizeof(RESULT_COLUMN)); - if ((newcol = (RESULT_COLUMN *)malloc(sizeof(RESULT_COLUMN))) == NULL) + if (!name || !newcol) { + MXS_FREE(name); + MXS_FREE(newcol); return 0; } - if ((newcol->name = strdup(name)) == NULL) - { - free(newcol); - return 0; - } + + newcol->name = name; newcol->type = type; newcol->len = len; newcol->next = NULL; @@ -117,7 +118,7 @@ resultset_add_column(RESULTSET *set, char *name, int len, RESULT_COL_TYPE type) } else { - ptr = set->column; + RESULT_COLUMN *ptr = set->column; while (ptr->next) { ptr = ptr->next; @@ -136,8 +137,8 @@ resultset_add_column(RESULTSET *set, char *name, int len, RESULT_COL_TYPE type) void resultset_column_free(RESULT_COLUMN *col) { - free(col->name); - free(col); + MXS_FREE(col->name); + MXS_FREE(col); } /** @@ -153,14 +154,14 @@ resultset_make_row(RESULTSET *set) RESULT_ROW *row; int i; - if ((row = (RESULT_ROW *)malloc(sizeof(RESULT_ROW))) == NULL) + if ((row = (RESULT_ROW *)MXS_MALLOC(sizeof(RESULT_ROW))) == NULL) { return NULL; } row->n_cols = set->n_cols; - if ((row->cols = (char **)malloc(row->n_cols * sizeof(char *))) == NULL) + if ((row->cols = (char **)MXS_MALLOC(row->n_cols * sizeof(char *))) == NULL) { - free(row); + MXS_FREE(row); return NULL; } @@ -188,11 +189,11 @@ resultset_free_row(RESULT_ROW *row) { if (row->cols[i]) { - free(row->cols[i]); + MXS_FREE(row->cols[i]); } } - free(row->cols); - free(row); + MXS_FREE(row->cols); + MXS_FREE(row); } /** @@ -214,7 +215,7 @@ resultset_row_set(RESULT_ROW *row, int col, char *value) } if (value) { - if ((row->cols[col] = strdup(value)) == NULL) + if ((row->cols[col] = MXS_STRDUP(value)) == NULL) { return 0; } @@ -222,7 +223,7 @@ resultset_row_set(RESULT_ROW *row, int col, char *value) } else if (row->cols[col]) { - free(row->cols[col]); + MXS_FREE(row->cols[col]); } row->cols[col] = NULL; return 1; @@ -423,7 +424,7 @@ mysql_send_row(DCB *dcb, RESULT_ROW *row, int seqno) { len = strlen(row->cols[i]); *ptr++ = len; - strncpy((char *)ptr, row->cols[i], len); + memcpy(ptr, row->cols[i], len); ptr += len; } else diff --git a/server/core/secrets.c b/server/core/secrets.c index d5edb927c..70014a2ee 100644 --- a/server/core/secrets.c +++ b/server/core/secrets.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -19,6 +19,7 @@ #include #include #include +#include #include "gw.h" @@ -53,30 +54,53 @@ secrets_random_str(unsigned char *output, int len) static MAXKEYS * secrets_readKeys(const char* path) { - char secret_file[PATH_MAX + 1]; - char *home; + static const char NAME[] = ".secrets"; + char secret_file[PATH_MAX + 1 + sizeof(NAME)]; // Worst case: maximum path + "/" + name. MAXKEYS *keys; struct stat secret_stats; - int fd; - int len; static int reported = 0; if (path != NULL) { - snprintf(secret_file, PATH_MAX, "%s", path); - - char *file; - if ((file = strrchr(secret_file, '.')) == NULL || strcmp(file, ".secrets") != 0) + size_t len = strlen(path); + if (len > PATH_MAX) { - /** This is a possible path to a directory */ - strncat(secret_file, "/.secrets", PATH_MAX); + MXS_ERROR("Too long (%lu > %d) path provided.", len, PATH_MAX); + return NULL; + } + + if (stat(path, &secret_stats) == 0) + { + if (S_ISDIR(secret_stats.st_mode)) + { + sprintf(secret_file, "%s/%s", path, NAME); + } + else + { + // If the provided path does not refer to a directory, then the + // file name *must* be ".secrets". + char *file; + if ((file = strrchr(secret_file, '.')) == NULL || strcmp(file, NAME) != 0) + { + MXS_ERROR("The name of the secrets file must be \"%s\".", NAME); + return NULL; + } + } + } + else + { + char errbuf[STRERROR_BUFLEN]; + MXS_ERROR("The provided path \"%s\" does not exist or cannot be accessed. " + "Error: %d, %s.", path, errno, strerror_r(errno, errbuf, sizeof(errbuf))); + return NULL; } clean_up_pathname(secret_file); } else { - snprintf(secret_file, PATH_MAX, "%s/.secrets", get_datadir()); + // We assume that get_datadir() returns a path shorter than PATH_MAX. + sprintf(secret_file, "%s/%s", get_datadir(), NAME); } /* Try to access secrets file */ if (access(secret_file, R_OK) == -1) @@ -108,7 +132,8 @@ secrets_readKeys(const char* path) } /* open secret file */ - if ((fd = open(secret_file, O_RDONLY)) < 0) + int fd = open(secret_file, O_RDONLY); + if (fd < 0) { int eno = errno; errno = 0; @@ -159,10 +184,9 @@ secrets_readKeys(const char* path) return NULL; } - if ((keys = (MAXKEYS *) malloc(sizeof(MAXKEYS))) == NULL) + if ((keys = (MAXKEYS *) MXS_MALLOC(sizeof(MAXKEYS))) == NULL) { close(fd); - MXS_ERROR("Memory allocation failed for key structure."); return NULL; } @@ -170,17 +194,17 @@ secrets_readKeys(const char* path) * Read all data from file. * MAXKEYS (secrets.h) is struct for key, _not_ length-related macro. */ - len = read(fd, keys, sizeof(MAXKEYS)); + ssize_t len = read(fd, keys, sizeof(MAXKEYS)); if (len != sizeof(MAXKEYS)) { int eno = errno; errno = 0; close(fd); - free(keys); + MXS_FREE(keys); char errbuf[STRERROR_BUFLEN]; MXS_ERROR("Read from secrets file " - "%s failed. Read %d, expected %d bytes. Error %d, %s.", + "%s failed. Read %ld, expected %d bytes. Error %d, %s.", secret_file, len, (int)sizeof(MAXKEYS), @@ -194,7 +218,7 @@ secrets_readKeys(const char* path) { int eno = errno; errno = 0; - free(keys); + MXS_FREE(keys); char errbuf[STRERROR_BUFLEN]; MXS_ERROR("Failed closing the " "secrets file %s. Error %d, %s.", @@ -221,23 +245,23 @@ secrets_readKeys(const char* path) * This routine writes into a binary file the AES encryption key * and the AES Init Vector * - * @param secret_file The file with secret keys + * @param dir The directory where the ".secrets" file should be created. * @return 0 on success and 1 on failure */ -int secrets_writeKeys(const char *path) +int secrets_writeKeys(const char *dir) { int fd, randfd; unsigned int randval; MAXKEYS key; char secret_file[PATH_MAX + 10]; - if (strlen(path) > PATH_MAX) + if (strlen(dir) > PATH_MAX) { MXS_ERROR("Pathname too long."); return 1; } - snprintf(secret_file, PATH_MAX + 9, "%s/.secrets", path); + snprintf(secret_file, PATH_MAX + 9, "%s/.secrets", dir); clean_up_pathname(secret_file); /* Open for writing | Create | Truncate the file for writing */ @@ -321,7 +345,7 @@ int secrets_writeKeys(const char *path) * Note the return is always a malloc'd string that the caller must free * * @param crypt The encrypted password - * @return The decrypted password + * @return The decrypted password or NULL if allocation failure. */ char * decryptPassword(const char *crypt) @@ -336,7 +360,7 @@ decryptPassword(const char *crypt) keys = secrets_readKeys(NULL); if (!keys) { - return strdup(crypt); + return MXS_STRDUP(crypt); } /* ** If the input is not a HEX string return the input @@ -346,24 +370,24 @@ decryptPassword(const char *crypt) { if (!isxdigit(*ptr)) { - free(keys); - return strdup(crypt); + MXS_FREE(keys); + return MXS_STRDUP(crypt); } } enlen = strlen(crypt) / 2; gw_hex2bin(encrypted, crypt, strlen(crypt)); - if ((plain = (unsigned char *) malloc(80)) == NULL) + if ((plain = (unsigned char *) MXS_MALLOC(80)) == NULL) { - free(keys); + MXS_FREE(keys); return NULL; } AES_set_decrypt_key(keys->enckey, 8 * MAXSCALE_KEYLEN, &aeskey); AES_cbc_encrypt(encrypted, plain, enlen, &aeskey, keys->initvector, AES_DECRYPT); - free(keys); + MXS_FREE(keys); return (char *) plain; } @@ -383,24 +407,27 @@ encryptPassword(const char* path, const char *password) AES_KEY aeskey; int padded_len; char *hex_output; - unsigned char padded_passwd[80]; - unsigned char encrypted[80]; + unsigned char padded_passwd[MXS_PASSWORD_MAXLEN + 1]; + unsigned char encrypted[MXS_PASSWORD_MAXLEN + 1]; if ((keys = secrets_readKeys(path)) == NULL) { return NULL; } - memset(padded_passwd, 0, 80); - strncpy((char *) padded_passwd, password, 79); - padded_len = ((strlen(password) / AES_BLOCK_SIZE) + 1) * AES_BLOCK_SIZE; + memset(padded_passwd, 0, MXS_PASSWORD_MAXLEN + 1); + strncpy((char *) padded_passwd, password, MXS_PASSWORD_MAXLEN); + padded_len = ((strlen((char*)padded_passwd) / AES_BLOCK_SIZE) + 1) * AES_BLOCK_SIZE; AES_set_encrypt_key(keys->enckey, 8 * MAXSCALE_KEYLEN, &aeskey); AES_cbc_encrypt(padded_passwd, encrypted, padded_len, &aeskey, keys->initvector, AES_ENCRYPT); - hex_output = (char *) malloc(padded_len * 2); - gw_bin2hex(hex_output, encrypted, padded_len); - free(keys); + hex_output = (char *) MXS_MALLOC(padded_len * 2); + if (hex_output) + { + gw_bin2hex(hex_output, encrypted, padded_len); + } + MXS_FREE(keys); return hex_output; } diff --git a/server/core/server.c b/server/core/server.c index e39c605a8..ceae72e38 100644 --- a/server/core/server.c +++ b/server/core/server.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -43,6 +43,7 @@ #include #include #include +#include static SPINLOCK server_spin = SPINLOCK_INIT; static SERVER *allServers = NULL; @@ -63,18 +64,25 @@ static void server_parameter_free(SERVER_PARAM *tofree); SERVER * server_alloc(char *servname, char *protocol, unsigned short port) { - SERVER *server; + servname = MXS_STRNDUP(servname, MAX_SERVER_NAME_LEN); + protocol = MXS_STRDUP(protocol); - if ((server = (SERVER *)calloc(1, sizeof(SERVER))) == NULL) + SERVER *server = (SERVER *)MXS_CALLOC(1, sizeof(SERVER)); + + if (!servname || !protocol || !server) { + MXS_FREE(servname); + MXS_FREE(protocol); + MXS_FREE(server); return NULL; } + #if defined(SS_DEBUG) server->server_chk_top = CHK_NUM_SERVER; server->server_chk_tail = CHK_NUM_SERVER; #endif - server->name = strndup(servname, MAX_SERVER_NAME_LEN); - server->protocol = strdup(protocol); + server->name = servname; + server->protocol = protocol; server->port = port; server->status = SERVER_RUNNING; server->node_id = -1; @@ -132,18 +140,18 @@ server_free(SERVER *tofreeserver) spinlock_release(&server_spin); /* Clean up session and free the memory */ - free(tofreeserver->name); - free(tofreeserver->protocol); - free(tofreeserver->unique_name); - free(tofreeserver->server_string); - free(tofreeserver->slaves); + MXS_FREE(tofreeserver->name); + MXS_FREE(tofreeserver->protocol); + MXS_FREE(tofreeserver->unique_name); + MXS_FREE(tofreeserver->server_string); + MXS_FREE(tofreeserver->slaves); server_parameter_free(tofreeserver->parameters); if (tofreeserver->persistent) { dcb_persistent_clean_count(tofreeserver->persistent, true); } - free(tofreeserver); + MXS_FREE(tofreeserver); return 1; } @@ -183,7 +191,7 @@ server_get_persistent(SERVER *server, char *user, const char *protocol) { previous->nextpersistent = dcb->nextpersistent; } - free(dcb->user); + MXS_FREE(dcb->user); dcb->user = NULL; spinlock_release(&server->persistlock); atomic_add(&server->stats.n_persistent, -1); @@ -221,7 +229,7 @@ server_get_persistent(SERVER *server, char *user, const char *protocol) void server_set_unique_name(SERVER *server, char *name) { - server->unique_name = strdup(name); + server->unique_name = MXS_STRDUP_A(name); } /** @@ -367,7 +375,7 @@ dprintAllServersJson(DCB *dcb) stat = server_status(server); dcb_printf(dcb, " \"status\": \"%s\",\n", stat); - free(stat); + MXS_FREE(stat); dcb_printf(dcb, " \"protocol\": \"%s\",\n", server->protocol); dcb_printf(dcb, " \"port\": \"%d\",\n", @@ -446,7 +454,7 @@ dprintServer(DCB *dcb, SERVER *server) dcb_printf(dcb, "\tServer: %s\n", server->name); char* stat = server_status(server); dcb_printf(dcb, "\tStatus: %s\n", stat); - free(stat); + MXS_FREE(stat); dcb_printf(dcb, "\tProtocol: %s\n", server->protocol); dcb_printf(dcb, "\tPort: %d\n", server->port); if (server->server_string) @@ -591,7 +599,7 @@ dListServers(DCB *dcb) server->unique_name, server->name, server->port, server->stats.n_current, stat); - free(stat); + MXS_FREE(stat); server = server->next; } if (allServers) @@ -613,7 +621,7 @@ server_status(SERVER *server) { char *status = NULL; - if (NULL == server || (status = (char *)malloc(256)) == NULL) + if (NULL == server || (status = (char *)MXS_MALLOC(256)) == NULL) { return NULL; } @@ -740,8 +748,8 @@ server_transfer_status(SERVER *dest_server, SERVER *source_server) void serverAddMonUser(SERVER *server, char *user, char *passwd) { - server->monuser = strdup(user); - server->monpw = strdup(passwd); + server->monuser = MXS_STRDUP_A(user); + server->monpw = MXS_STRDUP_A(passwd); } /** @@ -765,8 +773,8 @@ server_update(SERVER *server, char *protocol, char *user, char *passwd) MXS_NOTICE("Update server protocol for server %s to protocol %s.", server->name, protocol); - free(server->protocol); - server->protocol = strdup(protocol); + MXS_FREE(server->protocol); + server->protocol = MXS_STRDUP_A(protocol); } if (user != NULL && passwd != NULL) @@ -776,8 +784,8 @@ server_update(SERVER *server, char *protocol, char *user, char *passwd) { MXS_NOTICE("Update server monitor credentials for server %s", server->name); - free(server->monuser); - free(server->monpw); + MXS_FREE(server->monuser); + MXS_FREE(server->monpw); serverAddMonUser(server, user, passwd); } } @@ -797,24 +805,21 @@ server_update(SERVER *server, char *protocol, char *user, char *passwd) void serverAddParameter(SERVER *server, char *name, char *value) { - SERVER_PARAM *param; + name = MXS_STRDUP(name); + value = MXS_STRDUP(value); - if ((param = (SERVER_PARAM *)malloc(sizeof(SERVER_PARAM))) == NULL) + SERVER_PARAM *param = (SERVER_PARAM *)MXS_MALLOC(sizeof(SERVER_PARAM)); + + if (!name || !value || !param) { - return; - } - if ((param->name = strdup(name)) == NULL) - { - free(param); - return; - } - if ((param->value = strdup(value)) == NULL) - { - free(param->value); - free(param); + MXS_FREE(name); + MXS_FREE(value); + MXS_FREE(param); return; } + param->name = name; + param->value = value; param->next = server->parameters; server->parameters = param; } @@ -831,9 +836,9 @@ static void server_parameter_free(SERVER_PARAM *tofree) { param = tofree; tofree = tofree->next; - free(param->name); - free(param->value); - free(param); + MXS_FREE(param->name); + MXS_FREE(param->value); + MXS_FREE(param); } } @@ -886,7 +891,7 @@ serverRowCallback(RESULTSET *set, void *data) if (server == NULL) { spinlock_release(&server_spin); - free(data); + MXS_FREE(data); return NULL; } (*rowno)++; @@ -899,7 +904,7 @@ serverRowCallback(RESULTSET *set, void *data) resultset_row_set(row, 3, buf); stat = server_status(server); resultset_row_set(row, 4, stat); - free(stat); + MXS_FREE(stat); spinlock_release(&server_spin); return row; } @@ -915,14 +920,14 @@ serverGetList() RESULTSET *set; int *data; - if ((data = (int *)malloc(sizeof(int))) == NULL) + if ((data = (int *)MXS_MALLOC(sizeof(int))) == NULL) { return NULL; } *data = 0; if ((set = resultset_create(serverRowCallback, data)) == NULL) { - free(data); + MXS_FREE(data); return NULL; } resultset_add_column(set, "Server", 20, COL_TYPE_VARCHAR); @@ -949,9 +954,9 @@ server_update_address(SERVER *server, char *address) { if (server->name) { - free(server->name); + MXS_FREE(server->name); } - server->name = strdup(address); + server->name = MXS_STRDUP_A(address); } spinlock_release(&server_spin); } @@ -1021,13 +1026,20 @@ server_map_status(char *str) bool server_set_version_string(SERVER* server, const char* string) { bool rval = true; - spinlock_acquire(&server->lock); - free(server->server_string); - if ((server->server_string = strdup(string)) == NULL) + string = MXS_STRDUP(string); + + if (string) + { + spinlock_acquire(&server->lock); + char* old = server->server_string; + server->server_string = (char*)string; + spinlock_release(&server->lock); + MXS_FREE(old); + } + else { - MXS_ERROR("Memory allocation failed."); rval = false; } - spinlock_release(&server->lock); + return rval; } diff --git a/server/core/service.c b/server/core/service.c index ddb9ec99b..491e2e812 100644 --- a/server/core/service.c +++ b/server/core/service.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -66,6 +66,7 @@ #include #include #include +#include /** To be used with configuration type checks */ typedef struct typelib_st @@ -96,6 +97,7 @@ static int find_type(typelib_t* tl, const char* needle, int maxlen); static void service_add_qualified_param(SERVICE* svc, CONFIG_PARAMETER* param); static void service_internal_restart(void *data); +static void service_queue_check(void *data); /** * Allocate a new service for the gateway to support @@ -109,12 +111,19 @@ static void service_internal_restart(void *data); SERVICE * service_alloc(const char *servname, const char *router) { - SERVICE *service; + servname = MXS_STRDUP(servname); + router = MXS_STRDUP(router); - if ((service = (SERVICE *)calloc(1, sizeof(SERVICE))) == NULL) + SERVICE *service = (SERVICE *)MXS_CALLOC(1, sizeof(*service)); + + if (!servname || !router || !service) { + MXS_FREE((void*)servname); + MXS_FREE((void*)router); + MXS_FREE(service); return NULL; } + if ((service->router = load_module(router, MODULE_ROUTER)) == NULL) { char* home = get_libdir(); @@ -130,15 +139,17 @@ service_alloc(const char *servname, const char *router) home, ldpath ? "\t\t\t - " : "", ldpath ? ldpath : ""); - free(service); + MXS_FREE((void*)servname); + MXS_FREE((void*)router); + MXS_FREE(service); return NULL; } + service->client_count = 0; - service->name = strdup(servname); - service->routerModule = strdup(router); + service->name = (char*)servname; + service->routerModule = (char*)router; service->users_from_all = false; service->queued_connections = NULL; - service->resources = NULL; service->localhost_match_wildcard_host = SERVICE_PARAM_UNINIT; service->retry_start = true; service->conn_idle_timeout = SERVICE_NO_SESSION_TIMEOUT; @@ -147,7 +158,6 @@ service_alloc(const char *servname, const char *router) service->credentials.name = NULL; service->version_string = NULL; service->svc_config_param = NULL; - service->users = NULL; service->routerOptions = NULL; service->log_auth_warnings = true; service->strip_db_esc = true; @@ -155,9 +165,9 @@ service_alloc(const char *servname, const char *router) { if (service->name) { - free(service->name); + MXS_FREE(service->name); } - free(service); + MXS_FREE(service); return NULL; } service->stats.started = time(0); @@ -230,106 +240,83 @@ serviceStartPort(SERVICE *service, SERV_LISTENER *port) goto retblock; } + port->listener->service = service; + if (port->ssl) { listener_init_SSL(port->ssl); } + /** + * The following code should be located inside the authenticator modules. + * + * The authentication of users is done against the data being gathered here + * but the actual authentication happens inside the authenticator module. + * Since not all authenticators use the same data for authentication, it + * should be up to the authenticator to decide where the user data is retrieved + * and how it is stored. + * + * One way to do it would be to add a @c dcb->authfunc.loadusers(dcb) + * entry point which loads the users and a @c dcb->authfunc.freeusers(dcb) + * which in turn frees the users. It would also make sense to have a + * @c dcb->authfunc.reloadusers(dcb) entry point which would allow for a more + * optimized user reloading process instead of freeing and loading the users + * again. + */ + if (strcmp(port->protocol, "MySQLClient") == 0) { int loaded; - if (service->users == NULL) + if (port->users == NULL) { /* * Allocate specific data for MySQL users * including hosts and db names */ - service->users = mysql_users_alloc(); + port->users = mysql_users_alloc(); + const char* cachedir = get_cachedir(); - if ((loaded = load_mysql_users(service)) < 0) + if ((loaded = load_mysql_users(port)) < 0) { - MXS_ERROR("Unable to load users for " - "service %s listening at %s:%d.", - service->name, - (port->address == NULL ? "0.0.0.0" : port->address), + MXS_ERROR("Unable to load users for service %s listening at %s:%d.", + service->name, port->address ? port->address : "0.0.0.0", port->port); + /* Try loading authentication data from file cache */ + char path[PATH_MAX]; + sprintf(path, "%s/%s/%s/%s/%s", cachedir, service->name, port->name, + DBUSERS_DIR, DBUSERS_FILE); + + if ((loaded = dbusers_load(port->users, path)) == -1) { - /* Try loading authentication data from file cache */ - char *ptr, path[PATH_MAX + 1]; - strncpy(path, get_cachedir(), sizeof(path) - 1); - strncat(path, "/", sizeof(path) - 1); - strncat(path, service->name, sizeof(path) - 1); - strncat(path, "/.cache/dbusers", sizeof(path) - 1); - loaded = dbusers_load(service->users, path); - if (loaded != -1) - { - MXS_ERROR("Using cached credential information."); - } - } - if (loaded == -1) - { - users_free(service->users); - service->users = NULL; + MXS_ERROR("Failed to load cached users from '%s'.", path); + users_free(port->users); dcb_close(port->listener); - port->listener = NULL; goto retblock; } + else + { + MXS_WARNING("Using cached credential information."); + } } else { /* Save authentication data to file cache */ - char *ptr, path[PATH_MAX + 1]; - int mkdir_rval = 0; - strncpy(path, get_cachedir(), PATH_MAX); - strncat(path, "/", 4096); - strncat(path, service->name, PATH_MAX); - if (access(path, R_OK) == -1) - { - mkdir_rval = mkdir(path, 0777); - } + char path[PATH_MAX]; + sprintf(path, "%s/%s/%s/%s/", cachedir, service->name, port->name, DBUSERS_DIR); - if (mkdir_rval) + if (mxs_mkdir_all(path, 0777)) { - if (errno != EEXIST) - { - char errbuf[STRERROR_BUFLEN]; - MXS_ERROR("Failed to create directory '%s': [%d] %s", - path, - errno, - strerror_r(errno, errbuf, sizeof(errbuf))); - } - mkdir_rval = 0; + strcat(path, DBUSERS_FILE); + dbusers_save(port->users, path); } - - strncat(path, "/.cache", PATH_MAX); - if (access(path, R_OK) == -1) - { - mkdir_rval = mkdir(path, 0777); - } - - if (mkdir_rval) - { - if (errno != EEXIST) - { - char errbuf[STRERROR_BUFLEN]; - MXS_ERROR("Failed to create directory '%s': [%d] %s", - path, - errno, - strerror_r(errno, errbuf, sizeof(errbuf))); - } - mkdir_rval = 0; - } - strncat(path, "/dbusers", PATH_MAX); - dbusers_save(service->users, path); } + if (loaded == 0) { - MXS_ERROR("Service %s: failed to load any user " - "information. Authentication will " - "probably fail as a result.", - service->name); + MXS_ERROR("[%s]: failed to load any user information. Authentication" + " will probably fail as a result.", service->name); } /* At service start last update is set to USERS_REFRESH_TIME seconds earlier. @@ -344,19 +331,16 @@ serviceStartPort(SERVICE *service, SERV_LISTENER *port) } else { - if (service->users == NULL) + if (port->users == NULL) { /* Generic users table */ - service->users = users_alloc(); + port->users = users_alloc(); } } if ((funcs = (GWPROTOCOL *)load_module(port->protocol, MODULE_PROTOCOL)) == NULL) { - users_free(service->users); - service->users = NULL; dcb_close(port->listener); - service->users = NULL; port->listener = NULL; MXS_ERROR("Unable to load protocol module %s. Listener " "for service %s not started.", @@ -388,12 +372,8 @@ serviceStartPort(SERVICE *service, SERV_LISTENER *port) { MXS_ERROR("Failed to create session to service %s.", service->name); - - users_free(service->users); - service->users = NULL; dcb_close(port->listener); port->listener = NULL; - service->users = NULL; goto retblock; } } @@ -403,8 +383,6 @@ serviceStartPort(SERVICE *service, SERV_LISTENER *port) port->port, port->protocol, service->name); - users_free(service->users); - service->users = NULL; dcb_close(port->listener); port->listener = NULL; } @@ -477,13 +455,13 @@ static char** copy_string_array(char** original) values++; } - array = malloc(sizeof(char*) * (values + 1)); + array = MXS_MALLOC(sizeof(char*) * (values + 1)); if (array) { for (int i = 0; i < values; i++) { - array[i] = strdup(original[i]); + array[i] = MXS_STRDUP_A(original[i]); } array[values] = NULL; } @@ -498,9 +476,9 @@ static void free_string_array(char** array) { for (int i = 0; array[i]; i++) { - free(array[i]); + MXS_FREE(array[i]); } - free(array); + MXS_FREE(array); } } @@ -703,22 +681,20 @@ service_free(SERVICE *service) { srv = service->dbref; service->dbref = service->dbref->next; - free(srv); + MXS_FREE(srv); } - free(service->name); - free(service->routerModule); - free(service->weightby); - free(service->version_string); - free(service->credentials.name); - free(service->credentials.authdata); + MXS_FREE(service->name); + MXS_FREE(service->routerModule); + MXS_FREE(service->weightby); + MXS_FREE(service->version_string); + MXS_FREE(service->credentials.name); + MXS_FREE(service->credentials.authdata); free_config_parameter(service->svc_config_param); - users_free(service->users); - hashtable_free(service->resources); serviceClearRouterOptions(service); - free(service); + MXS_FREE(service); return 1; } @@ -734,12 +710,13 @@ service_free(SERVICE *service) * @return TRUE if the protocol/port could be added */ int -serviceAddProtocol(SERVICE *service, char *protocol, char *address, unsigned short port, char *authenticator, +serviceAddProtocol(SERVICE *service, char *name, char *protocol, char *address, unsigned short port, char *authenticator, SSL_LISTENER *ssl) { - SERV_LISTENER *proto; + SERV_LISTENER *proto = listener_alloc(service, name, protocol, address, + port, authenticator, ssl); - if ((proto = listener_alloc(protocol, address, port, authenticator, ssl)) != NULL) + if (proto) { spinlock_acquire(&service->spin); proto->next = service->ports; @@ -791,7 +768,7 @@ int serviceHasProtocol(SERVICE *service, const char *protocol, void serviceAddBackend(SERVICE *service, SERVER *server) { - SERVER_REF *sref = malloc(sizeof(SERVER_REF)); + SERVER_REF *sref = MXS_MALLOC(sizeof(SERVER_REF)); if (sref) { @@ -853,8 +830,9 @@ serviceAddRouterOption(SERVICE *service, char *option) spinlock_acquire(&service->spin); if (service->routerOptions == NULL) { - service->routerOptions = (char **)calloc(2, sizeof(char *)); - service->routerOptions[0] = strdup(option); + service->routerOptions = (char **)MXS_CALLOC(2, sizeof(char *)); + MXS_ABORT_IF_NULL(service->routerOptions); + service->routerOptions[0] = MXS_STRDUP_A(option); service->routerOptions[1] = NULL; } else @@ -863,9 +841,9 @@ serviceAddRouterOption(SERVICE *service, char *option) { ; } - service->routerOptions = (char **)realloc(service->routerOptions, - (i + 2) * sizeof(char *)); - service->routerOptions[i] = strdup(option); + service->routerOptions = (char **)MXS_REALLOC(service->routerOptions, (i + 2) * sizeof(char *)); + MXS_ABORT_IF_NULL(service->routerOptions); + service->routerOptions[i] = MXS_STRDUP_A(option); service->routerOptions[i + 1] = NULL; } spinlock_release(&service->spin); @@ -886,9 +864,9 @@ serviceClearRouterOptions(SERVICE *service) { for (i = 0; service->routerOptions[i]; i++) { - free(service->routerOptions[i]); + MXS_FREE(service->routerOptions[i]); } - free(service->routerOptions); + MXS_FREE(service->routerOptions); service->routerOptions = NULL; } spinlock_release(&service->spin); @@ -905,21 +883,22 @@ serviceClearRouterOptions(SERVICE *service) int serviceSetUser(SERVICE *service, char *user, char *auth) { - if (service->credentials.name) - { - free(service->credentials.name); - } - if (service->credentials.authdata) - { - free(service->credentials.authdata); - } - service->credentials.name = strdup(user); - service->credentials.authdata = strdup(auth); + user = MXS_STRDUP(user); + auth = MXS_STRDUP(auth); - if (service->credentials.name == NULL || service->credentials.authdata == NULL) + if (!user || !auth) { + MXS_FREE(user); + MXS_FREE(auth); return 0; } + + MXS_FREE(service->credentials.name); + MXS_FREE(service->credentials.authdata); + + service->credentials.name = user; + service->credentials.authdata = auth; + return 1; } @@ -1053,13 +1032,43 @@ serviceSetConnectionLimits(SERVICE *service, int max, int queued, int timeout) service->max_connections = max; if (queued && timeout) { + char callback_name[100]; + sprintf(callback_name, "Check queued connections %p", service); /* If memory allocation fails, result will be null so no queue */ service->queued_connections = mxs_queue_alloc(queued, timeout); + if (service->queued_connections) + { + hktask_add(callback_name, service_queue_check, (void *)service->queued_connections, 1); + } } return 1; } +/* + * @brief The callback function triggered by housekeeping every second + * + * This function removes any expired connection requests from the queue, and + * sends an error message "too many connections" for them. + * + * @param The parameter provided by the callback is the queue config + */ +static void +service_queue_check(void *data) +{ + QUEUE_ENTRY expired; + QUEUE_CONFIG *queue_config = (QUEUE_CONFIG *)data; + /* The queued connections are in a FIFO queue, so we only look at the */ + /* start of the queue, and remove any expired entries. As soon as this */ + /* returns nothing, we stop. */ + while ((mxs_dequeue_if_expired(queue_config, &expired))) + { + DCB *dcb = (DCB *)expired.queued_object; + dcb->func.connlimit(dcb, queue_config->queue_limit); + dcb_close(dcb); + } +} + /** * Enable or disable the restarting of the service on failure. * @param service Service to configure @@ -1089,9 +1098,8 @@ serviceSetFilters(SERVICE *service, char *filters) int n = 0; bool rval = true; - if ((flist = (FILTER_DEF **) malloc(sizeof(FILTER_DEF *))) == NULL) + if ((flist = (FILTER_DEF **) MXS_MALLOC(sizeof(FILTER_DEF *))) == NULL) { - MXS_ERROR("Out of memory adding filters to service.\n"); return false; } ptr = strtok_r(filters, "|", &brkt); @@ -1099,10 +1107,9 @@ serviceSetFilters(SERVICE *service, char *filters) { n++; FILTER_DEF **tmp; - if ((tmp = (FILTER_DEF **) realloc(flist, - (n + 1) * sizeof(FILTER_DEF *))) == NULL) + if ((tmp = (FILTER_DEF **) MXS_REALLOC(flist, + (n + 1) * sizeof(FILTER_DEF *))) == NULL) { - MXS_ERROR("Out of memory adding filters to service."); rval = false; break; } @@ -1139,7 +1146,7 @@ serviceSetFilters(SERVICE *service, char *filters) } else { - free(flist); + MXS_FREE(flist); } return rval; @@ -1203,7 +1210,14 @@ printService(SERVICE *service) } printf("\n"); } - printf("\tUsers data: %p\n", (void *)service->users); + + SERV_LISTENER *port = service->ports; + while (port) + { + printf("\tUsers data: %p\n", (void *)port->users); + port = port->next; + } + printf("\tTotal connections: %d\n", service->stats.n_sessions); printf("\tCurrently connected: %d\n", service->stats.n_current); } @@ -1313,8 +1327,14 @@ void dprintService(DCB *dcb, SERVICE *service) dcb_printf(dcb, "\tRouting weight parameter: %s\n", service->weightby); } - dcb_printf(dcb, "\tUsers data: %p\n", - service->users); + + SERV_LISTENER *port = service->ports; + while (port) + { + dcb_printf(dcb, "\tUsers data: %p\n", port->users); + port = port->next; + } + dcb_printf(dcb, "\tTotal connections: %d\n", service->stats.n_sessions); dcb_printf(dcb, "\tCurrently connected: %d\n", @@ -1429,8 +1449,8 @@ service_update(SERVICE *service, char *router, char *user, char *auth) MXS_NOTICE("Update router for service %s to %s.", service->name, router); - free(service->routerModule); - service->routerModule = strdup(router); + MXS_FREE(service->routerModule); + service->routerModule = MXS_STRDUP_A(router); service->router = router_obj; } } @@ -1484,7 +1504,17 @@ int service_refresh_users(SERVICE *service) service->rate_limit.last = time(NULL); } - ret = replace_mysql_users(service); + SERV_LISTENER *port = service->ports; + while (port) + { + if (replace_mysql_users(port) < 0) + { + ret = -1; + break; + } + ret++; + port = port->next; + } /* remove lock */ spinlock_release(&service->users_table_spin); @@ -1706,7 +1736,7 @@ static void service_add_qualified_param(SERVICE* svc, { svc->svc_config_param = p; } - free(old); + MXS_FREE(old); break; } prev = p; @@ -1749,9 +1779,9 @@ serviceWeightBy(SERVICE *service, char *weightby) { if (service->weightby) { - free(service->weightby); + MXS_FREE(service->weightby); } - service->weightby = strdup(weightby); + service->weightby = MXS_STRDUP_A(weightby); } /** @@ -1868,7 +1898,7 @@ serviceListenerRowCallback(RESULTSET *set, void *data) if (lptr == NULL) { spinlock_release(&service_spin); - free(data); + MXS_FREE(data); return NULL; } (*rowno)++; @@ -1897,14 +1927,14 @@ serviceGetListenerList() RESULTSET *set; int *data; - if ((data = (int *)malloc(sizeof(int))) == NULL) + if ((data = (int *)MXS_MALLOC(sizeof(int))) == NULL) { return NULL; } *data = 0; if ((set = resultset_create(serviceListenerRowCallback, data)) == NULL) { - free(data); + MXS_FREE(data); return NULL; } resultset_add_column(set, "Service Name", 25, COL_TYPE_VARCHAR); @@ -1942,7 +1972,7 @@ serviceRowCallback(RESULTSET *set, void *data) if (service == NULL) { spinlock_release(&service_spin); - free(data); + MXS_FREE(data); return NULL; } (*rowno)++; @@ -1968,14 +1998,14 @@ serviceGetList() RESULTSET *set; int *data; - if ((data = (int *)malloc(sizeof(int))) == NULL) + if ((data = (int *)MXS_MALLOC(sizeof(int))) == NULL) { return NULL; } *data = 0; if ((set = resultset_create(serviceRowCallback, data)) == NULL) { - free(data); + MXS_FREE(data); return NULL; } resultset_add_column(set, "Service Name", 25, COL_TYPE_VARCHAR); diff --git a/server/core/session.c b/server/core/session.c index be290256d..3aef74e1d 100644 --- a/server/core/session.c +++ b/server/core/session.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -23,6 +23,7 @@ * 29/05/14 Mark Riddoch Addition of filter mechanism * 23/08/15 Martin Brampton Tidying; slight improvement in safety * 17/09/15 Martin Brampton Keep failed session in existence - leave DCBs to close + * 27/06/16 Martin Brampton Amend to utilise list manager * * @endverbatim */ @@ -31,7 +32,9 @@ #include #include #include +#include #include +#include #include #include #include @@ -41,14 +44,15 @@ #include #include -/** Global session id; updated safely by holding session_spin */ -static size_t session_id; +/* This list of all sessions */ +LIST_CONFIG SESSIONlist = +{LIST_TYPE_RECYCLABLE, sizeof(SESSION), SPINLOCK_INIT}; -static SPINLOCK session_spin = SPINLOCK_INIT; -static SESSION *allSessions = NULL; -static SESSION *lastSession = NULL; -static SESSION *wasfreeSession = NULL; -static int freeSessionCount = 0; +/* A session with null values, used for initialization */ +static SESSION session_initialized = SESSION_INIT; + +/** Global session id; updated safely by use of atomic_add */ +static int session_id; static struct session session_dummy_struct; @@ -60,11 +64,43 @@ long next_timeout_check = 0; static SPINLOCK timeout_lock = SPINLOCK_INIT; +static void session_initialize(void *session); static int session_setup_filters(SESSION *session); static void session_simple_free(SESSION *session, DCB *dcb); static void session_add_to_all_list(SESSION *session); static SESSION *session_find_free(); static void session_final_free(SESSION *session); +static list_entry_t *skip_maybe_to_next_non_listener(list_entry_t *current, SESSIONLISTFILTER filter); + +/** + * @brief Initialize a session + * + * This routine puts initial values into the fields of the session pointed to + * by the parameter. The parameter has to be passed as void * because the + * function can be called by the generic list manager, which does not know + * the actual type of the list entries it handles. + * + * All fields can be initialized by the assignment of the static + * initialized session. + * + * @param *session Pointer to the session to be initialized + */ +static void +session_initialize(void *session) +{ + *(SESSION *)session = session_initialized; +} + +/* + * @brief Pre-allocate memory for a number of sessions + * + * @param The number of sessions to be pre-allocated + */ +bool +session_pre_alloc(int number) +{ + return list_pre_alloc(&SESSIONlist, number, session_initialize); +} /** * Allocate a new session for a new client of the specified service. @@ -82,32 +118,17 @@ session_alloc(SERVICE *service, DCB *client_dcb) { SESSION *session; - spinlock_acquire(&session_spin); - session = session_find_free(); - spinlock_release(&session_spin); + session = (SESSION *)list_find_free(&SESSIONlist, session_initialize); ss_info_dassert(session != NULL, "Allocating memory for session failed."); - - if (session == NULL) + if (NULL == session) { - char errbuf[STRERROR_BUFLEN]; - MXS_ERROR("Failed to allocate memory for " - "session object due error %d, %s.", - errno, - strerror_r(errno, errbuf, sizeof(errbuf))); + MXS_OOM(); return NULL; } -#if defined(SS_DEBUG) - session->ses_chk_top = CHK_NUM_SESSION; - session->ses_chk_tail = CHK_NUM_SESSION; -#endif session->ses_is_child = (bool) DCB_IS_CLONE(client_dcb); - spinlock_init(&session->ses_lock); session->service = service; session->client_dcb = client_dcb; - session->n_filters = 0; - memset(&session->stats, 0, sizeof(SESSION_STATS)); session->stats.connect = time(0); - session->state = SESSION_STATE_ALLOC; /*< * Associate the session to the client DCB and set the reference count on * the session to indicate that there is a single reference to the @@ -199,106 +220,17 @@ session_alloc(SERVICE *service, DCB *client_dcb) session->client_dcb->user, session->client_dcb->remote); } - spinlock_acquire(&session_spin); /** Assign a session id and increase, insert session into list */ - session->ses_id = ++session_id; - spinlock_release(&session_spin); + session->ses_id = (size_t)atomic_add(&session_id, 1) + 1; atomic_add(&service->stats.n_sessions, 1); atomic_add(&service->stats.n_current, 1); CHK_SESSION(session); client_dcb->session = session; + session->entry_is_ready = true; return SESSION_STATE_TO_BE_FREED == session->state ? NULL : session; } -/** - * Add a new session to the list of all sessions. - * - * Must be called with the general session lock held. - * - * A pointer, lastSession, is held to find the end of the list, and the new session - * is linked to the end of the list. The pointer, wasfreeSession, that is used to - * search for a free session is initialised if not already set. There cannot be - * any free sessions (or any at all) until this routine has been called at least - * once. Hence it will not be referred to until after it is initialised. - * - * @param session The session to be added to the list - */ -static void -session_add_to_all_list(SESSION *session) -{ - if (allSessions == NULL) - { - allSessions = session; - } - else - { - lastSession->next = session; - } - lastSession = session; - if (NULL == wasfreeSession) - { - wasfreeSession = session; - } -} - -/** - * Find a free session or allocate memory for a new one. - * - * This routine looks to see whether there are free session memory areas. - * If not, new memory is allocated, if possible, and the new session is added to - * the list of all sessions. - * - * Must be called with the general session lock held. - * - * @return An available session or NULL if none could be allocated. - */ -static SESSION * -session_find_free() -{ - SESSION *nextsession; - - if (freeSessionCount <= 0) - { - SESSION *newsession; - if ((newsession = calloc(1, sizeof(SESSION))) == NULL) - { - return NULL; - } - newsession->next = NULL; - session_add_to_all_list(newsession); - newsession->ses_is_in_use = true; - return newsession; - } - /* Starting at the last place a free session was found, loop through the */ - /* list of sessions searching for one that is not in use. */ - while (wasfreeSession->ses_is_in_use) - { - int loopcount = 0; - wasfreeSession = wasfreeSession->next; - if (NULL == wasfreeSession) - { - loopcount++; - if (loopcount > 1) - { - /* Shouldn't need to loop round more than once */ - MXS_ERROR("Find free session failed to find a session even" - " though free count was positive"); - return NULL; - } - wasfreeSession = allSessions; - } - } - /* Dropping out of the loop means we have found a session that is not in use */ - freeSessionCount--; - /* Clear the old data, then reset the list forward link */ - nextsession = wasfreeSession->next; - memset(wasfreeSession, 0, sizeof(SESSION)); - wasfreeSession->next = nextsession; - wasfreeSession->ses_is_in_use = true; - return wasfreeSession; -} - /** * Allocate a dummy session so that DCBs can always have sessions. * @@ -313,10 +245,10 @@ session_set_dummy(DCB *client_dcb) SESSION *session; session = &session_dummy_struct; -#if defined(SS_DEBUG) + session->list_entry_chk_top = CHK_NUM_MANAGED_LIST; + session->list_entry_chk_tail = CHK_NUM_MANAGED_LIST; session->ses_chk_top = CHK_NUM_SESSION; session->ses_chk_tail = CHK_NUM_SESSION; -#endif session->ses_is_child = false; spinlock_init(&session->ses_lock); session->service = NULL; @@ -405,7 +337,7 @@ session_simple_free(SESSION *session, DCB *dcb) { void * clientdata = dcb->data; dcb->data = NULL; - free(clientdata); + MXS_FREE(clientdata); } if (session) { @@ -487,7 +419,7 @@ session_free(SESSION *session) session->filters[i].session); } } - free(session->filters); + MXS_FREE(session->filters); } MXS_INFO("Stopped %s client session [%lu]", @@ -510,10 +442,7 @@ static void session_final_free(SESSION *session) { /* We never free the actual session, it is available for reuse*/ - spinlock_acquire(&session_spin); - session->ses_is_in_use = false; - freeSessionCount++; - spinlock_release(&session_spin); + list_free_entry(&SESSIONlist, (list_entry_t *)session); } /** @@ -525,21 +454,18 @@ session_final_free(SESSION *session) int session_isvalid(SESSION *session) { - SESSION *list_session; int rval = 0; - - spinlock_acquire(&session_spin); - list_session = allSessions; - while (list_session) + list_entry_t *current = list_start_iteration(&SESSIONlist); + while (current) { - if (list_session->ses_is_in_use && list_session == session) + if ((SESSION *)current == session) { rval = 1; + list_terminate_iteration_early(&SESSIONlist, current); break; } - list_session = list_session->next; + current = list_iterate(&SESSIONlist, current); } - spinlock_release(&session_spin); return rval; } @@ -572,19 +498,12 @@ printSession(SESSION *session) void printAllSessions() { - SESSION *list_session; - - spinlock_acquire(&session_spin); - list_session = allSessions; - while (list_session) + list_entry_t *current = list_start_iteration(&SESSIONlist); + while (current) { - if (list_session->ses_is_in_use) - { - printSession(list_session); - } - list_session = list_session->next; + printSession((SESSION *)current); + current = list_iterate(&SESSIONlist, current); } - spinlock_release(&session_spin); } @@ -597,19 +516,14 @@ printAllSessions() void CheckSessions() { - SESSION *list_session; + list_entry_t *current; int noclients = 0; int norouter = 0; - spinlock_acquire(&session_spin); - list_session = allSessions; - while (list_session) + current = list_start_iteration(&SESSIONlist); + while (current) { - if (false == list_session->ses_is_in_use) - { - list_session = list_session->next; - continue; - } + SESSION *list_session = (SESSION *)current; if (list_session->state != SESSION_STATE_LISTENER || list_session->state != SESSION_STATE_LISTENER_STOPPED) { @@ -624,22 +538,16 @@ CheckSessions() noclients++; } } - list_session = list_session->next; + current = list_iterate(&SESSIONlist, current); } - spinlock_release(&session_spin); if (noclients) { printf("%d Sessions have no clients\n", noclients); } - spinlock_acquire(&session_spin); - list_session = allSessions; - while (list_session) + current = list_start_iteration(&SESSIONlist); + while (current) { - if (false == list_session->ses_is_in_use) - { - list_session = list_session->next; - continue; - } + SESSION *list_session = (SESSION *)current; if (list_session->state != SESSION_STATE_LISTENER || list_session->state != SESSION_STATE_LISTENER_STOPPED) { @@ -654,15 +562,25 @@ CheckSessions() norouter++; } } - list_session = list_session->next; + current = list_iterate(&SESSIONlist, current); } - spinlock_release(&session_spin); if (norouter) { printf("%d Sessions have no router session\n", norouter); } } +/* + * @brief Print session list statistics + * + * @param pdcb DCB to print results to + */ +void +dprintSessionList(DCB *pdcb) +{ + dprintListStats(pdcb, &SESSIONlist, "All Sessions"); +} + /** * Print all sessions to a DCB * @@ -674,24 +592,14 @@ CheckSessions() void dprintAllSessions(DCB *dcb) { - SESSION *list_session; - spinlock_acquire(&session_spin); - list_session = allSessions; - while (list_session) + list_entry_t *current = list_start_iteration(&SESSIONlist); + while (current) { - if (false == list_session->ses_is_in_use) - { - list_session = list_session->next; - continue; - } - - dprintSession(dcb, list_session); - - list_session = list_session->next; + dprintSession(dcb, (SESSION *)current); + current = list_iterate(&SESSIONlist, current); } - spinlock_release(&session_spin); -} + } /** * Print a particular session to a DCB @@ -705,35 +613,35 @@ dprintAllSessions(DCB *dcb) void dprintSession(DCB *dcb, SESSION *print_session) { + struct tm result; + char buf[30]; + int i; + dcb_printf(dcb, "Session %lu (%p)\n", print_session->ses_id, print_session); dcb_printf(dcb, "\tState: %s\n", session_state(print_session->state)); dcb_printf(dcb, "\tService: %s (%p)\n", print_session->service->name, print_session->service); dcb_printf(dcb, "\tClient DCB: %p\n", print_session->client_dcb); if (print_session->client_dcb && print_session->client_dcb->remote) - { - dcb_printf(dcb, "\tClient Address: %s%s%s\n", - print_session->client_dcb->user ? print_session->client_dcb->user : "", - print_session->client_dcb->user ? "@" : "", - print_session->client_dcb->remote); - } - - struct tm result; - char buf[30]; - - dcb_printf(dcb, "\tConnected: %s", // asctime inserts newline. - asctime_r(localtime_r(&print_session->stats.connect, &result), buf)); - - if (print_session->client_dcb && print_session->client_dcb->state == DCB_STATE_POLLING) { double idle = (hkheartbeat - print_session->client_dcb->last_read); - idle = idle > 0 ? idle / 10.f : 0; - dcb_printf(dcb, "\tIdle: %.0f seconds\n", idle); + idle = idle > 0 ? idle/10.f : 0; + dcb_printf(dcb, "\tClient Address: %s%s%s\n", + print_session->client_dcb->user?print_session->client_dcb->user:"", + print_session->client_dcb->user?"@":"", + print_session->client_dcb->remote); + dcb_printf(dcb, "\tConnected: %s\n", + asctime_r(localtime_r(&print_session->stats.connect, &result), buf)); + if (print_session->client_dcb->state == DCB_STATE_POLLING) + { + dcb_printf(dcb, "\tIdle: %.0f seconds\n",idle); + } + } if (print_session->n_filters) { - for (int i = 0; i < print_session->n_filters; i++) + for (i = 0; i < print_session->n_filters; i++) { dcb_printf(dcb, "\tFilter: %s\n", print_session->filters[i].filter->name); @@ -755,36 +663,32 @@ dprintSession(DCB *dcb, SESSION *print_session) void dListSessions(DCB *dcb) { - SESSION *list_session; - - spinlock_acquire(&session_spin); - list_session = allSessions; - if (list_session) + bool written_heading = false; + list_entry_t *current = list_start_iteration(&SESSIONlist); + if (current) { dcb_printf(dcb, "Sessions.\n"); dcb_printf(dcb, "-----------------+-----------------+----------------+--------------------------\n"); dcb_printf(dcb, "Session | Client | Service | State\n"); dcb_printf(dcb, "-----------------+-----------------+----------------+--------------------------\n"); + written_heading = true; } - while (list_session) + while (current) { - if (list_session->ses_is_in_use) - { - dcb_printf(dcb, "%-16p | %-15s | %-14s | %s\n", list_session, - ((list_session->client_dcb && list_session->client_dcb->remote) - ? list_session->client_dcb->remote : ""), - (list_session->service && list_session->service->name ? list_session->service->name - : ""), - session_state(list_session->state)); - } - list_session = list_session->next; + SESSION *list_session = (SESSION *)current; + dcb_printf(dcb, "%-16p | %-15s | %-14s | %s\n", list_session, + ((list_session->client_dcb && list_session->client_dcb->remote) + ? list_session->client_dcb->remote : ""), + (list_session->service && list_session->service->name ? list_session->service->name + : ""), + session_state(list_session->state)); + current = list_iterate(&SESSIONlist, current); } - if (allSessions) + if (written_heading) { dcb_printf(dcb, "-----------------+-----------------+----------------+--------------------------\n\n"); } - spinlock_release(&session_spin); } /** @@ -821,20 +725,25 @@ session_state(session_state_t state) } } +/* + * @brief Find the session that relates to a given router session + * + * @param rses A router session + * @return The related session, or NULL if none + */ SESSION* get_session_by_router_ses(void* rses) { - SESSION* ses = allSessions; - - while (((ses->ses_is_in_use == false) || (ses->router_session != rses)) && ses->next != NULL) + list_entry_t *current = list_start_iteration(&SESSIONlist); + while (current) { - ses = ses->next; + if (((SESSION *)current)->router_session == rses) + { + list_terminate_iteration_early(&SESSIONlist, current); + return (SESSION *)current; + } + current = list_iterate(&SESSIONlist, current); } - - if (ses->ses_is_in_use == false || ses->router_session != rses) - { - ses = NULL; - } - return ses; + return NULL; } @@ -858,11 +767,9 @@ session_setup_filters(SESSION *session) UPSTREAM *tail; int i; - if ((session->filters = calloc(service->n_filters, - sizeof(SESSION_FILTER))) == NULL) + if ((session->filters = MXS_CALLOC(service->n_filters, + sizeof(SESSION_FILTER))) == NULL) { - MXS_ERROR("Insufficient memory to allocate session filter " - "tracking.\n"); return 0; } session->n_filters = service->n_filters; @@ -886,7 +793,7 @@ session_setup_filters(SESSION *session) session->filters[i].session = head->session; session->filters[i].instance = head->instance; session->head = *head; - free(head); + MXS_FREE(head); } for (i = 0; i < service->n_filters; i++) @@ -909,7 +816,7 @@ session_setup_filters(SESSION *session) if (tail != &session->tail) { session->tail = *tail; - free(tail); + MXS_FREE(tail); } } @@ -984,14 +891,6 @@ session_getUser(SESSION *session) { return (session && session->client_dcb) ? session->client_dcb->user : NULL; } -/** - * Return the pointer to the list of all sessions. - * @return Pointer to the list of all sessions. - */ -SESSION *get_all_sessions() -{ - return allSessions; -} /** * Enable the timing out of idle connections. @@ -1016,24 +915,22 @@ void process_idle_sessions() { if (hkheartbeat >= next_timeout_check) { + list_entry_t *current = list_start_iteration(&SESSIONlist); /** Because the resolution of the timeout is one second, we only need to * check for it once per second. One heartbeat is 100 milliseconds. */ next_timeout_check = hkheartbeat + 10; - spinlock_acquire(&session_spin); - SESSION *all_session = allSessions; - - while (all_session) + while (current) { - if (all_session->ses_is_in_use && - all_session->service && all_session->client_dcb && all_session->client_dcb->state == DCB_STATE_POLLING && + SESSION *all_session = (SESSION *)current; + + if (all_session->service && all_session->client_dcb && all_session->client_dcb->state == DCB_STATE_POLLING && hkheartbeat - all_session->client_dcb->last_read > all_session->service->conn_idle_timeout * 10) { dcb_close(all_session->client_dcb); } - all_session = all_session->next; + current = list_iterate(&SESSIONlist, current); } - spinlock_release(&session_spin); } spinlock_release(&timeout_lock); } @@ -1060,60 +957,72 @@ sessionRowCallback(RESULTSET *set, void *data) { SESSIONFILTER *cbdata = (SESSIONFILTER *)data; int i = 0; - char buf[20]; - RESULT_ROW *row; - SESSION *list_session; + list_entry_t *current = list_start_iteration(&SESSIONlist); - spinlock_acquire(&session_spin); - list_session = allSessions; /* Skip to the first non-listener if not showing listeners */ - while (false == list_session->ses_is_in_use || - (list_session && cbdata->filter == SESSION_LIST_CONNECTION && - list_session->state == SESSION_STATE_LISTENER)) + current = skip_maybe_to_next_non_listener(current, cbdata->filter); + + while (i < cbdata->index && current) { - list_session = list_session->next; - } - while (i < cbdata->index && list_session) - { - if (list_session->ses_is_in_use) + if (cbdata->filter == SESSION_LIST_ALL || + (cbdata->filter == SESSION_LIST_CONNECTION && + ((SESSION *)current)->state != SESSION_STATE_LISTENER)) { - if (cbdata->filter == SESSION_LIST_CONNECTION && - list_session->state != SESSION_STATE_LISTENER) - { - i++; - } - else if (cbdata->filter == SESSION_LIST_ALL) - { - i++; - } + i++; } - list_session = list_session->next; + current = list_iterate(&SESSIONlist, current); } + /* Skip to the next non-listener if not showing listeners */ - while (list_session && (false == list_session->ses_is_in_use || - (cbdata->filter == SESSION_LIST_CONNECTION && - list_session->state == SESSION_STATE_LISTENER))) + current = skip_maybe_to_next_non_listener(current, cbdata->filter); + + if (NULL == current) { - list_session = list_session->next; - } - if (list_session == NULL) - { - spinlock_release(&session_spin); - free(data); + MXS_FREE(data); return NULL; } - cbdata->index++; - row = resultset_make_row(set); - snprintf(buf,19, "%p", list_session); - buf[19] = '\0'; - resultset_row_set(row, 0, buf); - resultset_row_set(row, 1, ((list_session->client_dcb && list_session->client_dcb->remote) + else + { + char buf[20]; + RESULT_ROW *row; + SESSION *list_session = (SESSION *)current; + + cbdata->index++; + row = resultset_make_row(set); + snprintf(buf,19, "%p", list_session); + buf[19] = '\0'; + resultset_row_set(row, 0, buf); + resultset_row_set(row, 1, ((list_session->client_dcb && list_session->client_dcb->remote) ? list_session->client_dcb->remote : "")); - resultset_row_set(row, 2, (list_session->service && list_session->service->name + resultset_row_set(row, 2, (list_session->service && list_session->service->name ? list_session->service->name : "")); - resultset_row_set(row, 3, session_state(list_session->state)); - spinlock_release(&session_spin); - return row; + resultset_row_set(row, 3, session_state(list_session->state)); + list_terminate_iteration_early(&SESSIONlist, current); + return row; + } +} + +/* + * @brief Skip to the next non-listener session, if not showing listeners + * + * Based on a test of the filter that is the second parameter, along with the + * state of the sessions. + * + * @param current The session to start the possible skipping + * @param filter The filter the defines the operation + * + * @result The first session beyond those skipped, or the starting session; + * NULL if the list of sessions is exhausted. + */ +static list_entry_t *skip_maybe_to_next_non_listener(list_entry_t *current, SESSIONLISTFILTER filter) +{ + /* Skip to the first non-listener if not showing listeners */ + while (current && filter == SESSION_LIST_CONNECTION && + ((SESSION *)current)->state == SESSION_STATE_LISTENER) + { + current = list_iterate(&SESSIONlist, current); + } + return current; } /** @@ -1133,7 +1042,7 @@ sessionGetList(SESSIONLISTFILTER filter) RESULTSET *set; SESSIONFILTER *data; - if ((data = (SESSIONFILTER *)malloc(sizeof(SESSIONFILTER))) == NULL) + if ((data = (SESSIONFILTER *)MXS_MALLOC(sizeof(SESSIONFILTER))) == NULL) { return NULL; } @@ -1141,7 +1050,7 @@ sessionGetList(SESSIONLISTFILTER filter) data->filter = filter; if ((set = resultset_create(sessionRowCallback, data)) == NULL) { - free(data); + MXS_FREE(data); return NULL; } resultset_add_column(set, "Session", 16, COL_TYPE_VARCHAR); diff --git a/utils/skygw_utils.cc b/server/core/skygw_utils.cc similarity index 88% rename from utils/skygw_utils.cc rename to server/core/skygw_utils.cc index 8c5fc59a0..a8c13e9bd 100644 --- a/utils/skygw_utils.cc +++ b/server/core/skygw_utils.cc @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -31,7 +31,6 @@ #include #include -static bool file_write_header(skygw_file_t* file); static void simple_mutex_free_memory(simple_mutex_t* sm); static void thread_free_memory(skygw_thread_t* th, char* name); /** End of static function declarations */ @@ -831,155 +830,6 @@ return_mes_rc: ss_dassert(err == 0); } -static bool file_write_header(skygw_file_t* file) -{ - bool succp = false; - size_t wbytes1; - size_t wbytes2; - size_t wbytes3; - size_t wbytes4; - size_t len1; - size_t len2; - size_t len3; - size_t len4; - const char* header_buf1; - char* header_buf2 = NULL; - char* header_buf3 = NULL; - const char* header_buf4; - time_t* t; - struct tm* tm; -#if defined(LAPTOP_TEST) - struct timespec ts1; - ts1.tv_sec = 0; - ts1.tv_nsec = DISKWRITE_LATENCY * 1000000; -#endif - - t = (time_t *) malloc(sizeof (time_t)); - tm = (struct tm *) malloc(sizeof (struct tm)); - *t = time(NULL); - localtime_r(t, tm); - - CHK_FILE(file); - header_buf1 = "\n\nMariaDB Corporation MaxScale\t"; - header_buf2 = (char *) calloc(1, strlen(file->sf_fname) + 2); - snprintf(header_buf2, strlen(file->sf_fname) + 2, "%s ", file->sf_fname); - header_buf3 = strdup(asctime(tm)); - header_buf4 = "------------------------------------------------------" - "-----------------\n"; - - if (header_buf2 == NULL) - { - goto return_succp; - } - if (header_buf3 == NULL) - { - goto return_succp; - } - - len1 = strlen(header_buf1); - len2 = strlen(header_buf2); - len3 = strlen(header_buf3); - len4 = strlen(header_buf4); -#if defined(LAPTOP_TEST) - nanosleep(&ts1, NULL); -#else - wbytes1 = fwrite((void*) header_buf1, len1, 1, file->sf_file); - wbytes2 = fwrite((void*) header_buf2, len2, 1, file->sf_file); - wbytes3 = fwrite((void*) header_buf3, len3, 1, file->sf_file); - wbytes4 = fwrite((void*) header_buf4, len4, 1, file->sf_file); - - if (wbytes1 != 1 || wbytes2 != 1 || wbytes3 != 1 || wbytes4 != 1) - { - fprintf(stderr, "\nError : Writing header %s %s %s %s failed.\n", - header_buf1, header_buf2, header_buf3, header_buf4); - perror("Logfile header write"); - goto return_succp; - } -#endif - CHK_FILE(file); - - succp = true; -return_succp: - if (header_buf2 != NULL) - { - free(header_buf2); - } - if (header_buf3 != NULL) - { - free(header_buf3); - } - free(t); - free(tm); - return succp; -} - -static bool file_write_footer(skygw_file_t* file, bool shutdown) -{ - bool succp = false; - size_t wbytes1; - size_t wbytes3; - size_t wbytes4; - size_t len1; - size_t len4; - int tslen; - const char* header_buf1; - char* header_buf3 = NULL; - const char* header_buf4; -#if defined(LAPTOP_TEST) - struct timespec ts1; - ts1.tv_sec = 0; - ts1.tv_nsec = DISKWRITE_LATENCY * 1000000; -#endif - - CHK_FILE(file); - - if (shutdown) - { - header_buf1 = "MaxScale is shut down.\t"; - } - else - { - header_buf1 = "Closed file due log rotation.\t"; - } - tslen = get_timestamp_len(); - header_buf3 = (char *) malloc(tslen); - - if (header_buf3 == NULL) - { - goto return_succp; - } - tslen = snprint_timestamp(header_buf3, tslen); - header_buf4 = "\n--------------------------------------------" - "---------------------------\n"; - - len1 = strlen(header_buf1); - len4 = strlen(header_buf4); -#if defined(LAPTOP_TEST) - nanosleep(&ts1, NULL); -#else - wbytes3 = fwrite((void*) header_buf3, tslen, 1, file->sf_file); - wbytes1 = fwrite((void*) header_buf1, len1, 1, file->sf_file); - wbytes4 = fwrite((void*) header_buf4, len4, 1, file->sf_file); - - if (wbytes1 != 1 || wbytes3 != 1 || wbytes4 != 1) - { - fprintf(stderr, "\nError : Writing header %s %s to %s failed.\n", - header_buf1, header_buf3, header_buf4); - perror("Logfile header write"); - goto return_succp; - } -#endif - CHK_FILE(file); - - succp = true; -return_succp: - if (header_buf3 != NULL) - { - free(header_buf3); - } - return succp; -} - /** * Write data to a file. * @@ -1026,7 +876,7 @@ return_rc: return rc; } -skygw_file_t* skygw_file_alloc(char* fname) +skygw_file_t* skygw_file_alloc(const char* fname) { skygw_file_t* file; @@ -1043,7 +893,9 @@ skygw_file_t* skygw_file_alloc(char* fname) return file; } -skygw_file_t* skygw_file_init(char* fname, char* symlinkname) +skygw_file_t* skygw_file_init(const char* fname, + const char* symlinkname, + skygw_open_mode_t mode) { skygw_file_t* file; @@ -1053,7 +905,21 @@ skygw_file_t* skygw_file_init(char* fname, char* symlinkname) goto return_file; } - if ((file->sf_file = fopen(file->sf_fname, "a")) == NULL) + const char* mode_string; + + switch (mode) + { + case SKYGW_OPEN_TRUNCATE: + mode_string = "w"; + break; + + default: + ss_dassert(!true); + case SKYGW_OPEN_APPEND: + mode_string = "a"; + }; + + if ((file->sf_file = fopen(file->sf_fname, mode_string)) == NULL) { int eno = errno; errno = 0; @@ -1067,20 +933,7 @@ skygw_file_t* skygw_file_init(char* fname, char* symlinkname) setvbuf(file->sf_file, NULL, _IONBF, 0); - if (!file_write_header(file)) - { - int eno = errno; - errno = 0; - char errbuf[STRERROR_BUFLEN]; - fprintf(stderr, "\nError : Writing header of log file %s failed due %d, %s.\n", - file->sf_fname, eno, strerror_r(eno, errbuf, sizeof (errbuf))); - free(file); - file = NULL; - goto return_file; - } - CHK_FILE(file); - ss_dfprintf(stderr, "Opened %s\n", file->sf_fname); /** * Create symlink to newly created file if name was provided. @@ -1116,7 +969,7 @@ void skygw_file_free(skygw_file_t* file) } } -void skygw_file_close(skygw_file_t* file, bool shutdown) +void skygw_file_close(skygw_file_t* file) { int fd; int err; @@ -1125,11 +978,6 @@ void skygw_file_close(skygw_file_t* file, bool shutdown) { CHK_FILE(file); - if (!file_write_footer(file, shutdown)) - { - fprintf(stderr, "* Writing footer to log file %s failed.\n", file->sf_fname); - perror("Write fake footer\n"); - } fd = fileno(file->sf_file); fsync(fd); @@ -1507,29 +1355,6 @@ strip_escape_chars(char* val) return true; } -/** - * Calculate a hash value for a null-terminated string. - * @param key String to hash - * @return Hash value of the string - */ -int simple_str_hash(char* key) -{ - if (key == NULL) - { - return 0; - } - - int hash = 0, c = 0; - char* ptr = key; - - while ((c = *ptr++)) - { - hash = c + (hash << 6) + (hash << 16) - hash; - } - - return hash; -} - /** * Trim leading and trailing whitespace from a string * diff --git a/server/core/slist.c b/server/core/slist.c index 1ea731fe7..7f43b6421 100644 --- a/server/core/slist.c +++ b/server/core/slist.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -12,6 +12,7 @@ */ #include +#include #include static slist_cursor_t* slist_cursor_init(slist_t* list); @@ -31,7 +32,8 @@ static slist_t* slist_init_ex(bool create_cursors) { slist_t* list; - list = (slist_t*) calloc(1, sizeof (slist_t)); + list = (slist_t*) MXS_CALLOC(1, sizeof (slist_t)); + MXS_ABORT_IF_NULL(list); list->slist_chk_top = CHK_NUM_SLIST; list->slist_chk_tail = CHK_NUM_SLIST; @@ -47,7 +49,8 @@ static slist_node_t* slist_node_init(void* data, slist_cursor_t* cursor) { slist_node_t* node; - node = (slist_node_t*) calloc(1, sizeof (slist_node_t)); + node = (slist_node_t*) MXS_CALLOC(1, sizeof (slist_node_t)); + MXS_ABORT_IF_NULL(node); node->slnode_chk_top = CHK_NUM_SLIST_NODE; node->slnode_chk_tail = CHK_NUM_SLIST_NODE; node->slnode_data = data; @@ -128,7 +131,8 @@ static slist_cursor_t* slist_cursor_init(slist_t* list) CHK_SLIST(list); slist_cursor_t* c; - c = (slist_cursor_t *) calloc(1, sizeof (slist_cursor_t)); + c = (slist_cursor_t *) MXS_CALLOC(1, sizeof (slist_cursor_t)); + MXS_ABORT_IF_NULL(c); c->slcursor_chk_top = CHK_NUM_SLIST_CURSOR; c->slcursor_chk_tail = CHK_NUM_SLIST_CURSOR; c->slcursor_list = list; @@ -306,7 +310,7 @@ void slcursor_remove_data(slist_cursor_t* c) atomic_add((int*) &node->slnode_cursor_refcount, -1); if (node->slnode_cursor_refcount == 0) { - free(node); + MXS_FREE(node); } return; } @@ -320,7 +324,7 @@ void slcursor_remove_data(slist_cursor_t* c) atomic_add((int*) &node->slnode_cursor_refcount, -1); if (node->slnode_cursor_refcount == 0) { - free(node); + MXS_FREE(node); } return; } @@ -348,11 +352,11 @@ void slist_done(slist_cursor_t* c) while (succp) { data = slcursor_get_data(c); - free(data); + MXS_FREE(data); succp = slcursor_step_ahead(c); } - free(c->slcursor_list); - free(c); + MXS_FREE(c->slcursor_list); + MXS_FREE(c); } diff --git a/server/core/spinlock.c b/server/core/spinlock.c index 03c54934b..e2c59978a 100644 --- a/server/core/spinlock.c +++ b/server/core/spinlock.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -52,8 +52,9 @@ spinlock_init(SPINLOCK *lock) * @param lock The spinlock to acquire */ void -spinlock_acquire(SPINLOCK *lock) +spinlock_acquire(const SPINLOCK *const_lock) { + SPINLOCK *lock = (SPINLOCK*)const_lock; #if SPINLOCK_PROFILE int spins = 0; @@ -96,8 +97,9 @@ spinlock_acquire(SPINLOCK *lock) * @return True if the spinlock was acquired, otherwise false */ int -spinlock_acquire_nowait(SPINLOCK *lock) +spinlock_acquire_nowait(const SPINLOCK *const_lock) { + SPINLOCK *lock = (SPINLOCK*)const_lock; #ifdef __GNUC__ if (__sync_lock_test_and_set(&(lock->lock), 1)) { @@ -123,8 +125,9 @@ spinlock_acquire_nowait(SPINLOCK *lock) * @param lock The spinlock to release */ void -spinlock_release(SPINLOCK *lock) +spinlock_release(const SPINLOCK *const_lock) { + SPINLOCK *lock = (SPINLOCK*)const_lock; ss_dassert(lock->lock != 0); #if SPINLOCK_PROFILE if (lock->waiting > lock->max_waiting) @@ -154,7 +157,7 @@ spinlock_release(SPINLOCK *lock) * @param hdl A handle that is passed to the reporter function */ void -spinlock_stats(SPINLOCK *lock, void (*reporter)(void *, char *, int), void *hdl) +spinlock_stats(const SPINLOCK *lock, void (*reporter)(void *, char *, int), void *hdl) { #if SPINLOCK_PROFILE reporter(hdl, "Spinlock acquired", lock->acquired); diff --git a/server/core/statistics.c b/server/core/statistics.c index 26fe69ef0..94124b32a 100644 --- a/server/core/statistics.c +++ b/server/core/statistics.c @@ -4,50 +4,61 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General * Public License. */ +/** + * @file statistics.c - Functions to aid in the compilation of statistics + * + * @verbatim + * Revision History + * + * Date Who Description + * 15/06/16 Martin Brampton Removed some functions to statistics.h for inline + * + * @endverbatim + */ + #include +#include #include #include #include -thread_local int current_thread_id = 0; - static int thread_count = 0; -static bool initialized = false; +static bool stats_initialized = false; /** - * Initialize the statistics gathering + * @brief Initialize the statistics gathering */ void ts_stats_init() { - ss_dassert(!initialized); + ss_dassert(!stats_initialized); thread_count = config_threadcount(); - initialized = true; + stats_initialized = true; } /** - * End the statistics gathering + * @brief End the statistics gathering */ void ts_stats_end() { - ss_dassert(initialized); + ss_dassert(stats_initialized); } /** - * Create a new statistics object + * @brief Create a new statistics object * * @return New stats_t object or NULL if memory allocation failed */ ts_stats_t ts_stats_alloc() { - ss_dassert(initialized); - return calloc(thread_count, sizeof(int)); + ss_dassert(stats_initialized); + return MXS_CALLOC(thread_count, sizeof(int)); } /** @@ -57,56 +68,21 @@ ts_stats_t ts_stats_alloc() */ void ts_stats_free(ts_stats_t stats) { - ss_dassert(initialized); - free(stats); + ss_dassert(stats_initialized); + MXS_FREE(stats); } /** - * Set the current thread id + * @brief Read the total value of the statistics object * - * This should only be called only once by each thread. - * @param id Thread id - */ -void ts_stats_set_thread_id(int id) -{ - ss_dassert(initialized); - current_thread_id = id; -} - -/** - * Add @c value to @c stats - * - * @param stats Statistics to add to - * @param value Value to add - */ -void ts_stats_add(ts_stats_t stats, int value) -{ - ss_dassert(initialized); - ((int*)stats)[current_thread_id] += value; -} - -/** - * Assign a value to the statistics - * - * This sets the value for the current thread only. - * @param stats Statistics to set - * @param value Value to set to - */ -void ts_stats_set(ts_stats_t stats, int value) -{ - ss_dassert(initialized); - ((int*)stats)[current_thread_id] = value; -} - -/** - * Read the total value of the statistics object + * Add up the individual thread statistics to get the total for all threads. * * @param stats Statistics to read * @return Value of statistics */ int ts_stats_sum(ts_stats_t stats) { - ss_dassert(initialized); + ss_dassert(stats_initialized); int sum = 0; for (int i = 0; i < thread_count; i++) { diff --git a/server/core/test/CMakeLists.txt b/server/core/test/CMakeLists.txt index dbb72a769..c35a887c7 100644 --- a/server/core/test/CMakeLists.txt +++ b/server/core/test/CMakeLists.txt @@ -2,13 +2,16 @@ add_executable(test_adminusers testadminusers.c) add_executable(test_buffer testbuffer.c) add_executable(test_dcb testdcb.c) add_executable(test_filter testfilter.c) +add_executable(test_gwbitmask testgwbitmask.c) add_executable(test_hash testhash.c) add_executable(test_hint testhint.c) add_executable(test_log testlog.c) add_executable(test_logorder testlogorder.c) +add_executable(test_logthrottling testlogthrottling.cc) add_executable(test_modutil testmodutil.c) add_executable(test_mysql_users test_mysql_users.c) add_executable(test_poll testpoll.c) +add_executable(test_queuemanager testqueuemanager.c) add_executable(test_server testserver.c) add_executable(test_service testservice.c) add_executable(test_spinlock testspinlock.c) @@ -20,13 +23,16 @@ target_link_libraries(test_adminusers maxscale-common) target_link_libraries(test_buffer maxscale-common) target_link_libraries(test_dcb maxscale-common) target_link_libraries(test_filter maxscale-common) +target_link_libraries(test_gwbitmask maxscale-common) target_link_libraries(test_hash maxscale-common) target_link_libraries(test_hint maxscale-common) target_link_libraries(test_log maxscale-common) target_link_libraries(test_logorder maxscale-common) +target_link_libraries(test_logthrottling maxscale-common) target_link_libraries(test_modutil maxscale-common) target_link_libraries(test_mysql_users MySQLClient maxscale-common) target_link_libraries(test_poll maxscale-common) +target_link_libraries(test_queuemanager maxscale-common) target_link_libraries(test_server maxscale-common) target_link_libraries(test_service maxscale-common) target_link_libraries(test_spinlock maxscale-common) @@ -38,16 +44,19 @@ add_test(TestAdminUsers test_adminusers) add_test(TestBuffer test_buffer) add_test(TestDCB test_dcb) add_test(TestFilter test_filter) +add_test(TestBitmask test_gwbitmask) add_test(TestHash test_hash) add_test(TestHint test_hint) add_test(TestLog test_log) add_test(NAME TestLogOrder COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/logorder.sh 200 0 1000 ${CMAKE_CURRENT_BINARY_DIR}/logorder.log) +add_test(TestLogThrottling test_logthrottling) add_test(TestMaxScalePCRE2 testmaxscalepcre2) add_test(TestMemlog testmemlog) add_test(TestModutil test_modutil) add_test(TestMySQLUsers test_mysql_users) add_test(NAME TestMaxPasswd COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/testmaxpasswd.sh) add_test(TestPoll test_poll) +add_test(TestQueueManager test_queuemanager) add_test(TestServer test_server) add_test(TestService test_service) add_test(TestSpinlock test_spinlock) diff --git a/server/core/test/test_mysql_users.c b/server/core/test/test_mysql_users.c index ef87ab49a..18df9eedd 100644 --- a/server/core/test/test_mysql_users.c +++ b/server/core/test/test_mysql_users.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -39,6 +39,7 @@ #include #include #include +#include extern int setipaddress(); @@ -63,9 +64,8 @@ int set_and_get_single_mysql_users_ipv4(char *username, unsigned long ipv4, char fprintf(stderr, "dcb_alloc() failed\n"); return 1; } - if ((service = (SERVICE *)calloc(1, sizeof(SERVICE))) == NULL) + if ((service = (SERVICE *)MXS_CALLOC(1, sizeof(SERVICE))) == NULL) { - fprintf(stderr, "service_alloc() failed\n"); dcb_close(dcb); return 1; } @@ -99,7 +99,7 @@ int set_and_get_single_mysql_users_ipv4(char *username, unsigned long ipv4, char { fprintf(stderr, "Failed adding %s@%s(%lu)\n", username, ret_ip, fix_ipv4); users_free(mysql_users); - free(service); + MXS_FREE(service); dcb_close(dcb); return 1; } @@ -116,7 +116,7 @@ int set_and_get_single_mysql_users_ipv4(char *username, unsigned long ipv4, char fetch_data = mysql_users_fetch(mysql_users, &find_key); users_free(mysql_users); - free(service); + MXS_FREE(service); dcb_close(dcb); if (!fetch_data) @@ -207,21 +207,21 @@ int set_and_get_mysql_users_wildcards(char *username, char *hostname, char *pass DCB *dcb; SERVICE *service; MYSQL_session *data; - SERV_LISTENER dummy; - dcb = dcb_alloc(DCB_ROLE_INTERNAL, &dummy); + if ((service = (SERVICE *)MXS_CALLOC(1, sizeof(SERVICE))) == NULL) + { + return ret; + } + + SERV_LISTENER *port = listener_alloc(service, "testlistener", "MySQLClient", NULL, 4006, NULL, NULL); + + dcb = dcb_alloc(DCB_ROLE_INTERNAL, port); if (dcb == NULL) { fprintf(stderr, "dcb_alloc() failed\n"); return ret; } - if ((service = (SERVICE *)calloc(1, sizeof(SERVICE))) == NULL) - { - fprintf(stderr, "service_alloc() failed\n"); - dcb_close(dcb); - return ret; - } memset(&client_addr, 0, sizeof(client_addr)); @@ -230,16 +230,15 @@ int set_and_get_mysql_users_wildcards(char *username, char *hostname, char *pass if (!setipaddress(&client_addr.sin_addr, from)) { fprintf(stderr, "setipaddress failed for host [%s]\n", from); - free(service); + MXS_FREE(service); dcb_close(dcb); return ret; } } - if ((data = (MYSQL_session *) calloc(1, sizeof(MYSQL_session))) == NULL) + if ((data = (MYSQL_session *) MXS_CALLOC(1, sizeof(MYSQL_session))) == NULL) { - fprintf(stderr, "MYSQL_session alloc failed\n"); - free(service); + MXS_FREE(service); dcb_close(dcb); return ret; } @@ -252,15 +251,16 @@ int set_and_get_mysql_users_wildcards(char *username, char *hostname, char *pass mysql_users = mysql_users_alloc(); - service->users = mysql_users; + service->ports = port; + service->ports->users = mysql_users; if (db_from != NULL) { - strncpy(data->db, db_from, MYSQL_DATABASE_MAXLEN); + strcpy(data->db, db_from); } else { - strncpy(data->db, "", MYSQL_DATABASE_MAXLEN); + data->db[0] = 0; } /* freed by dcb_close(dcb) */ @@ -295,14 +295,14 @@ int set_and_get_mysql_users_wildcards(char *username, char *hostname, char *pass { unsigned char db_passwd[100] = ""; - dcb->remote = strdup(from); + dcb->remote = MXS_STRDUP_A(from); // returns 0 on success ret = gw_find_mysql_user_password_sha1(username, db_passwd, dcb); } users_free(mysql_users); - free(service); + MXS_FREE(service); dcb_close(dcb); return ret; diff --git a/server/core/test/testadminusers.c b/server/core/test/testadminusers.c index 2275b51ef..2cc75ed9e 100644 --- a/server/core/test/testadminusers.c +++ b/server/core/test/testadminusers.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -36,6 +36,7 @@ #include #include #include +#include /** @@ -246,7 +247,7 @@ main(int argc, char **argv) char *home, buf[1024]; /** Set datadir to /tmp */ - set_datadir(strdup("/tmp")); + set_datadir(MXS_STRDUP_A("/tmp")); /* Unlink any existing password file before running this test */ sprintf(buf, "%s/maxadmin-users", get_datadir()); diff --git a/server/core/test/testbuffer.c b/server/core/test/testbuffer.c index 8ec09d6d3..499a14886 100644 --- a/server/core/test/testbuffer.c +++ b/server/core/test/testbuffer.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -33,6 +33,7 @@ #include #include +#include #include #include @@ -44,7 +45,8 @@ */ uint8_t* generate_data(size_t count) { - uint8_t* data = malloc(count); + uint8_t* data = MXS_MALLOC(count); + MXS_ABORT_IF_NULL(data); srand(0); diff --git a/server/core/test/testdcb.c b/server/core/test/testdcb.c index 753a53f0b..c568380bc 100644 --- a/server/core/test/testdcb.c +++ b/server/core/test/testdcb.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/test/testfeedback.c b/server/core/test/testfeedback.c index 1ccaddc58..763332aa7 100644 --- a/server/core/test/testfeedback.c +++ b/server/core/test/testfeedback.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -39,6 +39,7 @@ #include #include #include +#include #include #include #include @@ -84,7 +85,8 @@ int main(int argc, char** argv) hkinit(); - cnf = malloc(sizeof(char) * (strlen(TEST_DIR) + strlen("/maxscale.cnf") + 1)); + cnf = MXS_MALLOC(sizeof(char) * (strlen(TEST_DIR) + strlen("/maxscale.cnf") + 1)); + MXS_ABORT_IF_NULL(cnf); sprintf(cnf, "%s/maxscale.cnf", TEST_DIR); printf("Config: %s\n", cnf); diff --git a/server/core/test/testfilter.c b/server/core/test/testfilter.c index 954e7b822..bc96c21f2 100644 --- a/server/core/test/testfilter.c +++ b/server/core/test/testfilter.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/test/testgwbitmask.c b/server/core/test/testgwbitmask.c index 9e0b1d6d0..7662c99e5 100644 --- a/server/core/test/testgwbitmask.c +++ b/server/core/test/testgwbitmask.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -66,6 +66,9 @@ test1() bitmask_clear(&bitmask, 17); ss_info_dassert(0 == bitmask_isset(&bitmask, 17), "Test bit should be clear"); ss_info_dassert(0 != bitmask_isallclear(&bitmask), "Should be all clear"); + // Testing the allocation mechanism, for use with valgrind. + bitmask_set(&bitmask, BIT_LENGTH_INC + 1); + bitmask_set(&bitmask, 2 * BIT_LENGTH_INC + 1); ss_dfprintf(stderr, "\t..done\nFree the bitmask."); bitmask_free(&bitmask); ss_info_dassert(0 == bitmask.length, "Length should be zero after bit mask freed."); diff --git a/server/core/test/testhash.c b/server/core/test/testhash.c index b36873be7..d7de067a5 100644 --- a/server/core/test/testhash.c +++ b/server/core/test/testhash.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -35,6 +35,8 @@ #include #include +#include +#include #include static void @@ -58,27 +60,24 @@ read_unlock(HASHTABLE *table) atomic_add(&table->n_readers, -1); } -static int hfun(void* key); -static int cmpfun (void *, void *); +static int hfun(const void* key); +static int cmpfun(const void *, const void *); -static int hfun( - void* key) +static int hfun(const void* key) { - int *i = (int *)key; + const int *i = (const int *)key; int j = (*i * 23) + 41; return j; /* return *(int *)key; */ } -static int cmpfun( - void* v1, - void* v2) +static int cmpfun(const void* v1, const void* v2) { int i1; int i2; - i1 = *(int *)v1; - i2 = *(int *)v2; + i1 = *(const int *)v1; + i2 = *(const int *)v2; return (i1 < i2 ? -1 : (i1 > i2 ? 1 : 0)); } @@ -116,7 +115,8 @@ static bool do_hashtest( argelems, (double)clock() - start); - val_arr = (int *)malloc(sizeof(void *)*argelems); + val_arr = (int *)MXS_MALLOC(sizeof(void *)*argelems); + MXS_ABORT_IF_NULL(val_arr); h = hashtable_alloc(argsize, hfun, cmpfun); @@ -176,7 +176,7 @@ static bool do_hashtest( hashtable_free(h); - free(val_arr); + MXS_FREE(val_arr); return succp; } diff --git a/server/core/test/testhint.c b/server/core/test/testhint.c index 6de9ccb6e..0174ddd92 100644 --- a/server/core/test/testhint.c +++ b/server/core/test/testhint.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -34,6 +34,7 @@ #include #include +#include /** * test1 Allocate table of users and mess around with it @@ -48,9 +49,9 @@ test1() /* Hint tests */ ss_dfprintf(stderr, "testhint : Add a parameter hint to a null list"); - char* name = strdup("name"); + char* name = MXS_STRDUP_A("name"); hint = hint_create_parameter(NULL, name, "value"); - free(name); + MXS_FREE(name); mxs_log_flush_sync(); ss_info_dassert(NULL != hint, "New hint list should not be null"); ss_info_dassert(0 == strcmp("value", hint->value), "Hint value should be correct"); diff --git a/server/core/test/testlog.c b/server/core/test/testlog.c index 9c3ec91dd..5db84cc8a 100644 --- a/server/core/test/testlog.c +++ b/server/core/test/testlog.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -110,12 +111,10 @@ int main(int argc, char* argv[]) printf("Using %d threads.\n", nthr); - thr = (thread_t **)calloc(1, nthr * sizeof(thread_t*)); + thr = (thread_t **)MXS_CALLOC(1, nthr * sizeof(thread_t*)); if (thr == NULL) { - fprintf(stderr, "Failed to allocate memory for thread " - "structure. Exiting.\n"); err = 1; goto return_err; } @@ -226,7 +225,8 @@ int main(int argc, char* argv[]) /** 1 */ for (i = 0; i < nthr; i++) { - thr[i] = (thread_t*)calloc(1, sizeof(thread_t)); + thr[i] = (thread_t*)MXS_CALLOC(1, sizeof(thread_t)); + MXS_ABORT_IF_NULL(thr[i]); thr[i]->mes = mes; thr[i]->mtx = mtx; thr[i]->nactive = &nactive; @@ -264,7 +264,7 @@ int main(int argc, char* argv[]) for (i = 0; i < nthr; i++) { - free(thr[i]); + MXS_FREE(thr[i]); } #endif @@ -275,7 +275,8 @@ int main(int argc, char* argv[]) /** 2 */ for (i = 0; i < nthr; i++) { - thr[i] = (thread_t*)calloc(1, sizeof(thread_t)); + thr[i] = (thread_t*)MXS_CALLOC(1, sizeof(thread_t)); + MXS_ABORT_IF_NULL(thr[i]); thr[i]->mes = mes; thr[i]->mtx = mtx; thr[i]->nactive = &nactive; @@ -324,7 +325,7 @@ int main(int argc, char* argv[]) for (i = 0; i < nthr; i++) { - free(thr[i]); + MXS_FREE(thr[i]); } /** Test ended here */ @@ -499,7 +500,7 @@ int main(int argc, char* argv[]) return_err: if (thr != NULL) { - free(thr); + MXS_FREE(thr); } return err; } diff --git a/server/core/test/testlogorder.c b/server/core/test/testlogorder.c index 3431fde69..d8116a6ab 100644 --- a/server/core/test/testlogorder.c +++ b/server/core/test/testlogorder.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -15,6 +15,7 @@ #include #include #include +#include #include #include @@ -59,7 +60,7 @@ int main(int argc, char** argv) } if (getcwd(cwd, sizeof(cwd)) == NULL || - (message = (char*)malloc(sizeof(char) * block_size)) == NULL) + (message = (char*)MXS_MALLOC(sizeof(char) * block_size)) == NULL) { fprintf(stderr, "Fatal Error, exiting..."); return 1; @@ -114,6 +115,6 @@ int main(int argc, char** argv) mxs_log_flush(); mxs_log_finish(); - free(message); + MXS_FREE(message); return 0; } diff --git a/server/core/test/testlogthrottling.cc b/server/core/test/testlogthrottling.cc new file mode 100644 index 000000000..45c0982d4 --- /dev/null +++ b/server/core/test/testlogthrottling.cc @@ -0,0 +1,262 @@ +/* + * Copyright (c) 2016 MariaDB Corporation Ab + * + * Use of this software is governed by the Business Source License included + * in the LICENSE.TXT file and at www.mariadb.com/bsl. + * + * Change Date: 2019-07-01 + * + * On the date above, in accordance with the Business Source License, use + * of this software will be governed by version 2 or later of the General + * Public License. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using std::cerr; +using std::cout; +using std::endl; +using std::ios_base; +using std::istream; +using std::ifstream; +using std::ostream; +using std::string; + +namespace +{ + +const char LOGNAME[] = "/tmp/maxscale.log"; +const size_t N_THREADS = 4; + +sem_t u_semstart; +sem_t u_semfinish; + +void ensure(bool ok) +{ + if (!ok) + { + perror("test_logthrottling"); + exit(EXIT_FAILURE); + } +} + +ostream& operator << (ostream& out, const MXS_LOG_THROTTLING& t) +{ + out << "{" << t.count << ", " << t.window_ms << ", " << t.suppress_ms << "}"; + return out; +} + +} + +bool check_messages(istream& in, size_t n_expected) +{ + string line; + + size_t count = 0; + + while (std::getline(in, line)) + { + ++count; + } + + cout << "Status: expected " << n_expected << " messages, found " << count << "." << endl; + + return count == n_expected; +} + +void log_messages(uint32_t id, size_t n_generate, int priority) +{ + for (size_t i = 0; i < n_generate; ++i) + { + MXS_LOG_MESSAGE(priority, "[%u] Message %lu.", id, i); + + pthread_yield(); + } +} + +struct THREAD_ARG +{ + uint32_t id; + size_t n_generate; + int priority; +}; + +void* thread_main(void* pv) +{ + THREAD_ARG *parg = static_cast(pv); + + sem_wait(&u_semstart); + + log_messages(parg->id, parg->n_generate, parg->priority); + + sem_post(&u_semfinish); + return 0; +} + +bool run(const MXS_LOG_THROTTLING& throttling, int priority, size_t n_generate, size_t n_expect) +{ + cout << "Logging " << n_generate << " messages with throttling as " << throttling << "," << endl; + + mxs_log_set_throttling(&throttling); // Causes message to be logged. + mxs_log_flush_sync(); + + ifstream in(LOGNAME); + in.seekg(0, ios_base::end); + + THREAD_ARG args[N_THREADS]; + + // Create the threads. + for (size_t i = 0; i < N_THREADS; ++i) + { + THREAD_ARG* parg = &args[i]; + parg->id = i; + parg->n_generate = n_generate; + parg->priority = priority; + + pthread_t tid; + int rc = pthread_create(&tid, 0, thread_main, parg); + ensure(rc == 0); + } + + sleep(1); + + // Let them loose. + for (size_t i = 0; i < N_THREADS; ++i) + { + int rc = sem_post(&u_semstart); + ensure(rc == 0); + } + + // Wait for the results. + for (size_t i = 0; i < N_THREADS; ++i) + { + int rc = sem_wait(&u_semfinish); + ensure(rc == 0); + } + + mxs_log_flush_sync(); + + return check_messages(in, n_expect); +} + +int main(int argc, char* argv[]) +{ + int rc; + + std::ios::sync_with_stdio(); + + rc = sem_init(&u_semstart, 0, 0); + ensure(rc == 0); + + rc = sem_init(&u_semfinish, 0, 0); + ensure(rc == 0); + + unlink(LOGNAME); + + if (mxs_log_init(NULL, "/tmp", MXS_LOG_TARGET_FS)) + { + MXS_LOG_THROTTLING t; + + t.count = 0; + t.window_ms = 0; + t.suppress_ms = 0; + + // No throttling, so we should get messages from all threads. + if (!run(t, LOG_ERR, 100, N_THREADS * 100)) + { + rc = EXIT_FAILURE; + } + + t.count = 10; + t.window_ms = 2000; + t.suppress_ms = 5000; + + // 100 messages * N_THREADS, but due to the throttling we should get only 10 messages. + if (!run(t, LOG_ERR, 100, 10)) + { + rc = EXIT_FAILURE; + } + + cout << "Sleeping 7 seconds." << endl; + sleep(7); + + // 100 messages * N_THREADS, but due to the throttling we should get only 10 messages. + // Since we slept longer than the suppression window, the previous message batch should + // not affect. + if (!run(t, LOG_ERR, 100, 10)) + { + rc = EXIT_FAILURE; + } + + cout << "Sleeping 1 seconds." << endl; + sleep(1); + + // 100 messages * N_THREADS, but since we should still be within the suppression + // window, we should get no messages. + if (!run(t, LOG_WARNING, 100, 0)) + { + rc = EXIT_FAILURE; + } + + cout << "Sleeping 6 seconds." << endl; + sleep(6); + + t.count = 20; + t.window_ms = 1000; + t.suppress_ms = 5000; + + // 100 messages * N_THREADS, and since we slept longer than the suppression window, + // we should get 20 messages. + if (!run(t, LOG_ERR, 100, 20)) + { + rc = EXIT_FAILURE; + } + + t.count = 10; + t.window_ms = 1000; + t.suppress_ms = 5000; + + // 20 messages * N_THREADS, and since we are logging NOTICE messages, we should + // get 20 * N_THREADS messages. + if (!run(t, LOG_NOTICE, 20, 20 * N_THREADS)) + { + rc = EXIT_FAILURE; + } + + mxs_log_set_priority_enabled(LOG_INFO, true); + + // 20 messages * N_THREADS, and since we are logging INFO messages, we should + // get 20 * N_THREADS messages. + if (!run(t, LOG_INFO, 20, 20 * N_THREADS + 1)) // There will be 1 info message about log flushing. + { + rc = EXIT_FAILURE; + } + + mxs_log_set_priority_enabled(LOG_INFO, false); + + mxs_log_set_priority_enabled(LOG_DEBUG, true); + + // 20 messages * N_THREADS, and since we are logging DEBUG messages, we should + // get 20 * N_THREADS messages. + if (!run(t, LOG_DEBUG, 20, 20 * N_THREADS)) + { + rc = EXIT_FAILURE; + } + + mxs_log_finish(); + } + else + { + rc = EXIT_FAILURE; + } + + return rc; +} diff --git a/server/core/test/testmaxscalepcre2.c b/server/core/test/testmaxscalepcre2.c index 010b7d78c..289de88d2 100644 --- a/server/core/test/testmaxscalepcre2.c +++ b/server/core/test/testmaxscalepcre2.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -33,6 +33,7 @@ #include #include #include +#include #include #include @@ -81,7 +82,8 @@ static int test2() pcre2_code *re2 = pcre2_compile((PCRE2_SPTR) pattern2, PCRE2_ZERO_TERMINATED, 0, &err, &erroff, NULL); size_t size = 1000; - char* dest = malloc(size); + char* dest = MXS_MALLOC(size); + MXS_ABORT_IF_NULL(dest); mxs_pcre2_result_t result = mxs_pcre2_substitute(re, subject, good_replace, &dest, &size); test_assert(result == MXS_PCRE2_MATCH, "Substitution should substitute"); diff --git a/server/core/test/testmemlog.c b/server/core/test/testmemlog.c index b1f2b30ce..21060661a 100644 --- a/server/core/test/testmemlog.c +++ b/server/core/test/testmemlog.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/test/testmodutil.c b/server/core/test/testmodutil.c index dcb6f745b..dd46665b0 100644 --- a/server/core/test/testmodutil.c +++ b/server/core/test/testmodutil.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -33,6 +33,7 @@ #include #include +#include #include #include @@ -90,7 +91,7 @@ test2() char* result = modutil_get_SQL(buffer); ss_dassert(strcmp(result, query) == 0); gwbuf_free(buffer); - free(result); + MXS_FREE(result); ss_dfprintf(stderr, "\t..done\n"); return 0; diff --git a/server/core/test/testpoll.c b/server/core/test/testpoll.c index 06946165d..0dce61a51 100644 --- a/server/core/test/testpoll.c +++ b/server/core/test/testpoll.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/test/testqueuemanager.c b/server/core/test/testqueuemanager.c new file mode 100644 index 000000000..7ae21dfb1 --- /dev/null +++ b/server/core/test/testqueuemanager.c @@ -0,0 +1,251 @@ +/* + * Copyright (c) 2016 MariaDB Corporation Ab + * + * Use of this software is governed by the Business Source License included + * in the LICENSE.TXT file and at www.mariadb.com/bsl. + * + * Change Date: 2019-07-01 + * + * On the date above, in accordance with the Business Source License, use + * of this software will be governed by version 2 or later of the General + * Public License. + */ + +/** + * + * @verbatim + * Revision History + * + * Date Who Description + * 21/06/2016 Martin Brampton Initial implementation + * + * @endverbatim + */ + +// To ensure that ss_info_assert asserts also when builing in non-debug mode. +#if !defined(SS_DEBUG) +#define SS_DEBUG +#endif +#if defined(NDEBUG) +#undef NDEBUG +#endif +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/** + * test1 Allocate a queue and do lots of other things + * + */ + +extern int debug_check_fail; + +#define TEST_QUEUE_SIZE 5 +#define HEARTBEATS_TO_EXPIRE 3 +#define NUMBER_OF_THREADS 4 +#define THREAD_TEST_COUNT 1000000 + +static QUEUE_CONFIG *thread_queue; + +static int +test1() +{ + QUEUE_CONFIG *queue; + int filled = 0; + int emptied = 0; + int expired = 0; + int input_counter = 0; + int output_counter = 0; + + hkheartbeat = 0; + + queue = mxs_queue_alloc(TEST_QUEUE_SIZE, HEARTBEATS_TO_EXPIRE); + + { + QUEUE_ENTRY entry; + if (mxs_dequeue(queue, &entry)) + { + ss_dfprintf(stderr, "\nError mxs_dequeue on empty queue did not return false.\n"); + return 1; + } + + if (mxs_dequeue_if_expired(queue, &entry)) + { + ss_dfprintf(stderr, "\nError mxs_dequeue_if_expired on empty queue did not return false.\n"); + return 1; + } + } + + while (filled < 250 || emptied < 250 || expired < 250) + { + ss_dfprintf(stderr, "Input counter %d and output counter %d\n", input_counter, output_counter); + ss_dfprintf(stderr, "Difference between counters %d\n", input_counter - output_counter); + ss_dfprintf(stderr, "Filled %d, emptied %d, expired %d\n", filled, emptied, expired); + if (random_jkiss() % 2) + { + int *entrynumber = MXS_MALLOC(sizeof(int)); + *entrynumber = input_counter; + if (mxs_enqueue(queue, entrynumber)) + { + input_counter++; + if ((input_counter - output_counter) > TEST_QUEUE_SIZE) + { + ss_dfprintf(stderr, "\nQueue full, but mxs_enqueue accepted entry.\n"); + return 3; + } + } + else + { + QUEUE_ENTRY entry; + + if ((input_counter - output_counter) != TEST_QUEUE_SIZE) + { + ss_dfprintf(stderr, "\nFailed enqueue, but input counter %d and output counter %d do not differ by %d.\n", + input_counter, + output_counter, + TEST_QUEUE_SIZE); + return 4; + } + filled++; + if (0 == (random_jkiss() % 5)) + { + if ((mxs_dequeue_if_expired(queue, &entry))) + { + if ((entry.heartbeat) > (hkheartbeat - HEARTBEATS_TO_EXPIRE)) + { + ss_dfprintf(stderr, "\nReturned an expired entry even though none or not expired.\n"); + return 5; + } + if (*(int *)entry.queued_object != output_counter) + { + ss_dfprintf(stderr, "\nOutput counter was %d, but dequeue gave %d.\n", + output_counter, + *(int *)entry.queued_object); + return 10; + } + output_counter++; + } + else + { + hkheartbeat += (HEARTBEATS_TO_EXPIRE + 1); + if (mxs_dequeue_if_expired(queue, &entry)) + { + if (*(int *)entry.queued_object != output_counter) + { + ss_dfprintf(stderr, "\nOutput counter was %d, but dequeue gave %d.\n", + output_counter, + *(int *)entry.queued_object); + return 6; + } + output_counter++; + } + else + { + ss_dfprintf(stderr, "\nReturned no expired entry even though all are expired.\n"); + return 7; + } + expired++; + } + } + } + } + else + { + QUEUE_ENTRY entry; + if (mxs_dequeue(queue, &entry)) + { + if (*(int *)entry.queued_object != output_counter) + { + ss_dfprintf(stderr, "\nOutput counter was %d, but dequeue gave %d.\n", + output_counter, + *(int *)entry.queued_object); + return 8; + } + output_counter++; + } + else + { + if (input_counter != output_counter) + { + ss_dfprintf(stderr, "\nNULL from dequeue, but input counter %d and output counter %d.\n", + input_counter, + output_counter); + return 9; + } + emptied++; + } + } + } + + ss_dfprintf(stderr, "Successfully ended test\n"); + + return 0; +} + +static void * +thread_test(void *arg) +{ + int i; + QUEUE_ENTRY entry; + int emptied = 0; + int filled = 0; + + for (i = 0; i < THREAD_TEST_COUNT; i++) + { + if (random_jkiss() % 2) + { + if (!mxs_enqueue(thread_queue, (void *)"Just for test")) + { + filled++; + } + } + else + { + if (!mxs_dequeue(thread_queue, &entry)) + { + emptied++; + } + } + } + ss_dfprintf(stderr, "Queue was full %d times, empty %d times\n", filled, emptied); + + return NULL; +} + +static int +test2() +{ + pthread_t tid[NUMBER_OF_THREADS - 1]; + int err, i, limit; + + thread_queue = mxs_queue_alloc(TEST_QUEUE_SIZE, HEARTBEATS_TO_EXPIRE); + limit = NUMBER_OF_THREADS; + for (i = 0; i < limit; i++) + { + err = pthread_create(&tid[i], NULL, thread_test, NULL); + ss_info_dassert((0 == err), "Must create threads successfully"); + } + for (i = 0; i < limit; i++) + { + err = pthread_join(tid[i], NULL); + ss_info_dassert((0 == err), "Must join threads successfully"); + ss_dfprintf(stderr, "\nThread %d ended with debug check fail at %d.\n", i, debug_check_fail); + } + return debug_check_fail ? 1 : 0; +} + +int main(int argc, char **argv) +{ + int result = 0; + + result += (test1() ? 1 : 0); + result += (test2() ? 1 : 0); + + exit(result); +} diff --git a/server/core/test/testserver.c b/server/core/test/testserver.c index bb4a515e1..a959765f4 100644 --- a/server/core/test/testserver.c +++ b/server/core/test/testserver.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -33,6 +33,7 @@ #include #include +#include #include #include /** @@ -73,21 +74,21 @@ test1() ss_info_dassert(0 == strcmp("Running", status), "Status of Server should be Running by default."); if (NULL != status) { - free(status); + MXS_FREE(status); } server_set_status(server, SERVER_MASTER); status = server_status(server); mxs_log_flush_sync(); ss_info_dassert(0 == strcmp("Master, Running", status), "Should find correct status."); server_clear_status(server, SERVER_MASTER); - free(status); + MXS_FREE(status); status = server_status(server); mxs_log_flush_sync(); ss_info_dassert(0 == strcmp("Running", status), "Status of Server should be Running after master status cleared."); if (NULL != status) { - free(status); + MXS_FREE(status); } ss_dfprintf(stderr, "\t..done\nRun Prints for Server and all Servers."); printServer(server); diff --git a/server/core/test/testservice.c b/server/core/test/testservice.c index 295ea86ca..e7dc7d765 100644 --- a/server/core/test/testservice.c +++ b/server/core/test/testservice.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -36,6 +36,7 @@ #include #include #include +#include /** * test1 Allocate a service and do lots of other things @@ -60,18 +61,19 @@ test1() ss_info_dassert(NULL == service, "New service with invalid router should be null"); ss_info_dassert(0 == service_isvalid(service), "Service must not be valid after incorrect creation"); ss_dfprintf(stderr, "\t..done\nValid service creation, router testroute."); - set_libdir(strdup("../../modules/routing/")); + set_libdir(MXS_STRDUP_A("../../modules/routing/")); service = service_alloc("MyService", "testroute"); mxs_log_flush_sync(); ss_info_dassert(NULL != service, "New service with valid router must not be null"); ss_info_dassert(0 != service_isvalid(service), "Service must be valid after creation"); ss_info_dassert(0 == strcmp("MyService", service_get_name(service)), "Service must have given name"); ss_dfprintf(stderr, "\t..done\nAdding protocol testprotocol."); - ss_info_dassert(0 != serviceAddProtocol(service, "testprotocol", "localhost", 9876, "MySQL", NULL), + ss_info_dassert(0 != serviceAddProtocol(service, "TestProtocol", "testprotocol", + "localhost", 9876, "MySQL", NULL), "Add Protocol should succeed"); ss_info_dassert(0 != serviceHasProtocol(service, "testprotocol", "localhost", 9876), "Service should have new protocol as requested"); - set_libdir(strdup("../../modules/protocol/")); + set_libdir(MXS_STRDUP_A("../../modules/protocol/")); serviceStartProtocol(service, "testprotocol", 9876); mxs_log_flush_sync(); ss_dfprintf(stderr, "\t..done\nStarting Service."); diff --git a/server/core/test/testsession.c b/server/core/test/testsession.c index e4d5c9ab0..94725d2b2 100644 --- a/server/core/test/testsession.c +++ b/server/core/test/testsession.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/test/testspinlock.c b/server/core/test/testspinlock.c index ba3ce8ec5..0992783c6 100644 --- a/server/core/test/testspinlock.c +++ b/server/core/test/testspinlock.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/test/testusers.c b/server/core/test/testusers.c index 671e5a16e..a37ee2579 100644 --- a/server/core/test/testusers.c +++ b/server/core/test/testusers.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/thread.c b/server/core/thread.c index a77e03187..415faac3b 100644 --- a/server/core/thread.c +++ b/server/core/thread.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/core/users.c b/server/core/users.c index 83422441c..f81acbf16 100644 --- a/server/core/users.c +++ b/server/core/users.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -14,6 +14,7 @@ #include #include #include +#include #include #include @@ -42,24 +43,22 @@ users_alloc() { USERS *rval; - if ((rval = calloc(1, sizeof(USERS))) == NULL) + if ((rval = MXS_CALLOC(1, sizeof(USERS))) == NULL) { - MXS_ERROR("[%s:%d]: Memory allocation failed.", __FUNCTION__, __LINE__); return NULL; } - if ((rval->data = hashtable_alloc(USERS_HASHTABLE_DEFAULT_SIZE, simple_str_hash, strcmp)) == NULL) + if ((rval->data = hashtable_alloc(USERS_HASHTABLE_DEFAULT_SIZE, + hashtable_item_strhash, hashtable_item_strcmp)) == NULL) { MXS_ERROR("[%s:%d]: Memory allocation failed.", __FUNCTION__, __LINE__); - free(rval); + MXS_FREE(rval); return NULL; } hashtable_memory_fns(rval->data, - (HASHMEMORYFN)strdup, - (HASHMEMORYFN)strdup, - (HASHMEMORYFN)free, - (HASHMEMORYFN)free); + hashtable_item_strdup, hashtable_item_strdup, + hashtable_item_free, hashtable_item_free); return rval; } @@ -82,7 +81,7 @@ users_free(USERS *users) { hashtable_free(users->data); } - free(users); + MXS_FREE(users); } /** @@ -199,7 +198,7 @@ dcb_usersPrint(DCB *dcb, USERS *users) if (custom_user) { dcb_printf(dcb, "%s%s", sep, custom_user); - free(custom_user); + MXS_FREE(custom_user); sep = ", "; } } diff --git a/server/core/utils.c b/server/core/utils.c index e568e02e9..c01dcde89 100644 --- a/server/core/utils.c +++ b/server/core/utils.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include @@ -256,7 +257,7 @@ char *create_hex_sha1_sha1_passwd(char *passwd) uint8_t hash2[SHA_DIGEST_LENGTH] = ""; char *hexpasswd = NULL; - if ((hexpasswd = (char *)calloc(SHA_DIGEST_LENGTH * 2 + 1, 1)) == NULL) + if ((hexpasswd = (char *)MXS_CALLOC(SHA_DIGEST_LENGTH * 2 + 1, 1)) == NULL) { return NULL; } @@ -313,3 +314,77 @@ void clean_up_pathname(char *path) } } } + +/** + * @brief Internal helper function for mkdir_all() + * + * @param path Path to create + * @param mask Bitmask to use + * @return True if directory exists or it was successfully created, false on error + */ +static bool mkdir_all_internal(char *path, mode_t mask) +{ + bool rval = false; + + if (mkdir(path, mask) == -1 && errno != EEXIST) + { + if (errno == ENOENT) + { + /** Try to create the parent directory */ + char *ndir = strrchr(path, '/'); + if (ndir) + { + *ndir = '\0'; + if (mkdir_all_internal(path, mask)) + { + /** Creation of the parent directory was successful, try to + * create the directory again */ + *ndir = '/'; + if (mkdir(path, mask) == 0) + { + rval = true; + } + else + { + char err[STRERROR_BUFLEN]; + MXS_ERROR("Failed to create directory '%s': %d, %s", + path, errno, strerror_r(errno, err, sizeof(err))); + } + } + } + } + else + { + char err[STRERROR_BUFLEN]; + MXS_ERROR("Failed to create directory '%s': %d, %s", + path, errno, strerror_r(errno, err, sizeof(err))); + } + } + else + { + rval = true; + } + + return rval; +} + +/** + * @brief Create a directory and any parent directories that do not exist + * + * + * @param path Path to create + * @param mask Bitmask to use + * @return True if directory exists or it was successfully created, false on error + */ +bool mxs_mkdir_all(const char *path, int mask) +{ + char local_path[strlen(path) + 1]; + strcpy(local_path, path); + + if (local_path[sizeof(local_path) - 2] == '/') + { + local_path[sizeof(local_path) - 2] = '\0'; + } + + return mkdir_all_internal(local_path, (mode_t)mask); +} diff --git a/server/include/CMakeLists.txt b/server/include/CMakeLists.txt new file mode 100644 index 000000000..468e419af --- /dev/null +++ b/server/include/CMakeLists.txt @@ -0,0 +1,7 @@ +file(GLOB HEADERS "*.h") +foreach(var ${HEADERS}) + get_filename_component(header ${var} NAME) + install_header(${header} devel) +endforeach() + +add_subdirectory(maxscale) diff --git a/server/include/adminusers.h.in b/server/include/adminusers.h.in index ea0268950..d00caf663 100644 --- a/server/include/adminusers.h.in +++ b/server/include/adminusers.h.in @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/atomic.h b/server/include/atomic.h index acc177aa9..bcd76f9a4 100644 --- a/server/include/atomic.h +++ b/server/include/atomic.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/buffer.h b/server/include/buffer.h index 42d0aa016..2235a6fbc 100644 --- a/server/include/buffer.h +++ b/server/include/buffer.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/dbusers.h b/server/include/dbusers.h index 4a5a9a0f9..7062c20af 100644 --- a/server/include/dbusers.h +++ b/server/include/dbusers.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -49,6 +49,10 @@ #define MYSQL_DATABASE_MAXLEN 128 #define MYSQL_TABLE_MAXLEN 64 +/** Cache directory and file names */ +static const char DBUSERS_DIR[] = "cache"; +static const char DBUSERS_FILE[] = "dbusers"; + /** * MySQL user and host data structure */ @@ -66,11 +70,11 @@ extern int add_mysql_users_with_host_ipv4(USERS *users, const char *user, const extern bool check_service_permissions(SERVICE* service); extern int dbusers_load(USERS *, const char *filename); extern int dbusers_save(USERS *, const char *filename); -extern int load_mysql_users(SERVICE *service); +extern int load_mysql_users(SERV_LISTENER *listener); extern int mysql_users_add(USERS *users, MYSQL_USER_HOST *key, char *auth); extern USERS *mysql_users_alloc(); extern char *mysql_users_fetch(USERS *users, MYSQL_USER_HOST *key); -extern int reload_mysql_users(SERVICE *service); -extern int replace_mysql_users(SERVICE *service); +extern int reload_mysql_users(SERV_LISTENER *listener); +extern int replace_mysql_users(SERV_LISTENER *listener); #endif diff --git a/server/include/dcb.h b/server/include/dcb.h index 321e86d04..5e41baacf 100644 --- a/server/include/dcb.h +++ b/server/include/dcb.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -14,6 +14,7 @@ */ #include #include +#include #include #include #include @@ -57,6 +58,7 @@ struct servlistener; * 19/06/2015 Martin Brampton Provision of persistent connections * 20/01/2016 Martin Brampton Moved GWPROTOCOL to gw_protocol.h * 01/02/2016 Martin Brampton Added fields for SSL and authentication + * 27/06/2016 Martin Brampton Changed DCB to conform to list manager * * @endverbatim */ @@ -88,6 +90,8 @@ typedef struct unsigned long started; } DCBEVENTQ; +#define DCBEVENTQ_INIT {NULL, NULL, 0, 0, 0, SPINLOCK_INIT, 0, 0} + #define DCBFD_CLOSED -1 /** @@ -103,6 +107,8 @@ typedef struct dcbstats int n_low_water; /*< Number of crosses of low water mark */ } DCBSTATS; +#define DCBSTATS_INIT {0} + /** * The data structure that is embedded witin a DCB and manages the complex memory * management issues of a DCB. @@ -127,12 +133,15 @@ typedef struct struct dcb *next; /*< Next pointer for the zombie list */ } DCBMM; +#define DCBMM_INIT {GWBITMASK_INIT} + /* DCB states */ typedef enum { DCB_STATE_UNDEFINED, /*< State variable with no state */ DCB_STATE_ALLOC, /*< Memory allocated but not populated */ DCB_STATE_POLLING, /*< Waiting in the poll loop */ + DCB_STATE_WAITING, /*< Client wanting a connection */ DCB_STATE_LISTENING, /*< The DCB is for a listening socket */ DCB_STATE_DISCONNECTED, /*< The socket is now closed */ DCB_STATE_NOPOLLING, /*< Removed from poll mask */ @@ -189,16 +198,19 @@ typedef enum * * A wrapper for a network descriptor within the gateway, it contains all the * state information necessary to allow for the implementation of the asynchronous - * operation of the potocol and gateway functions. It also provides links to the service + * operation of the protocol and gateway functions. It also provides links to the service * and session data that is required to route the information within the gateway. * * It is important to hold the state information here such that any thread within the * gateway may be selected to execute the required actions when a network event occurs. + * + * Note that the first few fields (up to and including "entry_is_ready") must + * precisely match the LIST_ENTRY structure defined in the list manager. */ typedef struct dcb { + LIST_ENTRY_FIELDS skygw_chk_t dcb_chk_top; - bool dcb_is_in_use; /**< Whether DCB is in use or for later reuse */ bool dcb_errhandle_called; /*< this can be called only once */ bool dcb_is_zombie; /**< Whether the DCB is in the zombie list */ bool draining_flag; /**< Set while write queue is drained */ @@ -232,7 +244,6 @@ typedef struct dcb DCBSTATS stats; /**< DCB related statistics */ unsigned int dcb_server_status; /*< the server role indicator from SERVER */ - struct dcb *next; /**< Next DCB in the chain of allocated DCB's */ struct dcb *nextpersistent; /**< Next DCB in the persistent pool for SERVER */ time_t persistentstart; /**< Time when DCB placed in persistent pool */ struct service *service; /**< The related service */ @@ -260,6 +271,14 @@ typedef struct dcb skygw_chk_t dcb_chk_tail; } DCB; +#define DCB_INIT {.dcb_chk_top = CHK_NUM_DCB, .dcb_initlock = SPINLOCK_INIT, \ + .evq = DCBEVENTQ_INIT, .ipv4 = {0}, .func = {0}, .authfunc = {0}, \ + .writeqlock = SPINLOCK_INIT, .delayqlock = SPINLOCK_INIT, \ + .authlock = SPINLOCK_INIT, .stats = {0}, .memdata = DCBMM_INIT, \ + .cb_lock = SPINLOCK_INIT, .pollinlock = SPINLOCK_INIT, \ + .fd = DCBFD_CLOSED, .stats = DCBSTATS_INIT, .ssl_state = SSL_HANDSHAKE_UNKNOWN, \ + .state = DCB_STATE_ALLOC, .polloutlock = SPINLOCK_INIT, .dcb_chk_tail = CHK_NUM_DCB} + /** * The DCB usage filer used for returning DCB's in use for a certain reason */ @@ -297,6 +316,7 @@ int fail_accept_errno; DCB *dcb_get_zombies(void); int dcb_write(DCB *, GWBUF *); DCB *dcb_accept(DCB *listener, GWPROTOCOL *protocol_funcs); +bool dcb_pre_alloc(int number); DCB *dcb_alloc(dcb_role_t, struct servlistener *); void dcb_free(DCB *); void dcb_free_all_memory(DCB *dcb); @@ -308,6 +328,7 @@ void dcb_close(DCB *); DCB *dcb_process_zombies(int); /* Process Zombies except the one behind the pointer */ void printAllDCBs(); /* Debug to print all DCB in the system */ void printDCB(DCB *); /* Debug print routine */ +void dprintDCBList(DCB *); /* Debug print DCB list statistics */ void dprintAllDCBs(DCB *); /* Debug to print all DCB in the system */ void dprintOneDCB(DCB *, DCB *); /* Debug to print one DCB */ void dprintDCB(DCB *, DCB *); /* Debug to print a DCB in the system */ diff --git a/server/include/def_monitor_event.h b/server/include/def_monitor_event.h index 9263fc0fc..6f0e09393 100644 --- a/server/include/def_monitor_event.h +++ b/server/include/def_monitor_event.h @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/externcmd.h b/server/include/externcmd.h index a9f5df053..f6e2d1568 100644 --- a/server/include/externcmd.h +++ b/server/include/externcmd.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/filter.h b/server/include/filter.h index b9f447cb0..144217f23 100644 --- a/server/include/filter.h +++ b/server/include/filter.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/gw.h b/server/include/gw.h index 5f6ed7f76..616327563 100644 --- a/server/include/gw.h +++ b/server/include/gw.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -82,4 +82,5 @@ int setipaddress(struct in_addr *, char *); char* get_libdir(); long get_processor_count(); void clean_up_pathname(char *path); +bool mxs_mkdir_all(const char *path, int mask); #endif diff --git a/server/include/gw_authenticator.h b/server/include/gw_authenticator.h index 3f76707a7..406cd8523 100644 --- a/server/include/gw_authenticator.h +++ b/server/include/gw_authenticator.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/gw_protocol.h b/server/include/gw_protocol.h index bd18164a8..7022d64a8 100644 --- a/server/include/gw_protocol.h +++ b/server/include/gw_protocol.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/gw_ssl.h b/server/include/gw_ssl.h index 74b329732..196f20a45 100644 --- a/server/include/gw_ssl.h +++ b/server/include/gw_ssl.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/gwbitmask.h b/server/include/gwbitmask.h index 1402c9291..b940dc475 100644 --- a/server/include/gwbitmask.h +++ b/server/include/gwbitmask.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -43,6 +43,8 @@ typedef struct } GWBITMASK; +#define GWBITMASK_INIT {SPINLOCK_INIT} + extern void bitmask_init(GWBITMASK *); extern void bitmask_free(GWBITMASK *); extern void bitmask_set(GWBITMASK *, int); diff --git a/server/include/gwdirs.h.in b/server/include/gwdirs.h.in index 9501c5405..f148d0365 100644 --- a/server/include/gwdirs.h.in +++ b/server/include/gwdirs.h.in @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/hashtable.h b/server/include/hashtable.h index 10ce10417..f6f2ab22e 100644 --- a/server/include/hashtable.h +++ b/server/include/hashtable.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -30,8 +30,8 @@ */ #include #include -#include -#include + +EXTERN_C_BLOCK_BEGIN /** * The entries within a hashtable. @@ -58,9 +58,24 @@ typedef struct hashiterator } HASHITERATOR; /** - * The type definition for the memory allocation functions + * The type definition for the hash function */ -typedef void *(*HASHMEMORYFN)(void *); +typedef int (*HASHHASHFN)(const void *); + +/** + * The type definition for the comparison function + */ +typedef int (*HASHCMPFN)(const void *, const void *); + +/** + * The type definition for the key/value copying functions + */ +typedef void *(*HASHCOPYFN)(const void *); + +/** + * The type definition for the key/value freeing functions + */ +typedef void (*HASHFREEFN)(void *); /** * The general purpose hashtable struct. @@ -72,12 +87,12 @@ typedef struct hashtable #endif int hashsize; /**< The number of HASHENTRIES */ HASHENTRIES **entries; /**< The entries themselves */ - int (*hashfn)(void *); /**< The hash function */ - int (*cmpfn)(void *, void *); /**< The key comparison function */ - HASHMEMORYFN kcopyfn; /**< Optional key copy function */ - HASHMEMORYFN vcopyfn; /**< Optional value copy function */ - HASHMEMORYFN kfreefn; /**< Optional key free function */ - HASHMEMORYFN vfreefn; /**< Optional value free function */ + HASHHASHFN hashfn; /**< The hash function */ + HASHCMPFN cmpfn; /**< The key comparison function */ + HASHCOPYFN kcopyfn; /**< Optional key copy function */ + HASHCOPYFN vcopyfn; /**< Optional value copy function */ + HASHFREEFN kfreefn; /**< Optional key free function */ + HASHFREEFN vfreefn; /**< Optional value free function */ SPINLOCK spin; /**< Internal spinlock for the hashtable */ int n_readers; /**< Number of clients reading the table */ int writelock; /**< The table is locked by a writer */ @@ -88,17 +103,17 @@ typedef struct hashtable #endif } HASHTABLE; -extern HASHTABLE *hashtable_alloc(int, int (*hashfn)(), int (*cmpfn)()); +extern HASHTABLE *hashtable_alloc(int, HASHHASHFN hashfn, HASHCMPFN cmpfn); HASHTABLE *hashtable_alloc_flat(HASHTABLE* target, int size, - int (*hashfn)(), - int (*cmpfn)()); + HASHHASHFN hashfn, + HASHCMPFN cmpfn); /**< Allocate a hashtable */ extern void hashtable_memory_fns(HASHTABLE *table, - HASHMEMORYFN kcopyfn, - HASHMEMORYFN vcopyfn, - HASHMEMORYFN kfreefn, - HASHMEMORYFN vfreefn); + HASHCOPYFN kcopyfn, + HASHCOPYFN vcopyfn, + HASHFREEFN kfreefn, + HASHFREEFN vfreefn); /**< Provide an interface to control key/value memory * manipulation */ @@ -128,4 +143,13 @@ extern void *hashtable_next(HASHITERATOR *); /**< Return the key of the hash table iterator */ extern void hashtable_iterator_free(HASHITERATOR *); extern int hashtable_size(HASHTABLE *table); + +extern void hashtable_item_free(void *data); +extern int hashtable_item_strcasecmp(const void* str1, const void* str2); +extern int hashtable_item_strcmp(const void* str1, const void* str2); +extern void* hashtable_item_strdup(const void *str); +extern int hashtable_item_strhash(const void *str); + +EXTERN_C_BLOCK_END + #endif diff --git a/server/include/hint.h b/server/include/hint.h index c8eea57b9..3f849d361 100644 --- a/server/include/hint.h +++ b/server/include/hint.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/housekeeper.h b/server/include/housekeeper.h index 9a51e7723..1cc6efd2e 100644 --- a/server/include/housekeeper.h +++ b/server/include/housekeeper.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/listener.h b/server/include/listener.h index 6f6081505..aeeca16fe 100644 --- a/server/include/listener.h +++ b/server/include/listener.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -29,8 +29,10 @@ #include #include +#include struct dcb; +struct service; /** * The servlistener structure is used to link a service to the protocols that @@ -40,17 +42,24 @@ struct dcb; */ typedef struct servlistener { + char *name; /**< Name of the listener */ char *protocol; /**< Protocol module to load */ unsigned short port; /**< Port to listen on */ char *address; /**< Address to listen with */ char *authenticator; /**< Name of authenticator */ SSL_LISTENER *ssl; /**< Structure of SSL data or NULL */ struct dcb *listener; /**< The DCB for the listener */ + struct users *users; /**< The user data for this listener */ + HASHTABLE *resources; /**< hastable for listener resources, i.e. database names */ + struct service* service; /**< The service which used by this listener */ + SPINLOCK lock; struct servlistener *next; /**< Next service protocol */ } SERV_LISTENER; -SERV_LISTENER *listener_alloc(char *protocol, char *address, unsigned short port, char *authenticator, +SERV_LISTENER *listener_alloc(struct service* service, char *name, char *protocol, + char *address, unsigned short port, char *authenticator, SSL_LISTENER *ssl); +void listener_free(SERV_LISTENER* listener); int listener_set_ssl_version(SSL_LISTENER *ssl_listener, char* version); void listener_set_certificates(SSL_LISTENER *ssl_listener, char* cert, char* key, char* ca_cert); int listener_init_SSL(SSL_LISTENER *ssl_listener); diff --git a/server/include/listmanager.h b/server/include/listmanager.h new file mode 100644 index 000000000..a0ff8ff02 --- /dev/null +++ b/server/include/listmanager.h @@ -0,0 +1,119 @@ +#ifndef _LISTMANAGER_H +#define _LISTMANAGER_H +/* + * Copyright (c) 2016 MariaDB Corporation Ab + * + * Use of this software is governed by the Business Source License included + * in the LICENSE.TXT file and at www.mariadb.com/bsl. + * + * Change Date: 2019-07-01 + * + * On the date above, in accordance with the Business Source License, use + * of this software will be governed by version 2 or later of the General + * Public License. + */ + +/** + * @file listmanager.h The List Manager header file + * + * + * @verbatim + * Revision History + * + * Date Who Description + * 20/04/2016 Martin Brampton Initial implementation + * + * @endverbatim + */ + +#include +#include + +struct dcb; + +/* + * The possible types of list that could be supported. At present, only + * LIST_TYPE_RECYCLABLE is fully tested. + */ +typedef enum +{ + LIST_TYPE_SIMPLE, /* A simple linked list with pointer to last entry */ + LIST_TYPE_RECYCLABLE, /* A simple linked list, entries are recycled */ + LIST_TYPE_DOUBLE, /* A doubly linked list, next and previous */ +} list_type_t; + +/* + * The list entry structure + * + * Each list entry must have this format, but will typically be extended + * well beyond these items. The "previous" field is only required for + * a doubly linked list, LIST_TYPE_DOUBLE. + * + * The first data to be used with the list manager is the DCB. Note that + * the first few fields in the DCB structure correspond exactly to the + * fields in the list entry structure. The pointers used need to be cast + * as appropriate in order to be able to use the same data as either a DCB + * or a list entry. + * + */ +#define LIST_ENTRY_FIELDS \ + skygw_chk_t list_entry_chk_top; \ + struct list_entry *next; \ + struct list_entry *previous; \ + bool entry_is_in_use; \ + bool entry_is_ready; \ + skygw_chk_t list_entry_chk_tail; + +typedef struct list_entry +{ + LIST_ENTRY_FIELDS +} list_entry_t; + +/* + * The list configuration structure + * + * This provides the basis for a list. It can be declared and initialised + * statically, for example in server/core/dcb.c. It includes an anchor + * pointer for the list, a pointer to the last entry in the list and the + * last entry that was found to be free and reused. + * + * The count tells us the current number of entries in live use, and maximum + * is the highest number ever observed in live use. The freecount is the + * number of entries currently free and ready for reuse. The entry_size is + * the actual size of the real entries, e.g. the DCB structure (NOT the size + * of the list entry structure). + * + * A spinlock is declared for use during list manipulation operations. + */ +typedef struct +{ + list_type_t list_type; + size_t entry_size; + SPINLOCK list_lock; + list_entry_t *all_entries; + list_entry_t *last_entry; + list_entry_t *last_free; + int count; + int maximum; + int freecount; + int num_malloc; +} LIST_CONFIG; + +void list_initialise(LIST_CONFIG *list_config, list_type_t type_of_list, size_t entry_size); +bool list_pre_alloc(LIST_CONFIG *list_config, int num_entries, void (*init_struct)(void *)); +list_entry_t *list_find_free(LIST_CONFIG *list_config, void (*init_struct)(void *)); +void dprintListStats(struct dcb *pdcb, LIST_CONFIG *list_config, const char *listname); +void list_free_entry (LIST_CONFIG *list_config, list_entry_t *to_be_freed); +list_entry_t *list_start_iteration(LIST_CONFIG *list_config); +list_entry_t *list_iterate(LIST_CONFIG *list_config, list_entry_t *current_entry); +void list_terminate_iteration_early(LIST_CONFIG *list_config, list_entry_t *current_entry); +bool list_is_entry_in_use(LIST_CONFIG *list_config, list_entry_t *to_be_found); +void list_add_to_end(LIST_CONFIG *list_config, list_entry_t *new_entry); +void list_map(LIST_CONFIG *list_config, bool (*callback)(void *, ...)); + +/* The following UNTESTED! */ +list_entry_t *list_remove_first(LIST_CONFIG *list_config); +list_entry_t *list_remove_last(LIST_CONFIG *list_config); + + +#endif /* LISTMANAGER_H */ diff --git a/server/include/log_manager.h b/server/include/log_manager.h index 19afa5f9a..3f1152a85 100644 --- a/server/include/log_manager.h +++ b/server/include/log_manager.h @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -28,6 +28,32 @@ extern "C" { #define STRERROR_BUFLEN 512 #endif +/** + * If MXS_MODULE_NAME is defined before log_manager.h is included, then all + * logged messages will be prefixed with that string enclosed in square brackets. + * For instance, the following + * + * #define MXS_MODULE_NAME "xyz" + * #include + * + * will lead to every logged message looking like: + * + * 2016-08-12 13:49:11 error : [xyz] The gadget was not ready + * + * In general, the value of MXS_MODULE_NAME should be the name of the shared + * library to which the source file, where MXS_MODULE_NAME is defined, belongs. + * + * Note that a file that is compiled into multiple modules should + * have MXS_MODULE_NAME defined as something else than the name of a real + * module, or not at all. + * + * Any file that is compiled into maxscale-common should *not* have + * MXS_MODULE_NAME defined. + */ +#if !defined(MXS_MODULE_NAME) +#define MXS_MODULE_NAME NULL +#endif + enum mxs_log_priorities { MXS_LOG_EMERG = (1 << LOG_EMERG), @@ -82,6 +108,13 @@ typedef enum MXS_LOG_AUGMENTATION_MASK = (MXS_LOG_AUGMENT_WITH_FUNCTION) } mxs_log_augmentation_t; +typedef struct mxs_log_throttling +{ + size_t count; // Maximum number of a specific message... + size_t window_ms; // ...during this many milliseconds. + size_t suppress_ms; // If exceeded, suppress such messages for this many ms. +} MXS_LOG_THROTTLING; + bool mxs_log_init(const char* ident, const char* logdir, mxs_log_target_t target); void mxs_log_finish(void); @@ -94,10 +127,14 @@ void mxs_log_set_syslog_enabled(bool enabled); void mxs_log_set_maxlog_enabled(bool enabled); void mxs_log_set_highprecision_enabled(bool enabled); void mxs_log_set_augmentation(int bits); +void mxs_log_set_throttling(const MXS_LOG_THROTTLING* throttling); + +void mxs_log_get_throttling(MXS_LOG_THROTTLING* throttling); int mxs_log_message(int priority, + const char* modname, const char* file, int line, const char* function, - const char* format, ...) __attribute__((format(printf, 5, 6))); + const char* format, ...) __attribute__((format(printf, 6, 7))); /** * Log an error, warning, notice, info, or debug message. * @@ -109,7 +146,7 @@ int mxs_log_message(int priority, * MXS_ERROR, MXS_WARNING, etc. macros instead. */ #define MXS_LOG_MESSAGE(priority, format, ...)\ - mxs_log_message(priority, __FILE__, __LINE__, __func__, format, ##__VA_ARGS__) + mxs_log_message(priority, MXS_MODULE_NAME, __FILE__, __LINE__, __func__, format, ##__VA_ARGS__) /** * Log an error, warning, notice, info, or debug message. @@ -123,6 +160,45 @@ int mxs_log_message(int priority, #define MXS_INFO(format, ...) MXS_LOG_MESSAGE(LOG_INFO, format, ##__VA_ARGS__) #define MXS_DEBUG(format, ...) MXS_LOG_MESSAGE(LOG_DEBUG, format, ##__VA_ARGS__) +/** + * Log an out of memory error using custom message. + * + * @param message Text to be logged. + */ +// TODO: In an OOM situation, the default logging will (most likely) *not* work, +// TODO: as memory is allocated as part of the process. A custom route, that does +// TODO: not allocate memory, must be created for OOM messages. +// TODO: So, currently these are primarily placeholders. +#define MXS_OOM_MESSAGE(message) MXS_ERROR("OOM: %s", message); + +/** + * Log an out of memory error using custom message, if the + * provided pointer is NULL. + * + * @param p If NULL, an OOM message will be logged. + * @param message Text to be logged. + */ +#define MXS_OOM_MESSAGE_IFNULL(p, m) do { if (!p) { MXS_OOM_MESSAGE(m); } } while (false) + +/** + * Log an out of memory error using a default message. + */ +#define MXS_OOM() MXS_OOM_MESSAGE(__func__) + +/** + * Log an out of memory error using a default message, if the + * provided pointer is NULL. + * + * @param p If NULL, an OOM message will be logged. + */ +#define MXS_OOM_IFNULL(p) do { if (!p) { MXS_OOM(); } } while (false) + +enum +{ + MXS_OOM_MESSAGE_MAXLEN = 80 /** Maximum length of an OOM message, including the + trailing NULL. If longer, it will be cut. */ +}; + #if defined(__cplusplus) } #endif diff --git a/server/include/maxconfig.h b/server/include/maxconfig.h index a49dd49d9..2e7970657 100644 --- a/server/include/maxconfig.h +++ b/server/include/maxconfig.h @@ -6,13 +6,14 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General * Public License. */ #include +#include #include #include #include @@ -34,7 +35,6 @@ #define DEFAULT_NBPOLLS 3 /**< Default number of non block polls before we block */ #define DEFAULT_POLLSLEEP 1000 /**< Default poll wait time (milliseconds) */ -#define _SYSNAME_STR_LENGTH 256 /**< sysname len */ #define _RELEASE_STR_LENGTH 256 /**< release len */ #define DEFAULT_NTHREADS 1 /**< Default number of polling threads */ /** @@ -109,8 +109,8 @@ typedef struct { int n_threads; /**< Number of polling threads */ char *version_string; /**< The version string of embedded db library */ - char release_string[_SYSNAME_STR_LENGTH]; /**< The release name string of the system */ - char sysname[_SYSNAME_STR_LENGTH]; /**< The release name string of the system */ + char release_string[_RELEASE_STR_LENGTH]; /**< The release name string of the system */ + char sysname[_UTSNAME_SYSNAME_LENGTH]; /**< The OS name of the system */ uint8_t mac_sha1[SHA_DIGEST_LENGTH]; /**< The SHA1 digest of an interface MAC address */ unsigned long id; /**< MaxScale ID */ unsigned int n_nbpoll; /**< Tune number of non-blocking polls */ diff --git a/server/include/maxscale.h b/server/include/maxscale.h index 940c7610c..020e858d0 100644 --- a/server/include/maxscale.h +++ b/server/include/maxscale.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/maxscale/CMakeLists.txt b/server/include/maxscale/CMakeLists.txt new file mode 100644 index 000000000..9729c1462 --- /dev/null +++ b/server/include/maxscale/CMakeLists.txt @@ -0,0 +1,5 @@ +file(GLOB HEADERS "*.h") +foreach(var ${HEADERS}) + get_filename_component(header ${var} NAME) + install_header(${header} devel) +endforeach() diff --git a/server/include/maxscale/alloc.h b/server/include/maxscale/alloc.h new file mode 100644 index 000000000..88a60bc0c --- /dev/null +++ b/server/include/maxscale/alloc.h @@ -0,0 +1,74 @@ +#ifndef _MAXSCALE_ALLOC_H +#define _MAXSCALE_ALLOC_H +/* + * Copyright (c) 2016 MariaDB Corporation Ab + * + * Use of this software is governed by the Business Source License included + * in the LICENSE.TXT file and at www.mariadb.com/bsl. + * + * Change Date: 2019-07-01 + * + * On the date above, in accordance with the Business Source License, use + * of this software will be governed by version 2 or later of the General + * Public License. + */ + +#include +#include +#include + +EXTERN_C_BLOCK_BEGIN + +/* + * NOTE: Do not use these functions directly, use the macros below. + */ + +// "caller" arg temporarily disabled so that existing code +// using the previous version of mxs_alloc etc. will continue +// to compile. +void *mxs_malloc(size_t size/*, const char *caller*/); +void *mxs_calloc(size_t nmemb, size_t size/*, const char *caller*/); +void *mxs_realloc(void *ptr, size_t size/*, const char *caller*/); +void mxs_free(void *ptr/*, const char *caller*/); + +char *mxs_strdup(const char *s/*, const char *caller*/); +char *mxs_strndup(const char *s, size_t n/*, const char *caller*/); + +char *mxs_strdup_a(const char *s/*, const char *caller*/); +char *mxs_strndup_a(const char *s, size_t n/*, const char *caller*/); + + +/* + * NOTE: USE these macros instead of the functions above. + */ +#define MXS_MALLOC(size) mxs_malloc(size/*, __func__*/) +#define MXS_CALLOC(nmemb, size) mxs_calloc(nmemb, size/*, __func__*/) +#define MXS_REALLOC(ptr, size) mxs_realloc(ptr, size/*, __func__*/) +#define MXS_FREE(ptr) mxs_free(ptr/*, __func__*/) + +#define MXS_STRDUP(s) mxs_strdup(s/*, __func__*/) +#define MXS_STRNDUP(s, n) mxs_strndup(s, n/*, __func__*/) + +#define MXS_STRDUP_A(s) mxs_strdup_a(s/*, __func__*/) +#define MXS_STRNDUP_A(s, n) mxs_strndup_a(s, n/*, __func__*/) + + +/** + * @brief Abort the process if the pointer is NULL. + * + * To be used in circumstances where a memory allocation failure + * cannot - currently - be dealt with properly. + */ +#define MXS_ABORT_IF_NULL(p) do { if (!p) { abort(); } } while (false) + +/** + * @brief Abort the process if the provided value is non-zero. + * + * To be used in circumstances where a memory allocation failure + * cannot - currently - be dealt with properly. + */ +#define MXS_ABORT_IF_TRUE(b) do { if (b) { abort(); } } while (false) + +EXTERN_C_BLOCK_END + +#endif diff --git a/server/include/maxscale/poll.h b/server/include/maxscale/poll.h index 044077f14..296915022 100644 --- a/server/include/maxscale/poll.h +++ b/server/include/maxscale/poll.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/maxscale_pcre2.h b/server/include/maxscale_pcre2.h index 99cb0050a..494b9e6a0 100644 --- a/server/include/maxscale_pcre2.h +++ b/server/include/maxscale_pcre2.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/memlog.h b/server/include/memlog.h index 756be41e6..49167eb14 100644 --- a/server/include/memlog.h +++ b/server/include/memlog.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/mlist.h b/server/include/mlist.h index 4eebe832f..11af38864 100644 --- a/server/include/mlist.h +++ b/server/include/mlist.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/modinfo.h b/server/include/modinfo.h index 3f14d6e11..686ac9d45 100644 --- a/server/include/modinfo.h +++ b/server/include/modinfo.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/modules.h b/server/include/modules.h index 430e751ce..c826e5a38 100644 --- a/server/include/modules.h +++ b/server/include/modules.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/modutil.h b/server/include/modutil.h index ced35f6c5..6d0c44e28 100644 --- a/server/include/modutil.h +++ b/server/include/modutil.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/monitor.h b/server/include/monitor.h index 00f6457c2..0a3e6535e 100644 --- a/server/include/monitor.h +++ b/server/include/monitor.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/mysql_binlog.h b/server/include/mysql_binlog.h index 8c3c90cf3..62b981d64 100644 --- a/server/include/mysql_binlog.h +++ b/server/include/mysql_binlog.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/mysql_utils.h b/server/include/mysql_utils.h index 57b20b87d..3cb26216f 100644 --- a/server/include/mysql_utils.h +++ b/server/include/mysql_utils.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/notification.h b/server/include/notification.h index 589b14f0b..1014f84a1 100644 --- a/server/include/notification.h +++ b/server/include/notification.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/platform.h b/server/include/platform.h index c128febaf..69011f3d4 100644 --- a/server/include/platform.h +++ b/server/include/platform.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/query_classifier.h b/server/include/query_classifier.h index ee434c295..9d9ed0911 100644 --- a/server/include/query_classifier.h +++ b/server/include/query_classifier.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/queuemanager.h b/server/include/queuemanager.h index ab61dda14..b91b2e6e8 100644 --- a/server/include/queuemanager.h +++ b/server/include/queuemanager.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -26,8 +26,8 @@ * @endverbatim */ +#include #include -#include #define CONNECTION_QUEUE_LIMIT 1000 @@ -35,29 +35,36 @@ typedef struct queue_entry { void *queued_object; long heartbeat; +#if defined(SS_DEBUG) + long sequence_check; +#endif /* SS_DEBUG */ } QUEUE_ENTRY; typedef struct queue_config { - int queue_size; int queue_limit; int start; int end; int timeout; + bool has_entries; SPINLOCK queue_lock; - QUEUE_ENTRY queue_array[CONNECTION_QUEUE_LIMIT]; + QUEUE_ENTRY *queue_array; +#if defined(SS_DEBUG) + long sequence_number; +#endif /* SS_DEBUG */ } QUEUE_CONFIG; QUEUE_CONFIG *mxs_queue_alloc(int limit, int timeout); void mxs_queue_free(QUEUE_CONFIG *queue_config); bool mxs_enqueue(QUEUE_CONFIG *queue_config, void *new_entry); -QUEUE_ENTRY *mxs_dequeue(QUEUE_CONFIG *queue_config); +bool mxs_dequeue(QUEUE_CONFIG *queue_config, QUEUE_ENTRY *result); +bool mxs_dequeue_if_expired(QUEUE_CONFIG *queue_config, QUEUE_ENTRY *result); static inline int mxs_queue_count(QUEUE_CONFIG *queue_config) { int count = queue_config->end - queue_config->start; - return count < 0 ? (count + queue_config->queue_size) : count; + return count < 0 ? (count + queue_config->queue_limit + 1): count; } #endif /* QUEUEMANAGER_H */ diff --git a/server/include/random_jkiss.h b/server/include/random_jkiss.h index f6e4b6062..baf9be28a 100644 --- a/server/include/random_jkiss.h +++ b/server/include/random_jkiss.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/rdtsc.h b/server/include/rdtsc.h index a766ca167..213d73b7f 100644 --- a/server/include/rdtsc.h +++ b/server/include/rdtsc.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/resultset.h b/server/include/resultset.h index ca1c498ec..6c7feb408 100644 --- a/server/include/resultset.h +++ b/server/include/resultset.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/router.h b/server/include/router.h index de1043d6a..e4e6a1a8b 100644 --- a/server/include/router.h +++ b/server/include/router.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/secrets.h b/server/include/secrets.h index ea45461cd..09286c00b 100644 --- a/server/include/secrets.h +++ b/server/include/secrets.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -47,7 +47,12 @@ typedef struct maxkeys unsigned char initvector[MAXSCALE_IV_LEN]; } MAXKEYS; -extern int secrets_writeKeys(const char *filename); +enum +{ + MXS_PASSWORD_MAXLEN = 79 +}; + +extern int secrets_writeKeys(const char *directory); extern char *decryptPassword(const char *); extern char *encryptPassword(const char*, const char *); diff --git a/server/include/server.h b/server/include/server.h index d8b3e1af5..543411ca7 100644 --- a/server/include/server.h +++ b/server/include/server.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/service.h b/server/include/service.h index 144de3a29..8e0d9de89 100644 --- a/server/include/service.h +++ b/server/include/service.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -134,10 +134,8 @@ typedef struct service SERVICE_USER credentials; /**< The cedentials of the service user */ SPINLOCK spin; /**< The service spinlock */ SERVICE_STATS stats; /**< The service statistics */ - struct users *users; /**< The user data for this service */ int enable_root; /**< Allow root user access */ int localhost_match_wildcard_host; /**< Match localhost against wildcard */ - HASHTABLE *resources; /**< hastable for service resources, i.e. database names */ CONFIG_PARAMETER* svc_config_param;/*< list of config params and values */ int svc_config_version; /*< Version number of configuration */ bool svc_do_shutdown; /*< tells the service to exit loops etc. */ @@ -173,7 +171,9 @@ extern SERVICE *service_alloc(const char *, const char *); extern int service_free(SERVICE *); extern SERVICE *service_find(char *); extern int service_isvalid(SERVICE *); -extern int serviceAddProtocol(SERVICE *, char *, char *, unsigned short, char *, SSL_LISTENER *); +extern int serviceAddProtocol(SERVICE *service, char *name, char *protocol, + char *address, unsigned short port, + char *authenticator, SSL_LISTENER *ssl); extern int serviceHasProtocol(SERVICE *service, const char *protocol, const char* address, unsigned short port); extern void serviceAddBackend(SERVICE *, SERVER *); diff --git a/server/include/session.h b/server/include/session.h index ecc0a76e1..f808dfbf6 100644 --- a/server/include/session.h +++ b/server/include/session.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -29,12 +29,14 @@ * 29-05-2014 Mark Riddoch Support for filter mechanism * added * 20-02-2015 Markus Mäkelä Added session timeouts + * 27/06/2016 Martin Brampton Modify session struct for list manager * * @endverbatim */ #include #include #include +#include #include #include #include @@ -52,6 +54,8 @@ typedef struct time_t connect; /**< Time when the session was started */ } SESSION_STATS; +#define SESSION_STATS_INIT {0} + typedef enum { SESSION_STATE_ALLOC, /*< for all sessions */ @@ -76,6 +80,8 @@ typedef struct int (*routeQuery)(void *instance, void *session, GWBUF *request); } DOWNSTREAM; +#define DOWNSTREAM_INIT {0} + /** * The upstream element in the filter chain. This may refer to * another filter or to the protocol implementation. @@ -88,6 +94,8 @@ typedef struct int (*error)(void *instance, void *session, void *); } UPSTREAM; +#define UPSTREAM_INIT {0} + /** * Structure used to track the filter instances and sessions of the filters * that are in use within a session. @@ -99,6 +107,8 @@ typedef struct void *session; } SESSION_FILTER; +#define SESSION_FILTER_INIT {0} + /** * Filter type for the sessionGetList call */ @@ -114,13 +124,14 @@ typedef enum * A session status block is created for each user (client) connection * to the database, it links the descriptors, routing implementation * and originating service together for the client session. + * + * Note that the first few fields (up to and including "entry_is_ready") must + * precisely match the LIST_ENTRY structure defined in the list manager. */ typedef struct session { -#if defined(SS_DEBUG) + LIST_ENTRY_FIELDS skygw_chk_t ses_chk_top; -#endif - bool ses_is_in_use; /**< Whether session is in use or for later reuse */ SPINLOCK ses_lock; session_state_t state; /*< Current descriptor state */ size_t ses_id; /*< Unique session identifier */ @@ -133,14 +144,15 @@ typedef struct session SESSION_FILTER *filters; /*< The filters in use within this session */ DOWNSTREAM head; /*< Head of the filter chain */ UPSTREAM tail; /*< The tail of the filter chain */ - struct session *next; /*< Linked list of all sessions */ int refcount; /*< Reference count on the session */ bool ses_is_child; /*< this is a child session */ -#if defined(SS_DEBUG) skygw_chk_t ses_chk_tail; -#endif } SESSION; +#define SESSION_INIT {.ses_chk_top = CHK_NUM_SESSION, .ses_lock = SPINLOCK_INIT, \ + .stats = SESSION_STATS_INIT, .head = DOWNSTREAM_INIT, .tail = UPSTREAM_INIT, \ + .state = SESSION_STATE_ALLOC, .ses_chk_tail = CHK_NUM_SESSION} + /** Whether to do session timeout checks */ extern bool check_timeouts; @@ -167,8 +179,8 @@ extern long next_timeout_check; ((sess)->tail.clientReply)((sess)->tail.instance, \ (sess)->tail.session, (buf)) -SESSION *get_all_sessions(); SESSION *session_alloc(struct service *, struct dcb *); +bool session_pre_alloc(int number); SESSION *session_set_dummy(struct dcb *); bool session_free(SESSION *); int session_isvalid(SESSION *); @@ -177,6 +189,7 @@ char *session_get_remote(SESSION *); char *session_getUser(SESSION *); void printAllSessions(); void printSession(SESSION *); +void dprintSessionList(DCB *pdcb); void dprintAllSessions(struct dcb *); void dprintSession(struct dcb *, SESSION *); void dListSessions(struct dcb *); diff --git a/utils/skygw_debug.h b/server/include/skygw_debug.h similarity index 98% rename from utils/skygw_debug.h rename to server/include/skygw_debug.h index dd0ce10c4..9a874de4e 100644 --- a/utils/skygw_debug.h +++ b/server/include/skygw_debug.h @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -135,12 +135,13 @@ typedef enum skygw_chk_t CHK_NUM_PREP_STMT, CHK_NUM_PINFO, CHK_NUM_MYSQLSES, - CHK_NUM_ADMINSES + CHK_NUM_ADMINSES, + CHK_NUM_MANAGED_LIST } skygw_chk_t; # define STRBOOL(b) ((b) ? "true" : "false") -# define STRQTYPE(t) ((t) == QUERY_TYPE_WRITE ? "QUERY_TYPE_WRITE" : \ +# define STRQTYPE(t) ((t) == QUERY_TYPE_WRITE ? "QUERY_TYPE_WRITE" : \ ((t) == QUERY_TYPE_READ ? "QUERY_TYPE_READ" : \ ((t) == QUERY_TYPE_SESSION_WRITE ? "QUERY_TYPE_SESSION_WRITE" : \ ((t) == QUERY_TYPE_UNKNOWN ? "QUERY_TYPE_UNKNOWN" : \ @@ -483,22 +484,30 @@ typedef enum skygw_chk_t "Hashtable under- or overflow"); \ } +#define CHK_MANAGED_LIST(l) { \ + ss_info_dassert(l->list_entry_chk_top == CHK_NUM_MANAGED_LIST && \ + l->list_entry_chk_tail == CHK_NUM_MANAGED_LIST, \ + "Managed list under- or overflow"); \ + } + #define CHK_DCB(d) { \ ss_info_dassert(d->dcb_chk_top == CHK_NUM_DCB && \ d->dcb_chk_tail == CHK_NUM_DCB, \ "Dcb under- or overflow"); \ + CHK_MANAGED_LIST(d) \ } -#define CHK_PROTOCOL(p) { \ +#define CHK_PROTOCOL(p) { \ ss_info_dassert(p->protocol_chk_top == CHK_NUM_PROTOCOL && \ p->protocol_chk_tail == CHK_NUM_PROTOCOL, \ "Protocol under- or overflow"); \ } -#define CHK_SESSION(s) { \ - ss_info_dassert(s->ses_chk_top == CHK_NUM_SESSION && \ +#define CHK_SESSION(s) { \ + ss_info_dassert(s->ses_chk_top == CHK_NUM_SESSION && \ s->ses_chk_tail == CHK_NUM_SESSION, \ "Session under- or overflow"); \ + CHK_MANAGED_LIST(s) \ } #define CHK_SERVER(s) { \ diff --git a/utils/skygw_types.h b/server/include/skygw_types.h similarity index 97% rename from utils/skygw_types.h rename to server/include/skygw_types.h index 52305f48c..0c8247eeb 100644 --- a/utils/skygw_types.h +++ b/server/include/skygw_types.h @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/utils/skygw_utils.h b/server/include/skygw_utils.h similarity index 93% rename from utils/skygw_utils.h rename to server/include/skygw_utils.h index cfbe50c03..602f5fa03 100644 --- a/utils/skygw_utils.h +++ b/server/include/skygw_utils.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -141,10 +141,18 @@ EXTERN_C_BLOCK_END /** Skygw thread routines */ /** Skygw file routines */ -skygw_file_t* skygw_file_alloc(char* fname); +typedef enum skygw_open_mode +{ + SKYGW_OPEN_APPEND, + SKYGW_OPEN_TRUNCATE, +} skygw_open_mode_t; + +skygw_file_t* skygw_file_alloc(const char* fname); void skygw_file_free(skygw_file_t* file); -skygw_file_t* skygw_file_init(char* fname, char* symlinkname); -void skygw_file_close(skygw_file_t* file, bool shutdown); +skygw_file_t* skygw_file_init(const char* fname, + const char* symlinkname, + skygw_open_mode_t mode); +void skygw_file_close(skygw_file_t* file); int skygw_file_write(skygw_file_t* file, void* data, size_t nbytes, @@ -194,7 +202,6 @@ bool is_valid_posix_path(char* path); bool strip_escape_chars(char*); char* trim(char *str); char* squeeze_whitespace(char* str); -int simple_str_hash(char* key); EXTERN_C_BLOCK_END diff --git a/server/include/slist.h b/server/include/slist.h index ad9de61dd..c7a179e95 100644 --- a/server/include/slist.h +++ b/server/include/slist.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/spinlock.h b/server/include/spinlock.h index 1598e7800..34794165c 100644 --- a/server/include/spinlock.h +++ b/server/include/spinlock.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -23,8 +23,9 @@ * for the lock to be released. However they are useful in that they do not involve * system calls and are light weight when the expected wait time for a lock is low. */ -#include -#include +#include + +EXTERN_C_BLOCK_BEGIN #define SPINLOCK_PROFILE 0 @@ -68,9 +69,11 @@ typedef struct spinlock #define SPINLOCK_IS_LOCKED(l) ((l)->lock != 0 ? true : false) extern void spinlock_init(SPINLOCK *lock); -extern void spinlock_acquire(SPINLOCK *lock); -extern int spinlock_acquire_nowait(SPINLOCK *lock); -extern void spinlock_release(SPINLOCK *lock); -extern void spinlock_stats(SPINLOCK *lock, void (*reporter)(void *, char *, int), void *hdl); +extern void spinlock_acquire(const SPINLOCK *lock); +extern int spinlock_acquire_nowait(const SPINLOCK *lock); +extern void spinlock_release(const SPINLOCK *lock); +extern void spinlock_stats(const SPINLOCK *lock, void (*reporter)(void *, char *, int), void *hdl); + +EXTERN_C_BLOCK_END #endif diff --git a/server/include/statistics.h b/server/include/statistics.h index 00750cd23..507742f98 100644 --- a/server/include/statistics.h +++ b/server/include/statistics.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -19,8 +19,9 @@ * @verbatim * Revision History * - * Date Who Description - * 21/01/16 Markus Makela Initial implementation + * Date Who Description + * 21/01/16 Markus Makela Initial implementation + * 15/06/16 Martin Brampton Frequently used functions inlined * @endverbatim */ @@ -32,13 +33,37 @@ void ts_stats_init(); /** No-op for now */ void ts_stats_end(); -/** Every thread should call set_current_thread_id only once */ -void ts_stats_set_thread_id(int id); - ts_stats_t ts_stats_alloc(); void ts_stats_free(ts_stats_t stats); -void ts_stats_add(ts_stats_t stats, int value); -void ts_stats_set(ts_stats_t stats, int value); int ts_stats_sum(ts_stats_t stats); +/** + * @brief Increment thread statistics by one + * + * @param stats Statistics to add to + * @param thread_id ID of thread + */ +static void inline +ts_stats_increment(ts_stats_t stats, int thread_id) +{ + ((int*)stats)[thread_id]++; +} + +/** + * @brief Assign a value to a statistics element + * + * This sets the value for the specified thread. + * + * @param stats Statistics to set + * @param value Value to set to + * @param thread_id ID of thread + * + * @note Appears to be unused + */ +static void inline +ts_stats_set(ts_stats_t stats, int value, int thread_id) +{ + ((int*)stats)[thread_id] = value; +} + #endif diff --git a/server/include/thread.h b/server/include/thread.h index 51aef4c81..3defe9f90 100644 --- a/server/include/thread.h +++ b/server/include/thread.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/users.h b/server/include/users.h index 1bf286ec9..592cf5469 100644 --- a/server/include/users.h +++ b/server/include/users.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/include/utils.h b/server/include/utils.h index 094c0e5f9..273f352ca 100644 --- a/server/include/utils.h +++ b/server/include/utils.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/maxscale_template.cnf b/server/maxscale.cnf.template similarity index 100% rename from server/maxscale_template.cnf rename to server/maxscale.cnf.template diff --git a/server/modules/CMakeLists.txt b/server/modules/CMakeLists.txt index f3d806251..7bfbb71fa 100644 --- a/server/modules/CMakeLists.txt +++ b/server/modules/CMakeLists.txt @@ -4,3 +4,4 @@ add_subdirectory(protocol) add_subdirectory(monitor) add_subdirectory(filter) add_subdirectory(authenticator) +add_subdirectory(include) diff --git a/server/modules/authenticator/CMakeLists.txt b/server/modules/authenticator/CMakeLists.txt index bb8fa05d7..abe32bec3 100644 --- a/server/modules/authenticator/CMakeLists.txt +++ b/server/modules/authenticator/CMakeLists.txt @@ -1,28 +1,38 @@ add_library(MySQLAuth SHARED mysql_auth.c) target_link_libraries(MySQLAuth maxscale-common) set_target_properties(MySQLAuth PROPERTIES VERSION "1.0.0") -install(TARGETS MySQLAuth DESTINATION ${MAXSCALE_LIBDIR}) +install_module(MySQLAuth core) -add_library(NullAuth SHARED null_auth.c) -target_link_libraries(NullAuth maxscale-common) -set_target_properties(NullAuth PROPERTIES VERSION "1.0.0") -install(TARGETS NullAuth DESTINATION ${MAXSCALE_LIBDIR}) +add_library(NullAuthAllow SHARED null_auth_allow.c) +target_link_libraries(NullAuthAllow maxscale-common) +set_target_properties(NullAuthAllow PROPERTIES VERSION "1.0.0") +install_module(NullAuthAllow core) + +add_library(NullAuthDeny SHARED null_auth_deny.c) +target_link_libraries(NullAuthDeny maxscale-common) +set_target_properties(NullAuthDeny PROPERTIES VERSION "1.0.0") +install_module(NullAuthDeny core) add_library(MaxAdminAuth SHARED max_admin_auth.c) target_link_libraries(MaxAdminAuth maxscale-common) set_target_properties(MaxAdminAuth PROPERTIES VERSION "1.0.0") -install(TARGETS MaxAdminAuth DESTINATION ${MAXSCALE_LIBDIR}) +install_module(MaxAdminAuth core) + +add_library(HTTPAuth SHARED http_auth.c) +target_link_libraries(HTTPAuth maxscale-common) +set_target_properties(HTTPAuth PROPERTIES VERSION "1.0.0") +install_module(HTTPAuth core) # if(BUILD_TESTS) # add_library(testprotocol SHARED testprotocol.c) # set_target_properties(testprotocol PROPERTIES VERSION "1.0.0") # target_link_libraries(testprotocol maxscale-common) -# install(TARGETS testprotocol DESTINATION ${MAXSCALE_LIBDIR}) +# install_module(testprotocol core) # endif() if(BUILD_CDC) add_library(CDCPlainAuth SHARED cdc_plain_auth.c) target_link_libraries(CDCPlainAuth maxscale-common) set_target_properties(CDCPlainAuth PROPERTIES VERSION "1.0.0") - install(TARGETS CDCPlainAuth DESTINATION ${MAXSCALE_LIBDIR}) + install_module(CDCPlainAuth core) endif() diff --git a/server/modules/authenticator/cdc_plain_auth.c b/server/modules/authenticator/cdc_plain_auth.c index 0c605f675..3059615f8 100644 --- a/server/modules/authenticator/cdc_plain_auth.c +++ b/server/modules/authenticator/cdc_plain_auth.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -30,6 +30,7 @@ #include #include #include +#include /* Allowed time interval (in seconds) after last update*/ #define CDC_USERS_REFRESH_TIME 30 @@ -53,11 +54,11 @@ static bool cdc_auth_is_client_ssl_capable(DCB *dcb); static int cdc_auth_authenticate(DCB *dcb); static void cdc_auth_free_client_data(DCB *dcb); -static int cdc_set_service_user(SERVICE *service); +static int cdc_set_service_user(SERV_LISTENER *listener); static int cdc_read_users(USERS *users, char *usersfile); -static int cdc_load_users(SERVICE *service); -static int cdc_refresh_users(SERVICE *service); -static int cdc_replace_users(SERVICE *service); +static int cdc_load_users(SERV_LISTENER *listener); +static int cdc_refresh_users(SERV_LISTENER *listener); +static int cdc_replace_users(SERV_LISTENER *listener); extern char *gw_bin2hex(char *out, const uint8_t *in, unsigned int len); extern void gw_sha1_str(const uint8_t *in, int in_len, uint8_t *out); @@ -140,15 +141,15 @@ static int cdc_auth_check(DCB *dcb, CDC_protocol *protocol, char *username, uint if (!cdc_load_users_init) { /* Load db users or set service user */ - if (cdc_load_users(dcb->service) < 1) + if (cdc_load_users(dcb->listener) < 1) { - cdc_set_service_user(dcb->service); + cdc_set_service_user(dcb->listener); } cdc_load_users_init = 1; } - user_password = users_fetch(dcb->service->users, username); + user_password = users_fetch(dcb->listener->users, username); if (!user_password) { @@ -200,7 +201,7 @@ cdc_auth_authenticate(DCB *dcb) auth_ret = cdc_auth_check(dcb, protocol, client_data->user, client_data->auth_data, client_data->flags); /* On failed authentication try to reload user table */ - if (CDC_STATE_AUTH_OK != auth_ret && 0 == cdc_refresh_users(dcb->service)) + if (CDC_STATE_AUTH_OK != auth_ret && 0 == cdc_refresh_users(dcb->listener)) { /* Call protocol authentication */ auth_ret = cdc_auth_check(dcb, protocol, client_data->user, client_data->auth_data, client_data->flags); @@ -209,7 +210,7 @@ cdc_auth_authenticate(DCB *dcb) /* on successful authentication, set user into dcb field */ if (CDC_STATE_AUTH_OK == auth_ret) { - dcb->user = strdup(client_data->user); + dcb->user = MXS_STRDUP_A(client_data->user); } else if (dcb->service->log_auth_warnings) { @@ -248,7 +249,7 @@ cdc_auth_set_protocol_data(DCB *dcb, GWBUF *buf) CHK_PROTOCOL(protocol); if (dcb->data == NULL) { - if (NULL == (client_data = (CDC_session *)calloc(1, sizeof(CDC_session)))) + if (NULL == (client_data = (CDC_session *)MXS_CALLOC(1, sizeof(CDC_session)))) { return CDC_STATE_AUTH_ERR; } @@ -356,7 +357,7 @@ cdc_auth_is_client_ssl_capable(DCB *dcb) static void cdc_auth_free_client_data(DCB *dcb) { - free(dcb->data); + MXS_FREE(dcb->data); } /* @@ -367,8 +368,9 @@ cdc_auth_free_client_data(DCB *dcb) * @return 0 on success, 1 on failure */ static int -cdc_set_service_user(SERVICE *service) +cdc_set_service_user(SERV_LISTENER *listener) { + SERVICE *service = listener->service; char *dpwd = NULL; char *newpasswd = NULL; char *service_user = NULL; @@ -400,15 +402,15 @@ cdc_set_service_user(SERVICE *service) MXS_ERROR("create hex_sha1_sha1_password failed for service user %s", service_user); - free(dpwd); + MXS_FREE(dpwd); return 1; } /* add service user */ - (void)users_add(service->users, service->credentials.name, newpasswd); + (void)users_add(listener->users, service->credentials.name, newpasswd); - free(newpasswd); - free(dpwd); + MXS_FREE(newpasswd); + MXS_FREE(dpwd); return 0; } @@ -420,8 +422,9 @@ cdc_set_service_user(SERVICE *service) * @return -1 on failure, 0 for no users found, > 0 for found users */ static int -cdc_load_users(SERVICE *service) +cdc_load_users(SERV_LISTENER *listener) { + SERVICE *service = listener->service; int loaded = -1; char path[PATH_MAX + 1] = ""; @@ -429,13 +432,13 @@ cdc_load_users(SERVICE *service) snprintf(path, PATH_MAX, "%s/%s/cdcusers", get_datadir(), service->name); /* Allocate users table */ - if (service->users == NULL) + if (listener->users == NULL) { - service->users = users_alloc(); + listener->users = users_alloc(); } /* Try loading authentication data from file cache */ - loaded = cdc_read_users(service->users, path); + loaded = cdc_read_users(listener->users, path); if (loaded == -1) { @@ -491,11 +494,8 @@ cdc_read_users(USERS *users, char *usersfile) filelen = statb.st_size; } - if ((all_users_data = malloc(filelen + 1)) == NULL) + if ((all_users_data = MXS_MALLOC(filelen + 1)) == NULL) { - MXS_ERROR("failed to allocate %i for service user data load %s", - filelen + 1, - usersfile); return -1; } @@ -532,7 +532,7 @@ cdc_read_users(USERS *users, char *usersfile) memcpy(users->cksum, hash, SHA_DIGEST_LENGTH); - free(all_users_data); + MXS_FREE(all_users_data); fclose(fp); @@ -549,9 +549,11 @@ cdc_read_users(USERS *users, char *usersfile) * * @return 0 on success and 1 on error * */ static int -cdc_refresh_users(SERVICE *service) +cdc_refresh_users(SERV_LISTENER *listener) { int ret = 1; + SERVICE *service = listener->service; + /* check for another running getUsers request */ if (!spinlock_acquire_nowait(&service->users_table_spin)) { @@ -582,7 +584,7 @@ cdc_refresh_users(SERVICE *service) service->rate_limit.last = time(NULL); } - ret = cdc_replace_users(service); + ret = cdc_replace_users(listener); /* remove lock */ spinlock_release(&service->users_table_spin); @@ -604,11 +606,11 @@ cdc_refresh_users(SERVICE *service) * @return -1 on any error or the number of users inserted (0 means no users at all) * */ static int -cdc_replace_users(SERVICE *service) +cdc_replace_users(SERV_LISTENER *listener) { + SERVICE *service = listener->service; int i; USERS *newusers, *oldusers; - HASHTABLE *oldresources; char path[PATH_MAX + 1] = ""; /* File path for router cached authentication data */ @@ -629,8 +631,8 @@ cdc_replace_users(SERVICE *service) return i; } - spinlock_acquire(&service->spin); - oldusers = service->users; + spinlock_acquire(&listener->lock); + oldusers = listener->users; /* digest compare */ if (oldusers != NULL && memcmp(oldusers->cksum, newusers->cksum, @@ -649,10 +651,10 @@ cdc_replace_users(SERVICE *service) /* replace the service with effective new data */ MXS_DEBUG("%lu [cdc_replace_users] users' tables replaced, checksum differs", pthread_self()); - service->users = newusers; + listener->users = newusers; } - spinlock_release(&service->spin); + spinlock_release(&listener->lock); if (i && oldusers) { diff --git a/server/modules/authenticator/http_auth.c b/server/modules/authenticator/http_auth.c new file mode 100644 index 000000000..8c34baef2 --- /dev/null +++ b/server/modules/authenticator/http_auth.c @@ -0,0 +1,223 @@ +/* + * Copyright (c) 2016 MariaDB Corporation Ab + * + * Use of this software is governed by the Business Source License included + * in the LICENSE.TXT file and at www.mariadb.com/bsl. + * + * Change Date: 2019-07-01 + * + * On the date above, in accordance with the Business Source License, use + * of this software will be governed by version 2 or later of the General + * Public License. + */ + +/** + * @file http_ba__auth.c + * + * MaxScale HTTP Basic Access authentication for the HTTPD protocol module. + * + * @verbatim + * Revision History + * Date Who Description + * 20/07/2016 Markus Makela Initial version + * + * @endverbatim + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +/* @see function load_module in load_utils.c for explanation of the following + * lint directives. + */ +/*lint -e14 */ +MODULE_INFO info = +{ + MODULE_API_AUTHENTICATOR, + MODULE_GA, + GWAUTHENTICATOR_VERSION, + "The MaxScale HTTP BA authenticator" +}; +/*lint +e14 */ + +static char *version_str = "V1.0.0"; + +static int http_auth_set_protocol_data(DCB *dcb, GWBUF *buf); +static bool http_auth_is_client_ssl_capable(DCB *dcb); +static int http_auth_authenticate(DCB *dcb); +static void http_auth_free_client_data(DCB *dcb); + +/* + * The "module object" for mysql client authenticator module. + */ +static GWAUTHENTICATOR MyObject = +{ + http_auth_set_protocol_data, /* Extract data into structure */ + http_auth_is_client_ssl_capable, /* Check if client supports SSL */ + http_auth_authenticate, /* Authenticate user credentials */ + http_auth_free_client_data, /* Free the client data held in DCB */ +}; + +typedef struct http_auth +{ + char* user; + char* pw; +}HTTP_AUTH; + +/** + * Implementation of the mandatory version entry point + * + * @return version string of the module + */ +/* @see function load_module in load_utils.c for explanation of the following + * lint directives. +*/ +/*lint -e14 */ +char* version() +{ + return version_str; +} + +/** + * The module initialisation routine, called when the module + * is first loaded. + */ +void ModuleInit() +{ +} + +/** + * The module entry point routine. It is this routine that + * must populate the structure that is referred to as the + * "module object", this is a structure with the set of + * external entry points for this module. + * + * @return The module object + */ +GWAUTHENTICATOR* GetModuleObject() +{ + return &MyObject; +} +/*lint +e14 */ + +/** + * @brief Authentication of a user/password combination. + * + * The validation is already done, the result is returned. + * + * @param dcb Request handler DCB connected to the client + * @return Authentication status - always 0 to denote success + */ +static int +http_auth_authenticate(DCB *dcb) +{ + int rval = 1; + HTTP_AUTH *ses = (HTTP_AUTH*)dcb->data; + char *user, *pw; + serviceGetUser(dcb->service, &user, &pw); + pw = decryptPassword(pw); + + if (ses && strcmp(ses->user, user) == 0 && strcmp(ses->pw, pw) == 0) + { + rval = 0; + } + + free(pw); + return rval; +} + +/** + * @brief Transfer data from the authentication request to the DCB. + * + * Expects a buffer containing a Base64 encoded username and password + * contatenated together by a semicolon as is specificed by HTTP Basic + * Access authentication. + * + * @param dcb Request handler DCB connected to the client + * @param buffer Pointer to pointer to buffers containing data from client + * @return Authentication status - 0 for success, 1 for failure + */ +static int +http_auth_set_protocol_data(DCB *dcb, GWBUF *buf) +{ + int rval = 1; + char* value = (char*)GWBUF_DATA(buf); + char* tok = strstr(value, "Basic"); + tok = tok ? strchr(tok, ' ') : NULL; + if (tok) + { + tok++; + char outbuf[strlen(tok) * 2 + 1]; + + BIO *b64 = BIO_new(BIO_f_base64()); + BIO_set_flags(b64, BIO_FLAGS_BASE64_NO_NL); + + BIO *bio = BIO_new_mem_buf(tok, -1); + BIO_push(b64, bio); + int nread = BIO_read(b64, outbuf, sizeof(outbuf)); + outbuf[nread] = '\0'; + BIO_free_all(b64); + char *pw_start = strchr(outbuf, ':'); + if (pw_start) + { + *pw_start++ = '\0'; + HTTP_AUTH *ses = MXS_MALLOC(sizeof(*ses)); + char* user = MXS_STRDUP(outbuf); + char* pw = MXS_STRDUP(pw_start); + + if (ses && user && pw) + { + ses->user = user; + ses->pw = pw; + dcb->data = ses; + rval = 0; + } + else + { + MXS_FREE(ses); + MXS_FREE(user); + MXS_FREE(pw); + } + } + } + + return rval; +} + +/** + * @brief Determine whether the client is SSL capable + * + * Always say that client is not SSL capable. Support for SSL is not yet + * available. + * + * @param dcb Request handler DCB connected to the client + * @return Boolean indicating whether client is SSL capable - false + */ +static bool +http_auth_is_client_ssl_capable(DCB *dcb) +{ + return false; +} + +/** + * @brief Free the client data pointed to by the passed DCB. + * + * The authentication data stored in the DCB is a pointer to a HTTP_AUTH + * structure allocated in http_auth_set_protocol_data(). + * + * @param dcb Request handler DCB connected to the client + */ +static void +http_auth_free_client_data(DCB *dcb) +{ + HTTP_AUTH *ses = (HTTP_AUTH*)dcb->data; + MXS_FREE(ses->user); + MXS_FREE(ses->pw); + MXS_FREE(ses); +} diff --git a/server/modules/authenticator/max_admin_auth.c b/server/modules/authenticator/max_admin_auth.c index c90d77c65..5da29d7a9 100644 --- a/server/modules/authenticator/max_admin_auth.c +++ b/server/modules/authenticator/max_admin_auth.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -27,6 +27,7 @@ */ #include +#include #include #include #include @@ -130,7 +131,7 @@ max_admin_auth_set_protocol_data(DCB *dcb, GWBUF *buf) max_admin_auth_free_client_data(dcb); - if ((session_data = (ADMIN_session *)calloc(1, sizeof(ADMIN_session))) != NULL) + if ((session_data = (ADMIN_session *)MXS_CALLOC(1, sizeof(ADMIN_session))) != NULL) { int user_len = (GWBUF_LENGTH(buf) > ADMIN_USER_MAXLEN) ? ADMIN_USER_MAXLEN : GWBUF_LENGTH(buf); #if defined(SS_DEBUG) @@ -170,12 +171,12 @@ max_admin_auth_is_client_ssl_capable(DCB *dcb) * @brief Free the client data pointed to by the passed DCB. * * The max_admin authenticator uses a simple structure that can be freed with - * a single call to free(). + * a single call to MXS_FREE(). * * @param dcb Request handler DCB connected to the client */ static void max_admin_auth_free_client_data(DCB *dcb) { - free(dcb->data); + MXS_FREE(dcb->data); } diff --git a/server/modules/authenticator/mysql_auth.c b/server/modules/authenticator/mysql_auth.c index b5fabe1d7..b40c7f6d7 100644 --- a/server/modules/authenticator/mysql_auth.c +++ b/server/modules/authenticator/mysql_auth.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -28,6 +28,7 @@ #include #include #include +#include #include /* @see function load_module in load_utils.c for explanation of the following @@ -181,7 +182,7 @@ mysql_auth_authenticate(DCB *dcb) /* on successful authentication, set user into dcb field */ if (MYSQL_AUTH_SUCCEEDED == auth_ret) { - dcb->user = strdup(client_data->user); + dcb->user = MXS_STRDUP_A(client_data->user); } else if (dcb->service->log_auth_warnings) { @@ -200,7 +201,7 @@ mysql_auth_authenticate(DCB *dcb) /* let's free the auth_token now */ if (client_data->auth_token) { - free(client_data->auth_token); + MXS_FREE(client_data->auth_token); client_data->auth_token = NULL; } } @@ -236,7 +237,7 @@ mysql_auth_set_protocol_data(DCB *dcb, GWBUF *buf) CHK_PROTOCOL(protocol); if (dcb->data == NULL) { - if (NULL == (client_data = (MYSQL_session *)calloc(1, sizeof(MYSQL_session)))) + if (NULL == (client_data = (MYSQL_session *)MXS_CALLOC(1, sizeof(MYSQL_session)))) { return MYSQL_FAILED_AUTH; } @@ -345,7 +346,7 @@ mysql_auth_set_client_data( (packet_length_used + client_data->auth_token_len)) { /* Packet is large enough for authentication token */ - if (NULL != (client_data->auth_token = (uint8_t *)malloc(client_data->auth_token_len))) + if (NULL != (client_data->auth_token = (uint8_t *)MXS_MALLOC(client_data->auth_token_len))) { /* The extra 1 is for the token length byte, just extracted*/ memcpy(client_data->auth_token, @@ -414,7 +415,7 @@ mysql_auth_is_client_ssl_capable(DCB *dcb) * gw_find_mysql_user_password_sha1 * * The routine fetches an user from the MaxScale users' table - * The users' table is dcb->service->users or a different one specified with void *repository + * The users' table is dcb->listener->users or a different one specified with void *repository * The user lookup uses username,host and db name (if passed in connection or change user) * * If found the HEX password, representing sha1(sha1(password)), is converted in binary data and @@ -428,20 +429,17 @@ mysql_auth_is_client_ssl_capable(DCB *dcb) */ int gw_find_mysql_user_password_sha1(char *username, uint8_t *gateway_password, DCB *dcb) { - SERVICE *service = NULL; - struct sockaddr_in *client; - char *user_password = NULL; - MYSQL_USER_HOST key; - MYSQL_session *client_data = NULL; - - client_data = (MYSQL_session *) dcb->data; - service = (SERVICE *) dcb->service; - client = (struct sockaddr_in *) &dcb->ipv4; + MYSQL_session *client_data = (MYSQL_session *) dcb->data; + SERVICE *service = (SERVICE *) dcb->service; + SERV_LISTENER *listener = dcb->listener; + struct sockaddr_in *client = (struct sockaddr_in *) &dcb->ipv4; + MYSQL_USER_HOST key = {}; key.user = username; memcpy(&key.ipv4, client, sizeof(struct sockaddr_in)); key.netmask = 32; key.resource = client_data->db; + if (strlen(dcb->remote) < MYSQL_HOST_MAXLEN) { strcpy(key.hostname, dcb->remote); @@ -455,7 +453,7 @@ int gw_find_mysql_user_password_sha1(char *username, uint8_t *gateway_password, key.resource != NULL ? key.resource : ""); /* look for user@current_ipv4 now */ - user_password = mysql_users_fetch(service->users, &key); + char *user_password = mysql_users_fetch(listener->users, &key); if (!user_password) { @@ -482,7 +480,7 @@ int gw_find_mysql_user_password_sha1(char *username, uint8_t *gateway_password, key.ipv4.sin_addr.s_addr &= 0x00FFFFFF; key.netmask -= 8; - user_password = mysql_users_fetch(service->users, &key); + user_password = mysql_users_fetch(listener->users, &key); if (user_password) { @@ -493,7 +491,7 @@ int gw_find_mysql_user_password_sha1(char *username, uint8_t *gateway_password, key.ipv4.sin_addr.s_addr &= 0x0000FFFF; key.netmask -= 8; - user_password = mysql_users_fetch(service->users, &key); + user_password = mysql_users_fetch(listener->users, &key); if (user_password) { @@ -504,7 +502,7 @@ int gw_find_mysql_user_password_sha1(char *username, uint8_t *gateway_password, key.ipv4.sin_addr.s_addr &= 0x000000FF; key.netmask -= 8; - user_password = mysql_users_fetch(service->users, &key); + user_password = mysql_users_fetch(listener->users, &key); if (user_password) { @@ -524,7 +522,7 @@ int gw_find_mysql_user_password_sha1(char *username, uint8_t *gateway_password, key.user, dcb->remote); - user_password = mysql_users_fetch(service->users, &key); + user_password = mysql_users_fetch(listener->users, &key); if (user_password) { @@ -723,9 +721,9 @@ check_db_name_after_auth(DCB *dcb, char *database, int auth_ret) if (database && strlen(database)) { /* if database names are loaded we can check if db name exists */ - if (dcb->service->resources != NULL) + if (dcb->listener->resources != NULL) { - if (hashtable_fetch(dcb->service->resources, database)) + if (hashtable_fetch(dcb->listener->resources, database)) { db_exists = 1; } @@ -811,5 +809,5 @@ static int combined_auth_check( static void mysql_auth_free_client_data(DCB *dcb) { - free(dcb->data); + MXS_FREE(dcb->data); } diff --git a/server/modules/authenticator/null_auth.c b/server/modules/authenticator/null_auth_allow.c similarity index 97% rename from server/modules/authenticator/null_auth.c rename to server/modules/authenticator/null_auth_allow.c index 6819e1d06..b164481de 100644 --- a/server/modules/authenticator/null_auth.c +++ b/server/modules/authenticator/null_auth_allow.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -12,7 +12,7 @@ */ /** - * @file null_auth.c + * @file null_auth_allow.c * * Null Authentication module for handling the checking of clients credentials * for protocols that do not have authentication, either temporarily or @@ -22,6 +22,7 @@ * Revision History * Date Who Description * 14/03/2016 Martin Brampton Initial version + * 17/05/2016 Martin Brampton Renamed * * @endverbatim */ diff --git a/server/modules/authenticator/null_auth_deny.c b/server/modules/authenticator/null_auth_deny.c new file mode 100644 index 000000000..b97475cbc --- /dev/null +++ b/server/modules/authenticator/null_auth_deny.c @@ -0,0 +1,154 @@ +/* + * Copyright (c) 2016 MariaDB Corporation Ab + * + * Use of this software is governed by the Business Source License included + * in the LICENSE.TXT file and at www.mariadb.com/bsl. + * + * Change Date: 2019-07-01 + * + * On the date above, in accordance with the Business Source License, use + * of this software will be governed by version 2 or later of the General + * Public License. + */ + +/** + * @file null_auth_deny.c + * + * Null Authentication module for handling the checking of clients credentials + * for protocols that do not have authentication, either temporarily or + * permanently. Always fails the authentication. + * + * @verbatim + * Revision History + * Date Who Description + * 14/03/2016 Martin Brampton Initial version + * 17/05/2016 Martin Brampton Version to fail, instead of succeed. + * + * @endverbatim + */ + +#include +#include +#include +#include + +/* @see function load_module in load_utils.c for explanation of the following + * lint directives. + */ +/*lint -e14 */ +MODULE_INFO info = +{ + MODULE_API_AUTHENTICATOR, + MODULE_GA, + GWAUTHENTICATOR_VERSION, + "The Null client authenticator implementation" +}; +/*lint +e14 */ + +static char *version_str = "V1.0.0"; + +static int null_auth_set_protocol_data(DCB *dcb, GWBUF *buf); +static bool null_auth_is_client_ssl_capable(DCB *dcb); +static int null_auth_authenticate(DCB *dcb); +static void null_auth_free_client_data(DCB *dcb); + +/* + * The "module object" for mysql client authenticator module. + */ +static GWAUTHENTICATOR MyObject = +{ + null_auth_set_protocol_data, /* Extract data into structure */ + null_auth_is_client_ssl_capable, /* Check if client supports SSL */ + null_auth_authenticate, /* Authenticate user credentials */ + null_auth_free_client_data, /* Free the client data held in DCB */ +}; + +/** + * Implementation of the mandatory version entry point + * + * @return version string of the module + * + * @see function load_module in load_utils.c for explanation of the following + * lint directives. + */ +/*lint -e14 */ +char* version() +{ + return version_str; +} + +/** + * The module initialisation routine, called when the module + * is first loaded. + */ +void ModuleInit() +{ +} + +/** + * The module entry point routine. It is this routine that + * must populate the structure that is referred to as the + * "module object", this is a structure with the set of + * external entry points for this module. + * + * @return The module object + */ +GWAUTHENTICATOR* GetModuleObject() +{ + return &MyObject; +} +/*lint +e14 */ + +/** + * @brief Null authentication of a user. + * + * Always returns success + * + * @param dcb Request handler DCB connected to the client + * @return Authentication status - always 1 to denote failure + */ +static int +null_auth_authenticate(DCB *dcb) +{ + return 1; +} + +/** + * @brief Transfer data from the authentication request to the DCB. + * + * Does not actually transfer any data + * + * @param dcb Request handler DCB connected to the client + * @param buffer Pointer to pointer to buffer containing data from client + * @return Authentication status - always 0 to indicate success + */ +static int +null_auth_set_protocol_data(DCB *dcb, GWBUF *buf) +{ + return 0; +} + +/** + * @brief Determine whether the client is SSL capable + * + * Always say that client is SSL capable. The null authenticator cannot be + * used in a context where the client is not SSL capable. + * + * @param dcb Request handler DCB connected to the client + * @return Boolean indicating whether client is SSL capable - always true + */ +static bool +null_auth_is_client_ssl_capable(DCB *dcb) +{ + return true; +} + +/** + * @brief Free the client data pointed to by the passed DCB. + * + * The null authenticator does not allocate any data, so nothing to do. + * + * @param dcb Request handler DCB connected to the client + */ +static void +null_auth_free_client_data(DCB *dcb) {} diff --git a/server/modules/filter/CMakeLists.txt b/server/modules/filter/CMakeLists.txt index 0ec2b0884..831cddf65 100644 --- a/server/modules/filter/CMakeLists.txt +++ b/server/modules/filter/CMakeLists.txt @@ -1,64 +1,66 @@ -if(BUILD_RABBITMQ) - if(RABBITMQ_FOUND) - include_directories(${RABBITMQ_HEADERS}) - add_library(mqfilter SHARED mqfilter.c) - target_link_libraries(mqfilter maxscale-common ${RABBITMQ_LIBRARIES}) - add_dependencies(mqfilter pcre2) - install(TARGETS mqfilter DESTINATION ${MAXSCALE_LIBDIR}) - else() - message(WARNING "Could not find librabbitmq, the mqfilter will not be built.") - endif() +find_package(RabbitMQ) +if(RABBITMQ_FOUND) + include_directories(${RABBITMQ_HEADERS}) + add_library(mqfilter SHARED mqfilter.c) + target_link_libraries(mqfilter maxscale-common ${RABBITMQ_LIBRARIES}) + add_dependencies(mqfilter pcre2) + set_target_properties(mqfilter PROPERTIES VERSION "1.0.2") + install_module(mqfilter core) +else() + message(WARNING "Could not find librabbitmq, the mqfilter will not be built.") endif() add_library(regexfilter SHARED regexfilter.c) target_link_libraries(regexfilter maxscale-common) add_dependencies(regexfilter pcre2) set_target_properties(regexfilter PROPERTIES VERSION "1.1.0") -install(TARGETS regexfilter DESTINATION ${MAXSCALE_LIBDIR}) +install_module(regexfilter core) add_library(testfilter SHARED testfilter.c) target_link_libraries(testfilter maxscale-common) set_target_properties(testfilter PROPERTIES VERSION "1.0.0") -install(TARGETS testfilter DESTINATION ${MAXSCALE_LIBDIR}) +install_module(testfilter core) + +add_library(gatekeeper SHARED gatekeeper.c) +target_link_libraries(gatekeeper maxscale-common) +set_target_properties(gatekeeper PROPERTIES VERSION "1.0.0") +install_module(gatekeeper experimental) add_library(qlafilter SHARED qlafilter.c) target_link_libraries(qlafilter maxscale-common) set_target_properties(qlafilter PROPERTIES VERSION "1.1.1") -install(TARGETS qlafilter DESTINATION ${MAXSCALE_LIBDIR}) +install_module(qlafilter core) add_library(tee SHARED tee.c) target_link_libraries(tee maxscale-common) set_target_properties(tee PROPERTIES VERSION "1.0.0") -install(TARGETS tee DESTINATION ${MAXSCALE_LIBDIR}) +install_module(tee core) add_library(topfilter SHARED topfilter.c) target_link_libraries(topfilter maxscale-common) set_target_properties(topfilter PROPERTIES VERSION "1.0.1") -install(TARGETS topfilter DESTINATION ${MAXSCALE_LIBDIR}) +install_module(topfilter core) -if(BUILD_LUAFILTER) - find_package(Lua) - if(LUA_FOUND) - include_directories(${LUA_INCLUDE_DIR}) - add_library(luafilter SHARED luafilter.c) - target_link_libraries(luafilter maxscale-common ${LUA_LIBRARIES}) - install(TARGETS luafilter DESTINATION ${MAXSCALE_LIBDIR}) - else() - message(STATUS "Lua was not found, luafilter will not be built.") - endif() +find_package(Lua) +if(LUA_FOUND) + include_directories(${LUA_INCLUDE_DIR}) + add_library(luafilter SHARED luafilter.c) + set_target_properties(luafilter PROPERTIES VERSION "1.0.0") + target_link_libraries(luafilter maxscale-common ${LUA_LIBRARIES}) + install_module(luafilter experimental) +else() + message(STATUS "Lua was not found, luafilter will not be built.") endif() add_library(namedserverfilter SHARED namedserverfilter.c) target_link_libraries(namedserverfilter maxscale-common) set_target_properties(namedserverfilter PROPERTIES VERSION "1.1.0") -install(TARGETS namedserverfilter DESTINATION ${MAXSCALE_LIBDIR}) +install_module(namedserverfilter core) -if(BUILD_SLAVELAG) - add_library(slavelag SHARED slavelag.c) - target_link_libraries(slavelag maxscale-common) - set_target_properties(slavelag PROPERTIES VERSION "1.1.0") - install(TARGETS slavelag DESTINATION ${MAXSCALE_LIBDIR}) -endif() +add_library(ccrfilter SHARED ccrfilter.c) +target_link_libraries(ccrfilter maxscale-common) +set_target_properties(ccrfilter PROPERTIES VERSION "1.0.0") +install_module(ccrfilter experimental) add_subdirectory(hint) add_subdirectory(dbfwfilter) diff --git a/server/modules/filter/slavelag.c b/server/modules/filter/ccrfilter.c similarity index 82% rename from server/modules/filter/slavelag.c rename to server/modules/filter/ccrfilter.c index 23574a8b0..68d8d4831 100644 --- a/server/modules/filter/slavelag.c +++ b/server/modules/filter/ccrfilter.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -20,9 +20,10 @@ #include #include #include +#include /** - * @file slavelag.c - a very simple filter designed to send queries to the + * @file ccrfilter.c - a very simple filter designed to send queries to the * master server after data modification has occurred. This is done to prevent * replication lag affecting the outcome of a select query. * @@ -42,6 +43,7 @@ * * Date Who Description * 03/03/2015 Markus Mäkelä Written for demonstrative purposes + * 10/08/2016 Markus Mäkelä Cleaned up code and renamed to ccrfilter * @endverbatim */ @@ -77,6 +79,8 @@ static FILTER_OBJECT MyObject = diagnostic, }; +#define CCR_DEFAULT_TIME 60 + typedef struct lagstats { int n_add_count; /*< No. of statements diverted based on count */ @@ -99,7 +103,7 @@ typedef struct LAGSTATS stats; regex_t re; /* Compiled regex text of match */ regex_t nore; /* Compiled regex text of ignore */ -} LAG_INSTANCE; +} CCR_INSTANCE; /** * The session structure for this filter @@ -108,9 +112,8 @@ typedef struct { DOWNSTREAM down; /*< The downstream filter */ int hints_left; /*< Number of hints left to add to queries*/ - time_t last_modification; /*< Time of the last modifying operation */ - int active; /*< Is filter active */ -} LAG_SESSION; + time_t last_modification; /*< Time of the last data modifying operation */ +} CCR_SESSION; /** * Implementation of the mandatory version entry point @@ -161,14 +164,14 @@ GetModuleObject() static FILTER * createInstance(char **options, FILTER_PARAMETER **params) { - LAG_INSTANCE *my_instance; + CCR_INSTANCE *my_instance; int i; - int cflags = 0; + int cflags = REG_ICASE; - if ((my_instance = calloc(1, sizeof(LAG_INSTANCE))) != NULL) + if ((my_instance = MXS_CALLOC(1, sizeof(CCR_INSTANCE))) != NULL) { my_instance->count = 0; - my_instance->time = 0; + my_instance->time = CCR_DEFAULT_TIME; my_instance->stats.n_add_count = 0; my_instance->stats.n_add_time = 0; my_instance->stats.n_modified = 0; @@ -187,15 +190,15 @@ createInstance(char **options, FILTER_PARAMETER **params) } else if (!strcmp(params[i]->name, "match")) { - my_instance->match = strdup(params[i]->value); + my_instance->match = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "ignore")) { - my_instance->nomatch = strdup(params[i]->value); + my_instance->nomatch = MXS_STRDUP_A(params[i]->value); } - else + else if (!filter_standard_parameter(params[i]->name)) { - MXS_ERROR("lagfilter: Unexpected parameter '%s'.\n", params[i]->name); + MXS_ERROR("ccrfilter: Unexpected parameter '%s'.\n", params[i]->name); } } @@ -211,9 +214,13 @@ createInstance(char **options, FILTER_PARAMETER **params) { cflags &= ~REG_ICASE; } + else if (!strcasecmp(options[i], "extended")) + { + cflags |= REG_EXTENDED; + } else { - MXS_ERROR("lagfilter: unsupported option '%s'.", options[i]); + MXS_ERROR("ccrfilter: unsupported option '%s'.", options[i]); } } } @@ -222,7 +229,7 @@ createInstance(char **options, FILTER_PARAMETER **params) { if (regcomp(&my_instance->re, my_instance->match, cflags)) { - MXS_ERROR("lagfilter: Failed to compile regex '%s'.", my_instance->match); + MXS_ERROR("ccrfilter: Failed to compile regex '%s'.", my_instance->match); } } @@ -230,7 +237,7 @@ createInstance(char **options, FILTER_PARAMETER **params) { if (regcomp(&my_instance->nore, my_instance->nomatch, cflags)) { - MXS_ERROR("lagfilter: Failed to compile regex '%s'.", my_instance->nomatch); + MXS_ERROR("ccrfilter: Failed to compile regex '%s'.", my_instance->nomatch); } } } @@ -249,12 +256,11 @@ createInstance(char **options, FILTER_PARAMETER **params) static void * newSession(FILTER *instance, SESSION *session) { - LAG_INSTANCE *my_instance = (LAG_INSTANCE *)instance; - LAG_SESSION *my_session; + CCR_INSTANCE *my_instance = (CCR_INSTANCE *)instance; + CCR_SESSION *my_session = MXS_MALLOC(sizeof(CCR_SESSION)); - if ((my_session = malloc(sizeof(LAG_SESSION))) != NULL) + if (my_session) { - my_session->active = 1; my_session->hints_left = 0; my_session->last_modification = 0; } @@ -283,7 +289,7 @@ closeSession(FILTER *instance, void *session) static void freeSession(FILTER *instance, void *session) { - free(session); + MXS_FREE(session); } /** @@ -296,7 +302,7 @@ freeSession(FILTER *instance, void *session) static void setDownstream(FILTER *instance, void *session, DOWNSTREAM *downstream) { - LAG_SESSION *my_session = (LAG_SESSION *)session; + CCR_SESSION *my_session = (CCR_SESSION *)session; my_session->down = *downstream; } @@ -318,19 +324,23 @@ setDownstream(FILTER *instance, void *session, DOWNSTREAM *downstream) static int routeQuery(FILTER *instance, void *session, GWBUF *queue) { - LAG_INSTANCE *my_instance = (LAG_INSTANCE *)instance; - LAG_SESSION *my_session = (LAG_SESSION *)session; + CCR_INSTANCE *my_instance = (CCR_INSTANCE *)instance; + CCR_SESSION *my_session = (CCR_SESSION *)session; char *sql; time_t now = time(NULL); if (modutil_is_SQL(queue)) { - if (queue->next != NULL) + if (queue->next) { queue = gwbuf_make_contiguous(queue); } - if (qc_get_operation(queue) & (QUERY_OP_DELETE | QUERY_OP_INSERT | QUERY_OP_UPDATE)) + /** + * Not a simple SELECT statement, possibly modifies data. If we're processing a statement + * with unknown query type, the safest thing to do is to treat it as a data modifying statement. + */ + if ((qc_get_operation(queue) & ~QUERY_OP_SELECT) != 0) { if ((sql = modutil_get_SQL(queue)) != NULL) { @@ -346,7 +356,7 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) } } - free(sql); + MXS_FREE(sql); } } else if (my_session->hints_left > 0) @@ -381,12 +391,23 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) static void diagnostic(FILTER *instance, void *fsession, DCB *dcb) { - LAG_INSTANCE *my_instance = (LAG_INSTANCE *)instance; - LAG_SESSION *my_session = (LAG_SESSION *)fsession; + CCR_INSTANCE *my_instance = (CCR_INSTANCE *)instance; + CCR_SESSION *my_session = (CCR_SESSION *)fsession; dcb_printf(dcb, "Configuration:\n\tCount: %d\n", my_instance->count); - dcb_printf(dcb, "\tTime: %d seconds\n\n", my_instance->time); - dcb_printf(dcb, "Statistics:\n"); + dcb_printf(dcb, "\tTime: %d seconds\n", my_instance->time); + + if (my_instance->match) + { + dcb_printf(dcb, "\tMatch regex: %s\n", my_instance->match); + } + + if (my_instance->nomatch) + { + dcb_printf(dcb, "\tExclude regex: %s\n", my_instance->nomatch); + } + + dcb_printf(dcb, "\nStatistics:\n"); dcb_printf(dcb, "\tNo. of data modifications: %d\n", my_instance->stats.n_modified); dcb_printf(dcb, "\tNo. of hints added based on count: %d\n", my_instance->stats.n_add_count); dcb_printf(dcb, "\tNo. of hints added based on time: %d\n", my_instance->stats.n_add_time); diff --git a/server/modules/filter/dbfwfilter/CMakeLists.txt b/server/modules/filter/dbfwfilter/CMakeLists.txt index 665467c3a..ad1b91371 100644 --- a/server/modules/filter/dbfwfilter/CMakeLists.txt +++ b/server/modules/filter/dbfwfilter/CMakeLists.txt @@ -9,13 +9,13 @@ if(BISON_FOUND AND FLEX_FOUND) add_library(dbfwfilter SHARED dbfwfilter.c ${BISON_ruleparser_OUTPUTS} ${FLEX_token_OUTPUTS}) target_link_libraries(dbfwfilter maxscale-common) set_target_properties(dbfwfilter PROPERTIES VERSION "1.0.0") - install(TARGETS dbfwfilter DESTINATION ${MAXSCALE_LIBDIR}) + install_module(dbfwfilter core) if(BUILD_TOOLS) add_executable(dbfwruleparser dbfwfilter.c ${BISON_ruleparser_OUTPUTS} ${FLEX_token_OUTPUTS}) target_compile_definitions(dbfwruleparser PUBLIC "BUILD_RULE_PARSER") target_link_libraries(dbfwruleparser maxscale-common) - install(TARGETS dbfwruleparser DESTINATION ${MAXSCALE_BINDIR}) + install_module(dbfwruleparser core) endif() else() message(FATAL_ERROR "Could not find Bison or Flex: ${BISON_EXECUTABLE} ${FLEX_EXECUTABLE}") diff --git a/server/modules/filter/dbfwfilter/dbfwfilter.c b/server/modules/filter/dbfwfilter/dbfwfilter.c index 9e4ce22b8..c349f765d 100644 --- a/server/modules/filter/dbfwfilter/dbfwfilter.c +++ b/server/modules/filter/dbfwfilter/dbfwfilter.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -79,6 +79,7 @@ #include #include #include +#include /** Older versions of Bison don't include the parsing function in the header */ #ifndef dbfw_yyparse @@ -291,17 +292,16 @@ bool parse_limit_queries(FW_INSTANCE* instance, RULE* ruledef, const char* rule, */ static STRLINK* strlink_push(STRLINK* head, const char* value) { - STRLINK* link = malloc(sizeof(STRLINK)); + STRLINK* link = MXS_MALLOC(sizeof(STRLINK)); - if (link && (link->value = strdup(value))) + if (link && (link->value = MXS_STRDUP(value))) { link->next = head; } else { - free(link); + MXS_FREE(link); link = NULL; - MXS_ERROR("dbfwfilter: Memory allocation failed."); } return link; } @@ -316,8 +316,8 @@ static STRLINK* strlink_pop(STRLINK* head) if (head) { STRLINK* next = head->next; - free(head->value); - free(head); + MXS_FREE(head->value); + MXS_FREE(head); return next; } return NULL; @@ -333,8 +333,8 @@ static void strlink_free(STRLINK* head) { STRLINK* tmp = head; head = head->next; - free(tmp->value); - free(tmp); + MXS_FREE(tmp->value); + MXS_FREE(tmp); } } @@ -366,7 +366,7 @@ static STRLINK* strlink_reverse_clone(STRLINK* head) static RULELIST* rulelist_push(RULELIST *head, RULE *rule) { - RULELIST *rval = malloc(sizeof(RULELIST)); + RULELIST *rval = MXS_MALLOC(sizeof(RULELIST)); if (rval) { @@ -385,7 +385,8 @@ static void* rulelist_clone(void* fval) while (ptr) { - RULELIST* tmp = (RULELIST*) malloc(sizeof(RULELIST)); + RULELIST* tmp = (RULELIST*) MXS_MALLOC(sizeof(RULELIST)); + MXS_ABORT_IF_NULL(tmp); tmp->next = rule; tmp->rule = ptr->rule; rule = tmp; @@ -402,22 +403,21 @@ static void* rulelist_free(void* fval) { RULELIST *tmp = ptr; ptr = ptr->next; - free(tmp); + MXS_FREE(tmp); } return NULL; } -static void* huserfree(void* fval) +static void huserfree(void* fval) { USER* value = (USER*) fval; rulelist_free(value->rules_and); rulelist_free(value->rules_or); rulelist_free(value->rules_strict_and); - free(value->qs_limit); - free(value->name); - free(value); - return NULL; + MXS_FREE(value->qs_limit); + MXS_FREE(value->name); + MXS_FREE(value); } /** @@ -622,7 +622,7 @@ static TIMERANGE* parse_time(const char* str) CHK_TIMES((&start)); CHK_TIMES((&end)); - tr = (TIMERANGE*) malloc(sizeof(TIMERANGE)); + tr = (TIMERANGE*) MXS_MALLOC(sizeof(TIMERANGE)); if (tr) { @@ -630,10 +630,6 @@ static TIMERANGE* parse_time(const char* str) tr->end = end; tr->next = NULL; } - else - { - MXS_ERROR("dbfwfilter: malloc returned NULL."); - } } } return tr; @@ -649,7 +645,8 @@ TIMERANGE* split_reverse_time(TIMERANGE* tr) { TIMERANGE* tmp = NULL; - tmp = (TIMERANGE*) calloc(1, sizeof(TIMERANGE)); + tmp = (TIMERANGE*) MXS_CALLOC(1, sizeof(TIMERANGE)); + MXS_ABORT_IF_NULL(tmp); tmp->next = tr; tmp->start.tm_hour = 0; tmp->start.tm_min = 0; @@ -724,15 +721,14 @@ static bool apply_rule_to_user(FW_INSTANCE *instance, char *username, if ((user = (USER*) hashtable_fetch(instance->htable, username)) == NULL) { /**New user*/ - if ((user = (USER*) calloc(1, sizeof(USER))) == NULL) + if ((user = (USER*) MXS_CALLOC(1, sizeof(USER))) == NULL) { - MXS_ERROR("dbfwfilter: failed to allocate memory when parsing rules."); return false; } spinlock_init(&user->lock); } - user->name = (char*) strdup(username); + user->name = (char*) MXS_STRDUP_A(username); user->qs_limit = NULL; RULELIST *tl = (RULELIST*) rulelist_clone(rulelist); RULELIST *tail = tl; @@ -775,7 +771,7 @@ void tr_free(TIMERANGE* tr) { tmp = node; node = node->next; - free(tmp); + MXS_FREE(tmp); } } @@ -874,9 +870,9 @@ void dbfw_yyerror(void* scanner, const char* error) bool create_rule(void* scanner, const char* name) { bool rval = true; - RULE *ruledef = malloc(sizeof(RULE)); + RULE *ruledef = MXS_MALLOC(sizeof(RULE)); - if (ruledef && (ruledef->name = strdup(name))) + if (ruledef && (ruledef->name = MXS_STRDUP(name))) { ruledef->type = RT_UNDEFINED; ruledef->on_queries = QUERY_OP_UNDEFINED; @@ -890,8 +886,7 @@ bool create_rule(void* scanner, const char* name) } else { - MXS_ERROR("Memory allocation failed when creating rule '%s'.", name); - free(ruledef); + MXS_FREE(ruledef); rval = false; } @@ -911,7 +906,7 @@ static void free_rules(RULE* rule) { TIMERANGE *tr = rule->active; rule->active = rule->active->next; - free(tr); + MXS_FREE(tr); } switch (rule->type) @@ -921,7 +916,7 @@ static void free_rules(RULE* rule) break; case RT_THROTTLE: - free(rule->data); + MXS_FREE(rule->data); break; case RT_REGEX: @@ -932,7 +927,7 @@ static void free_rules(RULE* rule) break; } - free(rule->name); + MXS_FREE(rule->name); rule = tmp; } } @@ -1021,9 +1016,9 @@ bool create_user_templates(void* scanner) while (user) { - user_template_t* newtemp = malloc(sizeof(user_template_t)); + user_template_t* newtemp = MXS_MALLOC(sizeof(user_template_t)); STRLINK* tmp; - if (newtemp && (newtemp->name = strdup(user->value)) && + if (newtemp && (newtemp->name = MXS_STRDUP(user->value)) && (newtemp->rulenames = strlink_reverse_clone(rstack->active_rules))) { newtemp->type = rstack->active_mode; @@ -1034,13 +1029,12 @@ bool create_user_templates(void* scanner) { if (newtemp) { - free(newtemp->name); - free(newtemp); + MXS_FREE(newtemp->name); + MXS_FREE(newtemp); } - free(templates->name); + MXS_FREE(templates->name); strlink_free(templates->rulenames); - free(templates); - MXS_ERROR("Memory allocation failed when processing rule file users definitions."); + MXS_FREE(templates); return false; } user = user->next; @@ -1063,8 +1057,8 @@ void free_user_templates(user_template_t *templates) user_template_t *tmp = templates; templates = templates->next; strlink_free(tmp->rulenames); - free(tmp->name); - free(tmp); + MXS_FREE(tmp->name); + MXS_FREE(tmp); } } @@ -1143,7 +1137,7 @@ bool define_limit_queries_rule(void* scanner, int max, int timeperiod, int holdo { struct parser_stack* rstack = dbfw_yyget_extra((yyscan_t) scanner); ss_dassert(rstack); - QUERYSPEED* qs = malloc(sizeof(QUERYSPEED)); + QUERYSPEED* qs = MXS_MALLOC(sizeof(QUERYSPEED)); if (qs) { @@ -1153,10 +1147,6 @@ bool define_limit_queries_rule(void* scanner, int max, int timeperiod, int holdo rstack->rule->type = RT_THROTTLE; rstack->rule->data = qs; } - else - { - MXS_ERROR("dbfwfilter: Memory allocation failed when adding limit_queries rule."); - } return qs != NULL; } @@ -1239,7 +1229,7 @@ static bool process_user_templates(FW_INSTANCE *instance, user_template_t *templ if (user == NULL) { - if ((user = malloc(sizeof(USER))) && (user->name = strdup(templates->name))) + if ((user = MXS_MALLOC(sizeof(USER))) && (user->name = MXS_STRDUP(templates->name))) { user->rules_and = NULL; user->rules_or = NULL; @@ -1249,11 +1239,9 @@ static bool process_user_templates(FW_INSTANCE *instance, user_template_t *templ } else { - free(user); + MXS_FREE(user); rval = false; break; - MXS_ERROR("Memory allocation failed when creating user '%s'.", - templates->name); } } @@ -1382,23 +1370,22 @@ createInstance(char **options, FILTER_PARAMETER **params) char *filename = NULL; bool err = false; - if ((my_instance = calloc(1, sizeof(FW_INSTANCE))) == NULL) + if ((my_instance = MXS_CALLOC(1, sizeof(FW_INSTANCE))) == NULL) { - free(my_instance); - MXS_ERROR("Memory allocation for firewall filter failed."); + MXS_FREE(my_instance); return NULL; } spinlock_init(&my_instance->lock); - if ((ht = hashtable_alloc(100, simple_str_hash, strcmp)) == NULL) + if ((ht = hashtable_alloc(100, hashtable_item_strhash, hashtable_item_strcmp)) == NULL) { MXS_ERROR("Unable to allocate hashtable."); - free(my_instance); + MXS_FREE(my_instance); return NULL; } - hashtable_memory_fns(ht, (HASHMEMORYFN) strdup, NULL, (HASHMEMORYFN) free, huserfree); + hashtable_memory_fns(ht, hashtable_item_strdup, NULL, hashtable_item_free, huserfree); my_instance->htable = ht; my_instance->action = FW_ACTION_BLOCK; @@ -1459,7 +1446,7 @@ createInstance(char **options, FILTER_PARAMETER **params) if (err || !process_rule_file(filename, my_instance)) { hashtable_free(my_instance->htable); - free(my_instance); + MXS_FREE(my_instance); my_instance = NULL; } @@ -1478,7 +1465,7 @@ newSession(FILTER *instance, SESSION *session) { FW_SESSION *my_session; - if ((my_session = calloc(1, sizeof(FW_SESSION))) == NULL) + if ((my_session = MXS_CALLOC(1, sizeof(FW_SESSION))) == NULL) { return NULL; } @@ -1510,9 +1497,9 @@ freeSession(FILTER *instance, void *session) FW_SESSION *my_session = (FW_SESSION *) session; if (my_session->errmsg) { - free(my_session->errmsg); + MXS_FREE(my_session->errmsg); } - free(my_session); + MXS_FREE(my_session); } /** @@ -1555,11 +1542,10 @@ GWBUF* gen_dummy_error(FW_SESSION* session, char* msg) dcb = session->session->client_dcb; mysql_session = (MYSQL_session*) dcb->data; errlen = msg != NULL ? strlen(msg) : 0; - errmsg = (char*) malloc((512 + errlen) * sizeof(char)); + errmsg = (char*) MXS_MALLOC((512 + errlen) * sizeof(char)); if (errmsg == NULL) { - MXS_ERROR("Memory allocation failed."); return NULL; } @@ -1582,7 +1568,7 @@ GWBUF* gen_dummy_error(FW_SESSION* session, char* msg) } buf = modutil_create_mysql_err_msg(1, 0, 1141, "HY000", (const char*) errmsg); - free(errmsg); + MXS_FREE(errmsg); return buf; } @@ -1679,7 +1665,7 @@ static char* create_parse_error(FW_INSTANCE* my_instance, { char msgbuf[len + 1]; // +1 for the "." sprintf(msgbuf, "%s.", message); - msg = strdup(msgbuf); + msg = MXS_STRDUP_A(msgbuf); if (my_instance->action == FW_ACTION_ALLOW) { @@ -1799,7 +1785,7 @@ bool rule_matches(FW_INSTANCE* my_instance, pcre2_match_data_free(mdata); if (matches) { - msg = strdup("Permission denied, query matched regular expression."); + msg = MXS_STRDUP_A("Permission denied, query matched regular expression."); MXS_INFO("dbfwfilter: rule '%s': regex matched on query", rulelist->rule->name); goto queryresolved; } @@ -1815,7 +1801,7 @@ bool rule_matches(FW_INSTANCE* my_instance, case RT_PERMISSION: { matches = true; - msg = strdup("Permission denied at this time."); + msg = MXS_STRDUP_A("Permission denied at this time."); char buffer[32]; // asctime documentation requires 26 asctime_r(&tm_now, buffer); MXS_INFO("dbfwfilter: rule '%s': query denied at: %s", rulelist->rule->name, buffer); @@ -1843,15 +1829,15 @@ bool rule_matches(FW_INSTANCE* my_instance, sprintf(emsg, "Permission denied to column '%s'.", strln->value); MXS_INFO("dbfwfilter: rule '%s': query targets forbidden column: %s", rulelist->rule->name, strln->value); - msg = strdup(emsg); - free(where); + msg = MXS_STRDUP_A(emsg); + MXS_FREE(where); goto queryresolved; } strln = strln->next; } tok = strtok_r(NULL, ",", &saveptr); } - free(where); + MXS_FREE(where); } } break; @@ -1869,13 +1855,13 @@ bool rule_matches(FW_INSTANCE* my_instance, if (strchr(strptr, '*')) { matches = true; - msg = strdup("Usage of wildcard denied."); + msg = MXS_STRDUP_A("Usage of wildcard denied."); MXS_INFO("dbfwfilter: rule '%s': query contains a wildcard.", rulelist->rule->name); - free(where); + MXS_FREE(where); goto queryresolved; } - free(where); + MXS_FREE(where); } } break; @@ -1906,7 +1892,8 @@ bool rule_matches(FW_INSTANCE* my_instance, { /**No match found*/ - queryspeed = (QUERYSPEED*) calloc(1, sizeof(QUERYSPEED)); + queryspeed = (QUERYSPEED*) MXS_CALLOC(1, sizeof(QUERYSPEED)); + MXS_ABORT_IF_NULL(queryspeed); queryspeed->period = rule_qs->period; queryspeed->cooldown = rule_qs->cooldown; queryspeed->limit = rule_qs->limit; @@ -1926,7 +1913,7 @@ bool rule_matches(FW_INSTANCE* my_instance, sprintf(emsg, "Queries denied for %f seconds", blocked_for); MXS_INFO("dbfwfilter: rule '%s': user denied for %f seconds", rulelist->rule->name, blocked_for); - msg = strdup(emsg); + msg = MXS_STRDUP_A(emsg); matches = true; } else @@ -1952,7 +1939,7 @@ bool rule_matches(FW_INSTANCE* my_instance, double blocked_for = queryspeed->cooldown - difftime(time_now, queryspeed->triggered); sprintf(emsg, "Queries denied for %f seconds", blocked_for); - msg = strdup(emsg); + msg = MXS_STRDUP_A(emsg); } else if (queryspeed->count > 0 && difftime(time_now, queryspeed->first_query) <= queryspeed->period) @@ -1972,7 +1959,7 @@ bool rule_matches(FW_INSTANCE* my_instance, !qc_query_has_clause(queue)) { matches = true; - msg = strdup("Required WHERE/HAVING clause is missing."); + msg = MXS_STRDUP_A("Required WHERE/HAVING clause is missing."); MXS_INFO("dbfwfilter: rule '%s': query has no where/having " "clause, query is denied.", rulelist->rule->name); } @@ -1989,7 +1976,7 @@ queryresolved: { if (my_session->errmsg) { - free(my_session->errmsg); + MXS_FREE(my_session->errmsg); } my_session->errmsg = msg; @@ -2031,14 +2018,14 @@ bool check_match_any(FW_INSTANCE* my_instance, FW_SESSION* my_session, } if (rule_matches(my_instance, my_session, queue, user, rulelist, fullquery)) { - *rulename = strdup(rulelist->rule->name); + *rulename = MXS_STRDUP_A(rulelist->rule->name); rval = true; break; } rulelist = rulelist->next; } - free(fullquery); + MXS_FREE(fullquery); } return rval; } @@ -2054,7 +2041,7 @@ void append_string(char** dest, size_t* size, const char* src) { if (*dest == NULL) { - *dest = strdup(src); + *dest = MXS_STRDUP_A(src); *size = strlen(src); } else @@ -2062,7 +2049,7 @@ void append_string(char** dest, size_t* size, const char* src) if (*size < strlen(*dest) + strlen(src) + 3) { size_t newsize = strlen(*dest) + strlen(src) + 3; - char* tmp = realloc(*dest, newsize); + char* tmp = MXS_REALLOC(*dest, newsize); if (tmp) { *size = newsize; @@ -2070,7 +2057,6 @@ void append_string(char** dest, size_t* size, const char* src) } else { - MXS_ERROR("Memory allocation failed"); return; } } @@ -2131,7 +2117,7 @@ bool check_match_all(FW_INSTANCE* my_instance, FW_SESSION* my_session, /** No active rules */ rval = false; } - free(fullquery); + MXS_FREE(fullquery); } /** Set the list of matched rule names */ @@ -2198,7 +2184,7 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) GWBUF* err = gen_dummy_error(my_session, "This filter does not support " "multi-statements."); gwbuf_free(queue); - free(my_session->errmsg); + MXS_FREE(my_session->errmsg); my_session->errmsg = NULL; rval = dcb->func.write(dcb, err); } @@ -2268,7 +2254,7 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) } } - free(rname); + MXS_FREE(rname); } /** If the instance is in whitelist mode, only users that have a rule * defined for them are allowed */ @@ -2286,7 +2272,7 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) { GWBUF* forward = gen_dummy_error(my_session, my_session->errmsg); gwbuf_free(queue); - free(my_session->errmsg); + MXS_FREE(my_session->errmsg); my_session->errmsg = NULL; rval = dcb->func.write(dcb, forward); } @@ -2378,10 +2364,11 @@ int main(int argc, char** argv) return 1; } - home = malloc(sizeof(char) * (PATH_MAX + 1)); + home = MXS_MALLOC(sizeof(char) * (PATH_MAX + 1)); + MXS_ABORT_IF_NULL(home); if (getcwd(home, PATH_MAX) == NULL) { - free(home); + MXS_FREE(home); home = NULL; } @@ -2399,8 +2386,8 @@ int main(int argc, char** argv) init_test_env(home); - ruleparam.name = strdup("rules"); - ruleparam.value = strdup(argv[1]); + ruleparam.name = MXS_STRDUP_A("rules"); + ruleparam.value = MXS_STRDUP_A(argv[1]); paramlist[0] = &ruleparam; paramlist[1] = NULL; diff --git a/server/modules/filter/dbfwfilter/dbfwfilter.h b/server/modules/filter/dbfwfilter/dbfwfilter.h index 2ebe748f3..9e828e361 100644 --- a/server/modules/filter/dbfwfilter/dbfwfilter.h +++ b/server/modules/filter/dbfwfilter/dbfwfilter.h @@ -6,7 +6,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/modules/filter/dbfwfilter/ruleparser.y b/server/modules/filter/dbfwfilter/ruleparser.y index 38f20221d..96dce952b 100644 --- a/server/modules/filter/dbfwfilter/ruleparser.y +++ b/server/modules/filter/dbfwfilter/ruleparser.y @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/modules/filter/dbfwfilter/token.l b/server/modules/filter/dbfwfilter/token.l index 869fe4743..f03303ce0 100644 --- a/server/modules/filter/dbfwfilter/token.l +++ b/server/modules/filter/dbfwfilter/token.l @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General diff --git a/server/modules/filter/gatekeeper.c b/server/modules/filter/gatekeeper.c new file mode 100644 index 000000000..daf61666b --- /dev/null +++ b/server/modules/filter/gatekeeper.c @@ -0,0 +1,587 @@ +/* + * Copyright (c) 2016 MariaDB Corporation Ab + * + * Use of this software is governed by the Business Source License included + * in the LICENSE.TXT file and at www.mariadb.com/bsl. + * + * Change Date: 2019-07-01 + * + * On the date above, in accordance with the Business Source License, use + * of this software will be governed by version 2 or later of the General + * Public License. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define GK_DEFAULT_HASHTABLE_SIZE 1000 + +/** + * @file gatekeeper.c - A learning firewall + * + * This filter will learn from input data read during a learning phase. + * After learning the characteristics of the input, the filter can then + * be set into an enforcing mode. In this mode the filter will block any + * queries that do not conform to the training set. + */ + +MODULE_INFO info = +{ + MODULE_API_FILTER, + MODULE_ALPHA_RELEASE, + FILTER_VERSION, + "Learning firewall filter" +}; + +enum firewall_mode +{ + ENFORCE, + LEARN +}; + +typedef struct +{ + unsigned long queries; /**< Number of queries received */ + unsigned int hit; /**< Number of queries that match a pattern */ + unsigned int miss; /**< Number of queries that do not match a pattern */ + unsigned int entries; /**< Number of new patterns created */ +} GK_STATS; + +typedef struct +{ + HASHTABLE *queryhash; /**< Canonicalized forms of the queries */ + char* datadir; /**< The data is stored in this directory as lfw.data */ + enum firewall_mode mode; /**< Filter mode, either ENFORCE or LEARN */ + GK_STATS stats; /**< Instance statistics */ + SPINLOCK lock; /**< Instance lock */ + bool updating; /**< If the datafile is being updated */ + bool need_update; /**< If the datafile needs updating */ +} GK_INSTANCE; + +typedef struct +{ + DCB* dcb; /**< Client DCB, used to send error messages */ + DOWNSTREAM down; + GK_STATS stats; /**< Session statistics */ +} GK_SESSION; + +static char *version_str = "V1.0.0"; +static const char* datafile_name = "gatekeeper.data"; + +/** Prefix all log messages with this tag */ +#define MODNAME "[gatekeeper] " + +/** This is passed as a value to @c hashtable_add to have @c hashtable_fetch + * return a non-NULL value when a hash hit is made */ +static bool trueval = true; + +static FILTER *createInstance(char **options, FILTER_PARAMETER **params); +static void *newSession(FILTER *instance, SESSION *session); +static void closeSession(FILTER *instance, void *session); +static void freeSession(FILTER *instance, void *session); +static void setDownstream(FILTER *instance, void *fsession, DOWNSTREAM *downstream); +static int routeQuery(FILTER *instance, void *fsession, GWBUF *queue); +static void diagnostic(FILTER *instance, void *fsession, DCB *dcb); +static bool read_stored_data(GK_INSTANCE *inst); +static bool write_stored_data(GK_INSTANCE *inst); + +static FILTER_OBJECT MyObject = +{ + createInstance, + newSession, + closeSession, + freeSession, + setDownstream, + NULL, // No upstream requirement + routeQuery, + NULL, + diagnostic, +}; + +/** + * Implementation of the mandatory version entry point + * + * @return version string of the module + */ +char* version() +{ + return version_str; +} + +/** + * The module initialization routine, called when the module + * is first loaded. + */ +void ModuleInit() +{ +} + +/** + * The module entry point routine. It is this routine that + * must populate the structure that is referred to as the + * "module object", this is a structure with the set of + * external entry points for this module. + * + * @return The module object + */ +FILTER_OBJECT* GetModuleObject() +{ + return &MyObject; +} + +/** + * Create an instance of the filter for a particular service + * within MaxScale. + * + * @param options The options for this filter + * @param params The array of name/value pair parameters for the filter + * + * @return The instance data for this new instance + */ +static FILTER* createInstance(char **options, FILTER_PARAMETER **params) +{ + GK_INSTANCE *inst = MXS_CALLOC(1, sizeof(GK_INSTANCE)); + + if (inst) + { + const char* datadir = get_datadir(); + bool ok = true; + spinlock_init(&inst->lock); + inst->mode = LEARN; + + for (int i = 0; params && params[i]; i++) + { + if (strcmp(params[i]->name, "mode") == 0) + { + if (strcasecmp(params[i]->value, "enforce") == 0) + { + inst->mode = ENFORCE; + } + else if (strcasecmp(params[i]->value, "learn") == 0) + { + inst->mode = LEARN; + } + else + { + MXS_ERROR(MODNAME"Unknown value for 'mode': %s", params[i]->value); + ok = false; + } + } + else if (strcmp(params[i]->name, "datadir") == 0) + { + if (access(params[i]->value, F_OK) == 0) + { + datadir = params[i]->value; + } + else + { + char err[STRERROR_BUFLEN]; + MXS_ERROR(MODNAME"File is not accessible: %d, %s", errno, + strerror_r(errno, err, sizeof(err))); + ok = false; + } + } + else if (!filter_standard_parameter(params[i]->name)) + { + MXS_ERROR(MODNAME"Unknown parameter '%s'.", params[i]->name); + ok = false; + } + } + + if (ok) + { + inst->queryhash = hashtable_alloc(GK_DEFAULT_HASHTABLE_SIZE, + hashtable_item_strhash, + hashtable_item_strcasecmp); + inst->datadir = MXS_STRDUP(datadir); + if (inst->queryhash && inst->datadir) + { + hashtable_memory_fns(inst->queryhash, NULL, NULL, hashtable_item_free, NULL); + if (read_stored_data(inst)) + { + MXS_NOTICE(MODNAME"Started in [%s] mode. Data is stored at: %s", + inst->mode == ENFORCE ? "ENFORCE" : "LEARN", inst->datadir); + } + else + { + ok = false; + } + } + else + { + ok = false; + } + } + + if (!ok) + { + hashtable_free(inst->queryhash); + MXS_FREE(inst->datadir); + MXS_FREE(inst); + inst = NULL; + } + } + + return (FILTER*)inst; +} + +/** + * Associate a new session with this instance of the filter. + * + * @param instance The filter instance data + * @param session The session itself + * @return Session specific data for this session + */ +static void* newSession(FILTER *instance, SESSION *session) +{ + GK_INSTANCE *inst = (GK_INSTANCE *) instance; + GK_SESSION *ses; + + if ((ses = MXS_CALLOC(1, sizeof(GK_SESSION))) != NULL) + { + ses->dcb = session->client_dcb; + } + + return ses; +} + +/** + * Close a session with the filter, this is the mechanism + * by which a filter may cleanup data structure etc. + * + * The gatekeeper flushes the hashtable to disk every time a session is closed. + * + * @param instance The filter instance data + * @param session The session being closed + */ +static void closeSession(FILTER *instance, void *session) +{ + GK_INSTANCE *inst = (GK_INSTANCE*) instance; + GK_SESSION *ses = (GK_SESSION *) session; + + /** If we added new data into the queryhash, update the datafile on disk */ + if (ses->stats.entries > 0) + { + spinlock_acquire(&inst->lock); + + bool update = !inst->updating; + if (update) + { + inst->updating = true; + inst->need_update = false; + } + else + { + /** If another thread is already updating the file, set + * the need_update flag */ + inst->need_update = true; + } + + spinlock_release(&inst->lock); + + while (update) + { + /** Store the hashtable to disk */ + write_stored_data(inst); + + spinlock_acquire(&inst->lock); + + /** Check if the hashtable has been update while we were writing + * the data to disk. */ + if ((update = inst->need_update)) + { + inst->need_update = false; + } + else + { + inst->updating = false; + } + + spinlock_release(&inst->lock); + } + } + + /** Add session stats to instance stats */ + spinlock_acquire(&inst->lock); + inst->stats.entries += ses->stats.entries; + inst->stats.hit += ses->stats.hit; + inst->stats.miss += ses->stats.miss; + inst->stats.queries += ses->stats.queries; + spinlock_release(&inst->lock); +} + +/** + * Free the memory associated with this filter session. + * + * @param instance The filter instance data + * @param session The session being closed + */ +static void freeSession(FILTER *instance, void *session) +{ + MXS_FREE(session); +} + +/** + * Set the downstream component for this filter. + * + * @param instance The filter instance data + * @param session The session being closed + * @param downstream The downstream filter or router + */ +static void setDownstream(FILTER *instance, void *session, DOWNSTREAM *downstream) +{ + GK_SESSION *ses = (GK_SESSION *) session; + ses->down = *downstream; +} + +/** + * @brief Main routing function + * + * @param instance The filter instance data + * @param session The filter session + * @param queue The query data + * @return 1 on success, 0 on error + */ +static int routeQuery(FILTER *instance, void *session, GWBUF *queue) +{ + GK_INSTANCE *inst = (GK_INSTANCE*) instance; + GK_SESSION *ses = (GK_SESSION *) session; + int rval = 1; + bool ok = true; + char* canon = qc_get_canonical(queue); + + ses->stats.queries++; + + /** Non-COM_QUERY packets are better handled on the backend database. For + * example a COM_INIT_DB does not get canonicalized and would be always + * denied. For this reason, queries that are not canonicalized are allowed. + * This means that the binary protocol and prepared statements are not + * processed by this filter. */ + if (canon) + { + if (inst->mode == ENFORCE) + { + if (hashtable_fetch(inst->queryhash, canon)) + { + ses->stats.hit++; + } + else + { + ses->stats.miss++; + ok = false; + MXS_WARNING(MODNAME"Query by %s@%s was not found from queryhash: %s", + ses->dcb->user, ses->dcb->remote, canon); + GWBUF* errbuf = modutil_create_mysql_err_msg(1, 0, 1, "00000", "Permission denied."); + rval = errbuf ? ses->dcb->func.write(ses->dcb, errbuf) : 0; + } + MXS_FREE(canon); + } + else if (inst->mode == LEARN) + { + if (hashtable_add(inst->queryhash, canon, &trueval)) + { + ses->stats.entries++; + } + else + { + MXS_FREE(canon); + } + } + } + + if (ok) + { + rval = ses->down.routeQuery(ses->down.instance, ses->down.session, queue); + } + + return rval; +} + +/** + * @brief Diagnostics routine + * + * @param instance The filter instance + * @param fsession Filter session + * @param dcb The DCB for output + */ +static void diagnostic(FILTER *instance, void *fsession, DCB *dcb) +{ + GK_INSTANCE *inst = (GK_INSTANCE *) instance; + + dcb_printf(dcb, "\t\tQueries: %lu\n", inst->stats.queries); + dcb_printf(dcb, "\t\tQueryhash entries: %u\n", inst->stats.entries); + dcb_printf(dcb, "\t\tQueryhash hits: %u\n", inst->stats.hit); + dcb_printf(dcb, "\t\tQueryhash misses: %u\n", inst->stats.miss); +} + +/** + * @brief Write query patterns from memory to disk + * + * The data is stored as length-encoded strings. A length-encoded string contains + * a 4 byte integer, which tells the length of the string, followed by the string + * itself. The stored file will consist of multiple consecutive length-encoded strings. + * + * @param inst Filter instance + * @return True on success + */ +static bool write_stored_data(GK_INSTANCE *inst) +{ + bool rval = false; + char filepath[PATH_MAX]; + sprintf(filepath, "%s/%s.tmp.XXXXXX", inst->datadir, datafile_name); + int fd = mkstemp(filepath); + HASHITERATOR *iter = hashtable_iterator(inst->queryhash); + + if (fd > 0 && iter) + { + char *key; + bool ok = true; + + while ((key = hashtable_next(iter))) + { + uint32_t len = strlen(key); + + /** First write the length of the string and then the string itself */ + if (write(fd, &len, sizeof(len)) != sizeof(len) || + write(fd, key, len) != len) + { + char err[STRERROR_BUFLEN]; + MXS_ERROR(MODNAME"Failed to write key '%s' to disk (%d, %s). The datafile at '%s' was " + "not updated but it will be updated when the next session closes. ", + key, errno, strerror_r(errno, err, sizeof(err)), inst->datadir); + ok = false; + break; + } + } + + if (ok) + { + /** Update the file by renaming the temporary file to the original one*/ + char newfilepath[PATH_MAX + 1]; + snprintf(newfilepath, sizeof(newfilepath), "%s/%s", inst->datadir, datafile_name); + rval = rename(filepath, newfilepath) == 0; + + if (!rval) + { + char err[STRERROR_BUFLEN]; + MXS_ERROR(MODNAME"Failed to rename file '%s' to '%s' when writing data: %d, %s", + filepath, newfilepath, errno, strerror_r(errno, err, sizeof(err))); + } + } + } + else if (fd == -1) + { + char err[STRERROR_BUFLEN]; + MXS_ERROR(MODNAME"Failed to open file '%s' when writing data: %d, %s", + filepath, errno, strerror_r(errno, err, sizeof(err))); + } + + if (fd > 0) + { + close(fd); + } + hashtable_iterator_free(iter); + + return rval; +} + +static void report_failed_read(FILE *file, int nexpected, int nread) +{ + if (ferror(file)) + { + char err[STRERROR_BUFLEN]; + MXS_ERROR(MODNAME"Failed to read %d bytes, only %d bytes read: %d, %s", + nexpected, nread, errno, strerror_r(errno, err, sizeof(err))); + } + else + { + MXS_ERROR(MODNAME"Partial read, expected %d bytes but read only %d.", + nexpected, nread); + } +} + +/** + * @brief Read query patterns from disk to memory + * + * See write_stored_data() for details on how the data is stored. + * + * @param inst Filter instance + * @return True if data was successfully read + */ +static bool read_stored_data(GK_INSTANCE *inst) +{ + char filepath[PATH_MAX + 1]; + snprintf(filepath, sizeof(filepath), "%s/%s", inst->datadir, datafile_name); + + if (access(filepath, F_OK) != 0) + { + if (inst->mode == ENFORCE) + { + MXS_ERROR(MODNAME"Started in ENFORCE mode but no datafile was found at '%s'.", filepath); + return false; + } + + /** Not finding a datafile in LEARN mode is OK since it will be created later on */ + return true; + } + + bool rval = true; + FILE *file = fopen(filepath, "rb"); + + if (file) + { + do + { + uint32_t len; + size_t nread; + char *data; + + /** Read the length of the string */ + if ((nread = fread(&len, 1, sizeof(len), file)) != sizeof(len)) + { + if (nread > 0 || !feof(file)) + { + report_failed_read(file, sizeof(len), nread); + rval = false; + } + break; + } + + if ((data = MXS_MALLOC(len + 1)) == NULL) + { + rval = false; + break; + } + + /** Read the string itself */ + if ((nread = fread(data, 1, len, file)) != len) + { + MXS_FREE(data); + report_failed_read(file, sizeof(len), nread); + rval = false; + break; + } + + data[len] = '\0'; + hashtable_add(inst->queryhash, data, &trueval); + } + while (!feof(file)); + + fclose(file); + } + else + { + char err[STRERROR_BUFLEN]; + MXS_ERROR(MODNAME"Failed to open file '%s' when reading stored data: %d, %s", + filepath, errno, strerror_r(errno, err, sizeof(err))); + } + + return rval; +} diff --git a/server/modules/filter/hint/CMakeLists.txt b/server/modules/filter/hint/CMakeLists.txt index 7efa8492d..7bd06705b 100644 --- a/server/modules/filter/hint/CMakeLists.txt +++ b/server/modules/filter/hint/CMakeLists.txt @@ -1,4 +1,4 @@ add_library(hintfilter SHARED hintfilter.c hintparser.c) set_target_properties(hintfilter PROPERTIES INSTALL_RPATH ${CMAKE_INSTALL_RPATH}:${MAXSCALE_LIBDIR} VERSION "1.0.0") target_link_libraries(hintfilter maxscale-common) -install(TARGETS hintfilter DESTINATION ${MAXSCALE_LIBDIR}) +install_module(hintfilter core) diff --git a/server/modules/filter/hint/hintfilter.c b/server/modules/filter/hint/hintfilter.c index 5b758afea..a8fc2242e 100644 --- a/server/modules/filter/hint/hintfilter.c +++ b/server/modules/filter/hint/hintfilter.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -12,6 +12,7 @@ */ #include #include +#include #include #include #include @@ -105,7 +106,7 @@ createInstance(char **options, FILTER_PARAMETER **params) { HINT_INSTANCE *my_instance; - if ((my_instance = calloc(1, sizeof(HINT_INSTANCE))) != NULL) + if ((my_instance = MXS_CALLOC(1, sizeof(HINT_INSTANCE))) != NULL) { my_instance->sessions = 0; } @@ -125,7 +126,7 @@ newSession(FILTER *instance, SESSION *session) HINT_INSTANCE *my_instance = (HINT_INSTANCE *)instance; HINT_SESSION *my_session; - if ((my_session = calloc(1, sizeof(HINT_SESSION))) != NULL) + if ((my_session = MXS_CALLOC(1, sizeof(HINT_SESSION))) != NULL) { my_session->query_len = 0; my_session->request = NULL; @@ -177,7 +178,7 @@ closeSession(FILTER *instance, void *session) static void freeSession(FILTER *instance, void *session) { - free(session); + MXS_FREE(session); return; } diff --git a/server/modules/filter/hint/hintparser.c b/server/modules/filter/hint/hintparser.c index d7197b4e5..bade7e684 100644 --- a/server/modules/filter/hint/hintparser.c +++ b/server/modules/filter/hint/hintparser.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -19,6 +19,7 @@ #include #include #include +#include /** * hintparser.c - Find any comment in the SQL packet and look for MAXSCALE @@ -83,9 +84,9 @@ void token_free(HINT_TOKEN* token) { if (token->value != NULL) { - free(token->value); + MXS_FREE(token->value); } - free(token); + MXS_FREE(token); } static const char* token_get_keyword( @@ -281,7 +282,7 @@ hint_parser(HINT_SESSION *session, GWBUF *request) break; case TOK_STRING: state = HS_NAME; - lvalue = strdup(tok->value); + lvalue = MXS_STRDUP_A(tok->value); break; case TOK_STOP: /* Action: pop active hint */ @@ -452,7 +453,7 @@ hint_parser(HINT_SESSION *session, GWBUF *request) /* We starting an already define set of named hints */ rval = lookup_named_hint(session, hintname); hint_push(session, hint_dup(rval)); - free(hintname); + MXS_FREE(hintname); rval = NULL; } else if (hintname == NULL && rval == NULL) @@ -518,7 +519,7 @@ hint_next_token(GWBUF **buf, char **ptr) int i, found; HINT_TOKEN *tok; - if ((tok = (HINT_TOKEN *)malloc(sizeof(HINT_TOKEN))) == NULL) + if ((tok = (HINT_TOKEN *)MXS_MALLOC(sizeof(HINT_TOKEN))) == NULL) { return NULL; } @@ -603,7 +604,7 @@ hint_next_token(GWBUF **buf, char **ptr) if (found == 0) { tok->token = TOK_STRING; - tok->value = strdup(word); + tok->value = MXS_STRDUP_A(word); } return tok; @@ -629,7 +630,7 @@ hint_pop(HINT_SESSION *session) ptr->hint = hint->next; hint_free(hint); } - free(ptr); + MXS_FREE(ptr); } } @@ -645,7 +646,7 @@ hint_push(HINT_SESSION *session, HINT *hint) { HINTSTACK *item; - if ((item = (HINTSTACK *)malloc(sizeof(HINTSTACK))) == NULL) + if ((item = (HINTSTACK *)MXS_MALLOC(sizeof(HINTSTACK))) == NULL) { return; } @@ -689,7 +690,7 @@ create_named_hint(HINT_SESSION *session, char *name, HINT *hint) { NAMEDHINTS *block; - if ((block = (NAMEDHINTS *)malloc(sizeof(NAMEDHINTS))) == NULL) + if ((block = (NAMEDHINTS *)MXS_MALLOC(sizeof(NAMEDHINTS))) == NULL) { return; } @@ -724,8 +725,8 @@ NAMEDHINTS* free_named_hint( hint_free(named_hint->hints); named_hint->hints = hint; } - free(named_hint->name); - free(named_hint); + MXS_FREE(named_hint->name); + MXS_FREE(named_hint); return next; } else @@ -758,7 +759,7 @@ HINTSTACK* free_hint_stack( hint_free(hint_stack->hint); hint_stack->hint = hint; } - free(hint_stack); + MXS_FREE(hint_stack); return next; } else diff --git a/server/modules/filter/luafilter.c b/server/modules/filter/luafilter.c index a3cd72a38..9123e3ca9 100644 --- a/server/modules/filter/luafilter.c +++ b/server/modules/filter/luafilter.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -47,6 +47,7 @@ #include "lua.h" #include "lualib.h" #include "lauxlib.h" +#include MODULE_INFO info = { @@ -173,7 +174,7 @@ createInstance(char **options, FILTER_PARAMETER **params) LUA_INSTANCE *my_instance; bool error = false; - if ((my_instance = (LUA_INSTANCE*) calloc(1, sizeof(LUA_INSTANCE))) == NULL) + if ((my_instance = (LUA_INSTANCE*) MXS_CALLOC(1, sizeof(LUA_INSTANCE))) == NULL) { return NULL; } @@ -184,11 +185,11 @@ createInstance(char **options, FILTER_PARAMETER **params) { if (strcmp(params[i]->name, "global_script") == 0) { - error = (my_instance->global_script = strdup(params[i]->value)) == NULL; + error = (my_instance->global_script = MXS_STRDUP(params[i]->value)) == NULL; } else if (strcmp(params[i]->name, "session_script") == 0) { - error = (my_instance->session_script = strdup(params[i]->value)) == NULL; + error = (my_instance->session_script = MXS_STRDUP(params[i]->value)) == NULL; } else if (!filter_standard_parameter(params[i]->name)) { @@ -199,9 +200,9 @@ createInstance(char **options, FILTER_PARAMETER **params) if (error) { - free(my_instance->global_script); - free(my_instance->session_script); - free(my_instance); + MXS_FREE(my_instance->global_script); + MXS_FREE(my_instance->session_script); + MXS_FREE(my_instance); return NULL; } @@ -215,9 +216,9 @@ createInstance(char **options, FILTER_PARAMETER **params) { MXS_ERROR("luafilter: Failed to execute global script at '%s':%s.", my_instance->global_script, lua_tostring(my_instance->global_lua_state, -1)); - free(my_instance->global_script); - free(my_instance->session_script); - free(my_instance); + MXS_FREE(my_instance->global_script); + MXS_FREE(my_instance->session_script); + MXS_FREE(my_instance); my_instance = NULL; } else if (my_instance->global_lua_state) @@ -234,7 +235,7 @@ createInstance(char **options, FILTER_PARAMETER **params) else { MXS_ERROR("Unable to initialize new Lua state."); - free(my_instance); + MXS_FREE(my_instance); my_instance = NULL; } } @@ -263,7 +264,7 @@ static void * newSession(FILTER *instance, SESSION *session) LUA_SESSION *my_session; LUA_INSTANCE *my_instance = (LUA_INSTANCE*) instance; - if ((my_session = (LUA_SESSION*) calloc(1, sizeof(LUA_SESSION))) == NULL) + if ((my_session = (LUA_SESSION*) MXS_CALLOC(1, sizeof(LUA_SESSION))) == NULL) { return NULL; } @@ -282,7 +283,7 @@ static void * newSession(FILTER *instance, SESSION *session) my_instance->session_script, lua_tostring(my_session->lua_state, -1)); lua_close(my_session->lua_state); - free(my_session); + MXS_FREE(my_session); my_session = NULL; } else @@ -367,7 +368,7 @@ static void freeSession(FILTER *instance, void *session) { LUA_SESSION *my_session = (LUA_SESSION *) session; lua_close(my_session->lua_state); - free(my_session); + MXS_FREE(my_session); } /** @@ -526,7 +527,7 @@ static int routeQuery(FILTER *instance, void *session, GWBUF *queue) spinlock_release(&my_instance->lock); } - free(fullquery); + MXS_FREE(fullquery); } if (!route) diff --git a/server/modules/filter/mqfilter.c b/server/modules/filter/mqfilter.c index 97a8b230e..09db393df 100644 --- a/server/modules/filter/mqfilter.c +++ b/server/modules/filter/mqfilter.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -76,6 +76,7 @@ #include #include #include +#include MODULE_INFO info = { @@ -474,11 +475,10 @@ char** parse_optstr(char* str, char* tok, int* szstore) size++; } - arr = malloc(sizeof(char*)*size); + arr = MXS_MALLOC(sizeof(char*)*size); if (arr == NULL) { - MXS_ERROR("Cannot allocate enough memory."); *szstore = 0; return NULL; } @@ -487,7 +487,7 @@ char** parse_optstr(char* str, char* tok, int* szstore) tk = strtok_r(str, tok, &lasts); while (tk && i < size) { - arr[i++] = strdup(tk); + arr[i++] = MXS_STRDUP_A(tk); tk = strtok_r(NULL, tok, &lasts); } return arr; @@ -510,17 +510,18 @@ createInstance(char **options, FILTER_PARAMETER **params) char** arr = NULL; char taskname[512]; - if ((my_instance = calloc(1, sizeof(MQ_INSTANCE)))) + if ((my_instance = MXS_CALLOC(1, sizeof(MQ_INSTANCE)))) { spinlock_init(&my_instance->rconn_lock); spinlock_init(&my_instance->msg_lock); uid_gen = 0; - paramlist = malloc(sizeof(FILTER_PARAMETER*) * 64); + paramlist = MXS_MALLOC(sizeof(FILTER_PARAMETER*) * 64); + MXS_ABORT_IF_NULL(paramlist); if ((my_instance->conn = amqp_new_connection()) == NULL) { - free(paramlist); - free(my_instance); + MXS_FREE(paramlist); + MXS_FREE(my_instance); return NULL; } my_instance->channel = 1; @@ -536,19 +537,19 @@ createInstance(char **options, FILTER_PARAMETER **params) { if (!strcmp(params[i]->name, "hostname")) { - my_instance->hostname = strdup(params[i]->value); + my_instance->hostname = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "username")) { - my_instance->username = strdup(params[i]->value); + my_instance->username = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "password")) { - my_instance->password = strdup(params[i]->value); + my_instance->password = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "vhost")) { - my_instance->vhost = strdup(params[i]->value); + my_instance->vhost = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "port")) { @@ -556,35 +557,34 @@ createInstance(char **options, FILTER_PARAMETER **params) } else if (!strcmp(params[i]->name, "exchange")) { - my_instance->exchange = strdup(params[i]->value); + my_instance->exchange = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "key")) { - my_instance->key = strdup(params[i]->value); + my_instance->key = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "queue")) { - my_instance->queue = strdup(params[i]->value); + my_instance->queue = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "ssl_client_certificate")) { - - my_instance->ssl_client_cert = strdup(params[i]->value); + my_instance->ssl_client_cert = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "ssl_client_key")) { - my_instance->ssl_client_key = strdup(params[i]->value); + my_instance->ssl_client_key = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "ssl_CA_cert")) { - my_instance->ssl_CA_cert = strdup(params[i]->value); + my_instance->ssl_CA_cert = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "exchange_type")) { - my_instance->exchange_type = strdup(params[i]->value); + my_instance->exchange_type = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "logging_trigger")) { @@ -619,9 +619,9 @@ createInstance(char **options, FILTER_PARAMETER **params) { for (int x = 0; x < arrsize; x++) { - free(arr[x]); + MXS_FREE(arr[x]); } - free(arr); + MXS_FREE(arr); arr = NULL; } arrsize = 0; @@ -634,9 +634,10 @@ createInstance(char **options, FILTER_PARAMETER **params) if (paramcount < parammax) { - paramlist[paramcount] = malloc(sizeof(FILTER_PARAMETER)); - paramlist[paramcount]->name = strdup(params[i]->name); - paramlist[paramcount]->value = strdup(params[i]->value); + paramlist[paramcount] = MXS_MALLOC(sizeof(FILTER_PARAMETER)); + MXS_ABORT_IF_NULL(paramlist[paramcount]); + paramlist[paramcount]->name = MXS_STRDUP_A(params[i]->name); + paramlist[paramcount]->value = MXS_STRDUP_A(params[i]->value); paramcount++; } } @@ -645,7 +646,8 @@ createInstance(char **options, FILTER_PARAMETER **params) if (my_instance->trgtype & TRG_SOURCE) { - my_instance->src_trg = (SRC_TRIG*) malloc(sizeof(SRC_TRIG)); + my_instance->src_trg = (SRC_TRIG*) MXS_MALLOC(sizeof(SRC_TRIG)); + MXS_ABORT_IF_NULL(my_instance->src_trg); my_instance->src_trg->user = NULL; my_instance->src_trg->host = NULL; my_instance->src_trg->usize = 0; @@ -656,7 +658,8 @@ createInstance(char **options, FILTER_PARAMETER **params) if (my_instance->trgtype & TRG_SCHEMA) { - my_instance->shm_trg = (SHM_TRIG*) malloc(sizeof(SHM_TRIG)); + my_instance->shm_trg = (SHM_TRIG*) MXS_MALLOC(sizeof(SHM_TRIG)); + MXS_ABORT_IF_NULL(my_instance->shm_trg); my_instance->shm_trg->objects = NULL; my_instance->shm_trg->size = 0; @@ -665,7 +668,8 @@ createInstance(char **options, FILTER_PARAMETER **params) if (my_instance->trgtype & TRG_OBJECT) { - my_instance->obj_trg = (OBJ_TRIG*) malloc(sizeof(OBJ_TRIG)); + my_instance->obj_trg = (OBJ_TRIG*) MXS_MALLOC(sizeof(OBJ_TRIG)); + MXS_ABORT_IF_NULL(my_instance->obj_trg); my_instance->obj_trg->objects = NULL; my_instance->obj_trg->size = 0; @@ -732,40 +736,40 @@ createInstance(char **options, FILTER_PARAMETER **params) my_instance->strict_logging = false; } } - free(paramlist[i]->name); - free(paramlist[i]->value); - free(paramlist[i]); + MXS_FREE(paramlist[i]->name); + MXS_FREE(paramlist[i]->value); + MXS_FREE(paramlist[i]); } - free(paramlist); + MXS_FREE(paramlist); if (my_instance->hostname == NULL) { - my_instance->hostname = strdup("localhost"); + my_instance->hostname = MXS_STRDUP_A("localhost"); } if (my_instance->username == NULL) { - my_instance->username = strdup("guest"); + my_instance->username = MXS_STRDUP_A("guest"); } if (my_instance->password == NULL) { - my_instance->password = strdup("guest"); + my_instance->password = MXS_STRDUP_A("guest"); } if (my_instance->vhost == NULL) { - my_instance->vhost = strdup("/"); + my_instance->vhost = MXS_STRDUP_A("/"); } if (my_instance->exchange == NULL) { - my_instance->exchange = strdup("default_exchange"); + my_instance->exchange = MXS_STRDUP_A("default_exchange"); } if (my_instance->key == NULL) { - my_instance->key = strdup("key"); + my_instance->key = MXS_STRDUP_A("key"); } if (my_instance->exchange_type == NULL) { - my_instance->exchange_type = strdup("direct"); + my_instance->exchange_type = MXS_STRDUP_A("direct"); } if (my_instance->ssl_client_cert != NULL && @@ -794,9 +798,9 @@ createInstance(char **options, FILTER_PARAMETER **params) { for (int x = 0; x < arrsize; x++) { - free(arr[x]); + MXS_FREE(arr[x]); } - free(arr); + MXS_FREE(arr); } } return (FILTER *) my_instance; @@ -910,9 +914,9 @@ void sendMessage(void* data) if (err_num == AMQP_STATUS_OK) { /**Message was sent successfully*/ - free(tmp->prop); - free(tmp->msg); - free(tmp); + MXS_FREE(tmp->prop); + MXS_FREE(tmp->msg); + MXS_FREE(tmp); atomic_add(&instance->stats.n_sent, 1); atomic_add(&instance->stats.n_queued, -1); @@ -948,7 +952,7 @@ void sendMessage(void* data) void pushMessage(MQ_INSTANCE *instance, amqp_basic_properties_t* prop, char* msg) { - mqmessage* newmsg = calloc(1, sizeof(mqmessage)); + mqmessage* newmsg = MXS_CALLOC(1, sizeof(mqmessage)); if (newmsg) { newmsg->msg = msg; @@ -956,9 +960,8 @@ void pushMessage(MQ_INSTANCE *instance, amqp_basic_properties_t* prop, char* msg } else { - MXS_ERROR("Cannot allocate enough memory."); - free(prop); - free(msg); + MXS_FREE(prop); + MXS_FREE(msg); return; } @@ -985,23 +988,36 @@ void pushMessage(MQ_INSTANCE *instance, amqp_basic_properties_t* prop, char* msg static void * newSession(FILTER *instance, SESSION *session) { - MQ_SESSION *my_session; - MYSQL_session* sessauth; + MYSQL_session *sessauth = session->client_dcb->data; + char *db = sessauth->db; + if (db) + { + if (strnlen(db, 128) > 0) + { + db = MXS_STRDUP(db); + if (!db) + { + return NULL; + } + } + else + { + db = NULL; + } + } - if ((my_session = calloc(1, sizeof(MQ_SESSION))) != NULL) + MQ_SESSION *my_session; + + if ((my_session = MXS_CALLOC(1, sizeof(MQ_SESSION))) != NULL) { my_session->was_query = false; my_session->uid = NULL; my_session->session = session; - sessauth = my_session->session->client_dcb->data; - if (sessauth->db && strnlen(sessauth->db, 128) > 0) - { - my_session->db = strdup(sessauth->db); - } - else - { - my_session->db = NULL; - } + my_session->db = db; + } + else + { + MXS_FREE(db); } return my_session; @@ -1028,9 +1044,9 @@ static void freeSession(FILTER *instance, void *session) { MQ_SESSION *my_session = (MQ_SESSION *) session; - free(my_session->uid); - free(my_session->db); - free(my_session); + MXS_FREE(my_session->uid); + MXS_FREE(my_session->db); + MXS_FREE(my_session); return; } @@ -1117,10 +1133,11 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) { if (my_session->db) { - free(my_session->db); + MXS_FREE(my_session->db); } plen = pktlen(queue->start); - my_session->db = calloc(plen, sizeof(char)); + my_session->db = MXS_CALLOC(plen, sizeof(char)); + MXS_ABORT_IF_NULL(my_session->db); memcpy(my_session->db, queue->start + 5, plen - 1); } @@ -1229,9 +1246,9 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) { all_remotes = false; } - free(tblnames[z]); + MXS_FREE(tblnames[z]); } - free(tblnames); + MXS_FREE(tblnames); if (!schema_ok && !all_remotes && my_session->db && strlen(my_session->db) > 0) { @@ -1305,9 +1322,9 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) { for (j = 0; j < dbcount; j++) { - free(sesstbls[j]); + MXS_FREE(sesstbls[j]); } - free(sesstbls); + MXS_FREE(sesstbls); dbcount = 0; } @@ -1344,17 +1361,12 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) if (my_session->uid == NULL) { - my_session->uid = calloc(33, sizeof(char)); + my_session->uid = MXS_CALLOC(33, sizeof(char)); - if (!my_session->uid) - { - MXS_ERROR("Out of memory."); - } - else + if (my_session->uid) { genkey(my_session->uid, 32); } - } if (queue->next != NULL) @@ -1367,7 +1379,7 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) my_session->was_query = true; - if ((prop = malloc(sizeof(amqp_basic_properties_t)))) + if ((prop = MXS_MALLOC(sizeof(amqp_basic_properties_t)))) { prop->_flags = AMQP_BASIC_CONTENT_TYPE_FLAG | AMQP_BASIC_DELIVERY_MODE_FLAG | @@ -1379,8 +1391,6 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) prop->message_id = amqp_cstring_bytes("query"); } - - if (success) { @@ -1396,15 +1406,13 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) sprintf(t_buf, "%lu|", (unsigned long) time(NULL)); int qlen = strnlen(canon_q, length) + strnlen(t_buf, 128); - if ((combined = malloc((qlen + 1) * sizeof(char))) == NULL) - { - MXS_ERROR("Out of memory"); - } + combined = MXS_MALLOC((qlen + 1) * sizeof(char)); + MXS_ABORT_IF_NULL(combined); strcpy(combined, t_buf); strncat(combined, canon_q, length); pushMessage(my_instance, prop, combined); - free(canon_q); + MXS_FREE(canon_q); } } @@ -1494,7 +1502,7 @@ unsigned int consume_leitoi(unsigned char** c) char* consume_lestr(unsigned char** c) { unsigned int slen = consume_leitoi(c); - char *str = calloc((slen + 1), sizeof(char)); + char *str = MXS_CALLOC((slen + 1), sizeof(char)); if (str) { memcpy(str, *c, slen); @@ -1545,7 +1553,7 @@ static int clientReply(FILTER* instance, void *session, GWBUF *reply) if (pkt_len > 0) { - if ((prop = malloc(sizeof(amqp_basic_properties_t)))) + if ((prop = MXS_MALLOC(sizeof(amqp_basic_properties_t)))) { prop->_flags = AMQP_BASIC_CONTENT_TYPE_FLAG | AMQP_BASIC_DELIVERY_MODE_FLAG | @@ -1556,10 +1564,9 @@ static int clientReply(FILTER* instance, void *session, GWBUF *reply) prop->correlation_id = amqp_cstring_bytes(my_session->uid); prop->message_id = amqp_cstring_bytes("reply"); } - if (!(combined = calloc(GWBUF_LENGTH(reply) + 256, sizeof(char)))) - { - MXS_ERROR("Out of memory"); - } + + combined = MXS_CALLOC(GWBUF_LENGTH(reply) + 256, sizeof(char)); + MXS_ABORT_IF_NULL(combined); memset(t_buf, 0, 128); sprintf(t_buf, "%lu|", (unsigned long) time(NULL)); @@ -1627,13 +1634,14 @@ static int clientReply(FILTER* instance, void *session, GWBUF *reply) char *tmp; unsigned int col_cnt = consume_leitoi(&rset); - tmp = calloc(256, sizeof(char)); + tmp = MXS_CALLOC(256, sizeof(char)); + MXS_ABORT_IF_NULL(tmp); sprintf(tmp, "Columns: %d", col_cnt); memcpy(combined + offset, tmp, strnlen(tmp, 256)); offset += strnlen(tmp, 256); memcpy(combined + offset, "\n", 1); offset++; - free(tmp); + MXS_FREE(tmp); packet_ok = 1; was_last = 1; @@ -1649,7 +1657,7 @@ static int clientReply(FILTER* instance, void *session, GWBUF *reply) /**Successful reply received and sent, releasing uid*/ - free(my_session->uid); + MXS_FREE(my_session->uid); my_session->uid = NULL; } diff --git a/server/modules/filter/namedserverfilter.c b/server/modules/filter/namedserverfilter.c index 13146db39..0ce3aaa28 100644 --- a/server/modules/filter/namedserverfilter.c +++ b/server/modules/filter/namedserverfilter.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -12,6 +12,7 @@ */ #include +#include #include #include #include @@ -20,6 +21,7 @@ #include #include #include +#include /** * @file namedserverfilter.c - a very simple regular expression based filter @@ -143,10 +145,9 @@ GetModuleObject() static FILTER * createInstance(char **options, FILTER_PARAMETER **params) { - REGEXHINT_INSTANCE *my_instance; - int cflags = REG_ICASE; + REGEXHINT_INSTANCE *my_instance = (REGEXHINT_INSTANCE*)MXS_MALLOC(sizeof(REGEXHINT_INSTANCE)); - if ((my_instance = malloc(sizeof(REGEXHINT_INSTANCE))) != NULL) + if (my_instance) { my_instance->match = NULL; my_instance->server = NULL; @@ -158,19 +159,19 @@ createInstance(char **options, FILTER_PARAMETER **params) { if (!strcmp(params[i]->name, "match")) { - my_instance->match = strdup(params[i]->value); + my_instance->match = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "server")) { - my_instance->server = strdup(params[i]->value); + my_instance->server = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "source")) { - my_instance->source = strdup(params[i]->value); + my_instance->source = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "user")) { - my_instance->user = strdup(params[i]->value); + my_instance->user = MXS_STRDUP_A(params[i]->value); } else if (!filter_standard_parameter(params[i]->name)) { @@ -180,6 +181,8 @@ createInstance(char **options, FILTER_PARAMETER **params) } } + int cflags = REG_ICASE; + if (options) { for (int i = 0; options[i]; i++) @@ -221,7 +224,7 @@ createInstance(char **options, FILTER_PARAMETER **params) { MXS_ERROR("namedserverfilter: Invalid regular expression '%s'.\n", my_instance->match); - free(my_instance->match); + MXS_FREE(my_instance->match); my_instance->match = NULL; error = true; } @@ -231,12 +234,12 @@ createInstance(char **options, FILTER_PARAMETER **params) if (my_instance->match) { regfree(&my_instance->re); - free(my_instance->match); + MXS_FREE(my_instance->match); } - free(my_instance->server); - free(my_instance->source); - free(my_instance->user); - free(my_instance); + MXS_FREE(my_instance->server); + MXS_FREE(my_instance->source); + MXS_FREE(my_instance->user); + MXS_FREE(my_instance); my_instance = NULL; } @@ -258,7 +261,7 @@ newSession(FILTER *instance, SESSION *session) REGEXHINT_SESSION *my_session; char *remote, *user; - if ((my_session = calloc(1, sizeof(REGEXHINT_SESSION))) != NULL) + if ((my_session = MXS_CALLOC(1, sizeof(REGEXHINT_SESSION))) != NULL) { my_session->n_diverted = 0; my_session->n_undiverted = 0; @@ -303,7 +306,7 @@ closeSession(FILTER *instance, void *session) static void freeSession(FILTER *instance, void *session) { - free(session); + MXS_FREE(session); return; } @@ -361,7 +364,7 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) { my_session->n_undiverted++; } - free(sql); + MXS_FREE(sql); } } return my_session->down.routeQuery(my_session->down.instance, diff --git a/server/modules/filter/qlafilter.c b/server/modules/filter/qlafilter.c index 24d760496..5c32cc49e 100644 --- a/server/modules/filter/qlafilter.c +++ b/server/modules/filter/qlafilter.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -47,6 +47,7 @@ #include #include #include +#include MODULE_INFO info = { @@ -173,10 +174,9 @@ GetModuleObject() static FILTER * createInstance(char **options, FILTER_PARAMETER **params) { - QLA_INSTANCE *my_instance; - int i; + QLA_INSTANCE *my_instance = (QLA_INSTANCE*) MXS_MALLOC(sizeof(QLA_INSTANCE)); - if ((my_instance = malloc(sizeof(QLA_INSTANCE))) != NULL) + if (my_instance) { my_instance->source = NULL; my_instance->userName = NULL; @@ -187,27 +187,27 @@ createInstance(char **options, FILTER_PARAMETER **params) if (params) { - for (i = 0; params[i]; i++) + for (int i = 0; params[i]; i++) { if (!strcmp(params[i]->name, "match")) { - my_instance->match = strdup(params[i]->value); + my_instance->match = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "exclude")) { - my_instance->nomatch = strdup(params[i]->value); + my_instance->nomatch = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "source")) { - my_instance->source = strdup(params[i]->value); + my_instance->source = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "user")) { - my_instance->userName = strdup(params[i]->value); + my_instance->userName = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "filebase")) { - my_instance->filebase = strdup(params[i]->value); + my_instance->filebase = MXS_STRDUP_A(params[i]->value); } else if (!filter_standard_parameter(params[i]->name)) { @@ -222,7 +222,7 @@ createInstance(char **options, FILTER_PARAMETER **params) if (options) { - for (i = 0; options[i]; i++) + for (int i = 0; options[i]; i++) { if (!strcasecmp(options[i], "ignorecase")) { @@ -258,7 +258,7 @@ createInstance(char **options, FILTER_PARAMETER **params) MXS_ERROR("qlafilter: Invalid regular expression '%s'" " for the 'match' parameter.\n", my_instance->match); - free(my_instance->match); + MXS_FREE(my_instance->match); my_instance->match = NULL; error = true; } @@ -268,7 +268,7 @@ createInstance(char **options, FILTER_PARAMETER **params) MXS_ERROR("qlafilter: Invalid regular expression '%s'" " for the 'nomatch' parameter.", my_instance->nomatch); - free(my_instance->nomatch); + MXS_FREE(my_instance->nomatch); my_instance->nomatch = NULL; error = true; } @@ -277,19 +277,19 @@ createInstance(char **options, FILTER_PARAMETER **params) { if (my_instance->match) { - free(my_instance->match); + MXS_FREE(my_instance->match); regfree(&my_instance->re); } if (my_instance->nomatch) { - free(my_instance->nomatch); + MXS_FREE(my_instance->nomatch); regfree(&my_instance->nore); } - free(my_instance->filebase); - free(my_instance->source); - free(my_instance->userName); - free(my_instance); + MXS_FREE(my_instance->filebase); + MXS_FREE(my_instance->source); + MXS_FREE(my_instance->userName); + MXS_FREE(my_instance); my_instance = NULL; } } @@ -312,16 +312,11 @@ newSession(FILTER *instance, SESSION *session) QLA_SESSION *my_session; char *remote, *userName; - if ((my_session = calloc(1, sizeof(QLA_SESSION))) != NULL) + if ((my_session = MXS_CALLOC(1, sizeof(QLA_SESSION))) != NULL) { - if ((my_session->filename = (char *)malloc(strlen(my_instance->filebase) + 20)) == NULL) + if ((my_session->filename = (char *)MXS_MALLOC(strlen(my_instance->filebase) + 20)) == NULL) { - char errbuf[STRERROR_BUFLEN]; - MXS_ERROR("Memory allocation for qla filter " - "file name failed due to %d, %s.", - errno, - strerror_r(errno, errbuf, sizeof(errbuf))); - free(my_session); + MXS_FREE(my_session); return NULL; } my_session->active = 1; @@ -359,8 +354,8 @@ newSession(FILTER *instance, SESSION *session) "fileter failed due to %d, %s", errno, strerror_r(errno, errbuf, sizeof(errbuf))); - free(my_session->filename); - free(my_session); + MXS_FREE(my_session->filename); + MXS_FREE(my_session); my_session = NULL; } } @@ -406,8 +401,8 @@ freeSession(FILTER *instance, void *session) { QLA_SESSION *my_session = (QLA_SESSION *) session; - free(my_session->filename); - free(session); + MXS_FREE(my_session->filename); + MXS_FREE(session); return; } @@ -467,7 +462,7 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) fprintf(my_session->fp, "%s,%s@%s,%s\n", buffer, my_session->user, my_session->remote, trim(squeeze_whitespace(ptr))); } - free(ptr); + MXS_FREE(ptr); } } /* Pass the query downstream */ diff --git a/server/modules/filter/regexfilter.c b/server/modules/filter/regexfilter.c index 65802d83b..85d268c35 100644 --- a/server/modules/filter/regexfilter.c +++ b/server/modules/filter/regexfilter.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -21,6 +21,7 @@ #include #include #include +#include #include "maxconfig.h" /** @@ -159,11 +160,11 @@ void free_instance(REGEX_INSTANCE *instance) pcre2_match_data_free(instance->match_data); } - free(instance->match); - free(instance->replace); - free(instance->source); - free(instance->user); - free(instance); + MXS_FREE(instance->match); + MXS_FREE(instance->replace); + MXS_FREE(instance->source); + MXS_FREE(instance->user); + MXS_FREE(instance); } } @@ -185,7 +186,7 @@ createInstance(char **options, FILTER_PARAMETER **params) char *logfile = NULL; const char *errmsg; - if ((my_instance = calloc(1, sizeof(REGEX_INSTANCE))) != NULL) + if ((my_instance = MXS_CALLOC(1, sizeof(REGEX_INSTANCE))) != NULL) { my_instance->match = NULL; my_instance->replace = NULL; @@ -194,19 +195,19 @@ createInstance(char **options, FILTER_PARAMETER **params) { if (!strcmp(params[i]->name, "match")) { - my_instance->match = strdup(params[i]->value); + my_instance->match = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "replace")) { - my_instance->replace = strdup(params[i]->value); + my_instance->replace = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "source")) { - my_instance->source = strdup(params[i]->value); + my_instance->source = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "user")) { - my_instance->user = strdup(params[i]->value); + my_instance->user = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "log_trace")) { @@ -216,9 +217,9 @@ createInstance(char **options, FILTER_PARAMETER **params) { if (logfile) { - free(logfile); + MXS_FREE(logfile); } - logfile = strdup(params[i]->value); + logfile = MXS_STRDUP_A(params[i]->value); } else if (!filter_standard_parameter(params[i]->name)) { @@ -253,14 +254,14 @@ createInstance(char **options, FILTER_PARAMETER **params) { MXS_ERROR("regexfilter: Failed to open file '%s'.", logfile); free_instance(my_instance); - free(logfile); + MXS_FREE(logfile); return NULL; } fprintf(my_instance->logfile, "\nOpened regex filter log\n"); fflush(my_instance->logfile); } - free(logfile); + MXS_FREE(logfile); if (my_instance->match == NULL || my_instance->replace == NULL) { @@ -309,7 +310,7 @@ newSession(FILTER *instance, SESSION *session) REGEX_SESSION *my_session; char *remote, *user; - if ((my_session = calloc(1, sizeof(REGEX_SESSION))) != NULL) + if ((my_session = MXS_CALLOC(1, sizeof(REGEX_SESSION))) != NULL) { my_session->no_change = 0; my_session->replacements = 0; @@ -354,7 +355,7 @@ closeSession(FILTER *instance, void *session) static void freeSession(FILTER *instance, void *session) { - free(session); + MXS_FREE(session); return; } @@ -408,7 +409,7 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) spinlock_acquire(&my_session->lock); log_match(my_instance, my_instance->match, sql, newsql); spinlock_release(&my_session->lock); - free(newsql); + MXS_FREE(newsql); my_session->replacements++; } else @@ -418,7 +419,7 @@ routeQuery(FILTER *instance, void *session, GWBUF *queue) spinlock_release(&my_session->lock); my_session->no_change++; } - free(sql); + MXS_FREE(sql); } } @@ -485,7 +486,7 @@ regex_replace(const char *sql, pcre2_code *re, pcre2_match_data *match_data, con if (pcre2_match(re, (PCRE2_SPTR) sql, PCRE2_ZERO_TERMINATED, 0, 0, match_data, NULL)) { result_size = strlen(sql) + strlen(replace); - result = malloc(result_size); + result = MXS_MALLOC(result_size); while (result && pcre2_substitute(re, (PCRE2_SPTR) sql, PCRE2_ZERO_TERMINATED, 0, @@ -494,9 +495,9 @@ regex_replace(const char *sql, pcre2_code *re, pcre2_match_data *match_data, con (PCRE2_UCHAR*) result, (PCRE2_SIZE*) & result_size) == PCRE2_ERROR_NOMEMORY) { char *tmp; - if ((tmp = realloc(result, (result_size *= 1.5))) == NULL) + if ((tmp = MXS_REALLOC(result, (result_size *= 1.5))) == NULL) { - free(result); + MXS_FREE(result); result = NULL; } result = tmp; diff --git a/server/modules/filter/tee.c b/server/modules/filter/tee.c index 600bda685..cffda5060 100644 --- a/server/modules/filter/tee.c +++ b/server/modules/filter/tee.c @@ -4,7 +4,7 @@ * Use of this software is governed by the Business Source License included * in the LICENSE.TXT file and at www.mariadb.com/bsl. * - * Change Date: 2019-01-01 + * Change Date: 2019-07-01 * * On the date above, in accordance with the Business Source License, use * of this software will be governed by version 2 or later of the General @@ -58,6 +58,8 @@ #include #include #include +#include +#include #define MYSQL_COM_QUIT 0x01 #define MYSQL_COM_INITDB 0x02 @@ -209,6 +211,8 @@ int route_single_query(TEE_INSTANCE* my_instance, int reset_session_state(TEE_SESSION* my_session, GWBUF* buffer); void create_orphan(SESSION* ses); +extern LIST_CONFIG SESSIONlist; + static void orphan_free(void* data) { @@ -293,8 +297,8 @@ orphan_free(void* data) tmp->session->router_session); tmp->session->state = SESSION_STATE_FREE; - free(tmp->session); - free(tmp); + list_free_entry(&SESSIONlist, (list_entry_t*)tmp->session); + MXS_FREE(tmp); } #ifdef SS_DEBUG @@ -358,7 +362,7 @@ createInstance(char **options, FILTER_PARAMETER **params) TEE_INSTANCE *my_instance; int i; - if ((my_instance = calloc(1, sizeof(TEE_INSTANCE))) != NULL) + if ((my_instance = MXS_CALLOC(1, sizeof(TEE_INSTANCE))) != NULL) { if (options) { @@ -384,19 +388,19 @@ createInstance(char **options, FILTER_PARAMETER **params) } else if (!strcmp(params[i]->name, "match")) { - my_instance->match = strdup(params[i]->value); + my_instance->match = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "exclude")) { - my_instance->nomatch = strdup(params[i]->value); + my_instance->nomatch = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "source")) { - my_instance->source = strdup(params[i]->value); + my_instance->source = MXS_STRDUP_A(params[i]->value); } else if (!strcmp(params[i]->name, "user")) { - my_instance->userName = strdup(params[i]->value); + my_instance->userName = MXS_STRDUP_A(params[i]->value); } else if (!filter_standard_parameter(params[i]->name)) { @@ -434,9 +438,9 @@ createInstance(char **options, FILTER_PARAMETER **params) if (my_instance->service == NULL) { - free(my_instance->match); - free(my_instance->source); - free(my_instance); + MXS_FREE(my_instance->match); + MXS_FREE(my_instance->source); + MXS_FREE(my_instance); return NULL; } @@ -446,10 +450,10 @@ createInstance(char **options, FILTER_PARAMETER **params) MXS_ERROR("tee: Invalid regular expression '%s'" " for the match parameter.", my_instance->match); - free(my_instance->match); - free(my_instance->nomatch); - free(my_instance->source); - free(my_instance); + MXS_FREE(my_instance->match); + MXS_FREE(my_instance->nomatch); + MXS_FREE(my_instance->source); + MXS_FREE(my_instance); return NULL; } if (my_instance->nomatch && @@ -461,11 +465,11 @@ createInstance(char **options, FILTER_PARAMETER **params) if (my_instance->match) { regfree(&my_instance->re); - free(my_instance->match); + MXS_FREE(my_instance->match); } - free(my_instance->nomatch); - free(my_instance->source); - free(my_instance); + MXS_FREE(my_instance->nomatch); + MXS_FREE(my_instance->source); + MXS_FREE(my_instance); return NULL; } } @@ -496,7 +500,7 @@ newSession(FILTER *instance, SESSION *session) goto retblock; } - HASHTABLE* ht = hashtable_alloc(100, simple_str_hash, strcmp); + HASHTABLE* ht = hashtable_alloc(100, hashtable_item_strhash, hashtable_item_strcmp); bool is_loop = detect_loops(my_instance, ht, session->service); hashtable_free(ht); @@ -508,7 +512,7 @@ newSession(FILTER *instance, SESSION *session) goto retblock; } - if ((my_session = calloc(1, sizeof(TEE_SESSION))) != NULL) + if ((my_session = MXS_CALLOC(1, sizeof(TEE_SESSION))) != NULL) { my_session->active = 1; my_session->residual = 0; @@ -597,7 +601,7 @@ newSession(FILTER *instance, SESSION *session) filter_free(dummy); closeSession(instance, (void*) my_session); dcb_close(dcb); - free(my_session); + MXS_FREE(my_session); MXS_ERROR("tee: Allocating memory for" "dummy upstream failed." " Terminating session."); @@ -608,7 +612,7 @@ newSession(FILTER *instance, SESSION *session) ses->tail = *dummy_upstream; MySQLProtocol* protocol = (MySQLProtocol*) session->client_dcb->protocol; my_session->use_ok = protocol->client_capabilities & (1 << 6); - free(dummy_upstream); + MXS_FREE(dummy_upstream); } } retblock: @@ -707,7 +711,7 @@ freeSession(FILTER *instance, void *session) ses->router_session); /** Free memory of branch client session */ ses->state = SESSION_STATE_FREE; - free(ses); + MXS_FREE(ses); /** This indicates that branch session is not available anymore */ my_session->branch_session = NULL; } @@ -724,7 +728,7 @@ freeSession(FILTER *instance, void *session) { gwbuf_free(my_session->tee_replybuf); } - free(session); + MXS_FREE(session); orphan_free(NULL); @@ -1248,7 +1252,7 @@ GWBUF* clone_query(TEE_INSTANCE* my_instance, TEE_SESSION* my_session, GWBUF* bu clone = gwbuf_clone_all(buffer); my_session->residual = residual; } - free(ptr); + MXS_FREE(ptr); } else if (packet_is_required(buffer)) { @@ -1357,14 +1361,8 @@ int reset_session_state(TEE_SESSION* my_session, GWBUF* buffer) void create_orphan(SESSION* ses) { - orphan_session_t* orphan; - if ((orphan = malloc(sizeof(orphan_session_t))) == NULL) - { - MXS_ERROR("Failed to " - "allocate memory for orphan session struct, " - "child session might leak memory."); - } - else + orphan_session_t* orphan = MXS_MALLOC(sizeof(orphan_session_t)); + if (orphan) { orphan->session = ses; spinlock_acquire(&orphanLock); diff --git a/server/modules/filter/test/CMakeLists.txt b/server/modules/filter/test/CMakeLists.txt deleted file mode 100644 index efccd4ec4..000000000 --- a/server/modules/filter/test/CMakeLists.txt +++ /dev/null @@ -1,26 +0,0 @@ -include_directories(${CMAKE_CURRENT_SOURCE_DIR}) -add_executable(harness_ui harness_ui.c harness_common.c) -add_executable(harness harness_util.c harness_common.c) -target_link_libraries(harness_ui maxscale-common) -target_link_libraries(harness maxscale-common) -execute_process(COMMAND ${CMAKE_COMMAND} -E copy ${ERRMSG} ${CMAKE_CURRENT_BINARY_DIR}) -execute_process(COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_SOURCE_DIR}/harness.cnf ${CMAKE_CURRENT_BINARY_DIR}) -configure_file(${CMAKE_CURRENT_SOURCE_DIR}/testdriver.sh ${CMAKE_CURRENT_BINARY_DIR}/testdriver.sh @ONLY) - -configure_file(${CMAKE_CURRENT_SOURCE_DIR}/hintfilter/hint_testing.cnf ${CMAKE_CURRENT_BINARY_DIR}/hintfilter/hint_testing.cnf) -add_test(TestHintfilter testdriver.sh hintfilter/hint_testing.cnf hintfilter/hint_testing.input hintfilter/hint_testing.output hintfilter/hint_testing.expected) - -configure_file(${CMAKE_CURRENT_SOURCE_DIR}/regexfilter/regextest.cnf ${CMAKE_CURRENT_BINARY_DIR}/regexfilter/regextest.cnf) -add_test(TestRegexfilter testdriver.sh regexfilter/regextest.cnf regexfilter/regextest.input regexfilter/regextest.output regexfilter/regextest.expected) - -configure_file(${CMAKE_CURRENT_SOURCE_DIR}/fwfilter/fwtest.cnf.in ${CMAKE_CURRENT_BINARY_DIR}/fwfilter/fwtest.cnf) -add_test(TestFwfilter1 testdriver.sh fwfilter/fwtest.cnf fwfilter/fwtest.input fwfilter/fwtest.output fwfilter/fwtest.expected) -add_test(TestFwfilter2 testdriver.sh fwfilter/fwtest.cnf fwfilter/fwtest2.input fwfilter/fwtest2.output fwfilter/fwtest2.expected) - -add_test(TestTeeRecursion ${CMAKE_CURRENT_SOURCE_DIR}/tee_recursion.sh - ${CMAKE_BINARY_DIR} - ${CMAKE_SOURCE_DIR} - ${TEST_USER} - ${TEST_PASSWORD} - ${TEST_HOST} - ${TEST_PORT}) diff --git a/server/modules/filter/test/README b/server/modules/filter/test/README deleted file mode 100644 index ee8f81edd..000000000 --- a/server/modules/filter/test/README +++ /dev/null @@ -1,20 +0,0 @@ -Filter Test Harness - -For a more detailed description of the filter harness, either generate the documentation or read the harness.h file. - -Running the program without arguments enters the interactive mode. Type 'help' for a list of all commands. - -The default values for threads and sessions are stored in the 'harness.cnf' file - -Mandatory parameters for the command line mode are -c and -i. - -Parameters for the command line: - - -h Display this information - -c Path to the MaxScale configuration file to parse for filters - -i Name of the input file for buffers - -o Name of the output file for results - -q Suppress printing to stdout - -t Number of threads - -s Number of sessions - -d Routing delay diff --git a/server/modules/filter/test/doxygen.conf b/server/modules/filter/test/doxygen.conf deleted file mode 100644 index a3c6eae77..000000000 --- a/server/modules/filter/test/doxygen.conf +++ /dev/null @@ -1,2303 +0,0 @@ -# Doxyfile 1.8.6 - -# This file describes the settings to be used by the documentation system -# doxygen (www.doxygen.org) for a project. -# -# All text after a double hash (##) is considered a comment and is placed in -# front of the TAG it is preceding. -# -# All text after a single hash (#) is considered a comment and will be ignored. -# The format is: -# TAG = value [value, ...] -# For lists, items can also be appended using: -# TAG += value [value, ...] -# Values that contain spaces should be placed between quotes (\" \"). - -#--------------------------------------------------------------------------- -# Project related configuration options -#--------------------------------------------------------------------------- - -# This tag specifies the encoding used for all characters in the config file -# that follow. The default is UTF-8 which is also the encoding used for all text -# before the first occurrence of this tag. Doxygen uses libiconv (or the iconv -# built into libc) for the transcoding. See http://www.gnu.org/software/libiconv -# for the list of possible encodings. -# The default value is: UTF-8. - -DOXYFILE_ENCODING = UTF-8 - -# The PROJECT_NAME tag is a single word (or a sequence of words surrounded by -# double-quotes, unless you are using Doxywizard) that should identify the -# project for which the documentation is generated. This name is used in the -# title of most generated pages and in a few other places. -# The default value is: My Project. - -PROJECT_NAME = "Filter Harness" - -# The PROJECT_NUMBER tag can be used to enter a project or revision number. This -# could be handy for archiving the generated documentation or if some version -# control system is used. - -PROJECT_NUMBER = 1.1 - -# Using the PROJECT_BRIEF tag one can provide an optional one line description -# for a project that appears at the top of each page and should give viewer a -# quick idea about the purpose of the project. Keep the description short. - -PROJECT_BRIEF = "Test harness for independent testing of filters" - -# With the PROJECT_LOGO tag one can specify an logo or icon that is included in -# the documentation. The maximum height of the logo should not exceed 55 pixels -# and the maximum width should not exceed 200 pixels. Doxygen will copy the logo -# to the output directory. - -PROJECT_LOGO = - -# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) path -# into which the generated documentation will be written. If a relative path is -# entered, it will be relative to the location where doxygen was started. If -# left blank the current directory will be used. - -OUTPUT_DIRECTORY = doc/ - -# If the CREATE_SUBDIRS tag is set to YES, then doxygen will create 4096 sub- -# directories (in 2 levels) under the output directory of each output format and -# will distribute the generated files over these directories. Enabling this -# option can be useful when feeding doxygen a huge amount of source files, where -# putting all generated files in the same directory would otherwise causes -# performance problems for the file system. -# The default value is: NO. - -CREATE_SUBDIRS = NO - -# The OUTPUT_LANGUAGE tag is used to specify the language in which all -# documentation generated by doxygen is written. Doxygen will use this -# information to generate all constant output in the proper language. -# Possible values are: Afrikaans, Arabic, Armenian, Brazilian, Catalan, Chinese, -# Chinese-Traditional, Croatian, Czech, Danish, Dutch, English (United States), -# Esperanto, Farsi (Persian), Finnish, French, German, Greek, Hungarian, -# Indonesian, Italian, Japanese, Japanese-en (Japanese with English messages), -# Korean, Korean-en (Korean with English messages), Latvian, Lithuanian, -# Macedonian, Norwegian, Persian (Farsi), Polish, Portuguese, Romanian, Russian, -# Serbian, Serbian-Cyrillic, Slovak, Slovene, Spanish, Swedish, Turkish, -# Ukrainian and Vietnamese. -# The default value is: English. - -OUTPUT_LANGUAGE = English - -# If the BRIEF_MEMBER_DESC tag is set to YES doxygen will include brief member -# descriptions after the members that are listed in the file and class -# documentation (similar to Javadoc). Set to NO to disable this. -# The default value is: YES. - -BRIEF_MEMBER_DESC = YES - -# If the REPEAT_BRIEF tag is set to YES doxygen will prepend the brief -# description of a member or function before the detailed description -# -# Note: If both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the -# brief descriptions will be completely suppressed. -# The default value is: YES. - -REPEAT_BRIEF = YES - -# This tag implements a quasi-intelligent brief description abbreviator that is -# used to form the text in various listings. Each string in this list, if found -# as the leading text of the brief description, will be stripped from the text -# and the result, after processing the whole list, is used as the annotated -# text. Otherwise, the brief description is used as-is. If left blank, the -# following values are used ($name is automatically replaced with the name of -# the entity):The $name class, The $name widget, The $name file, is, provides, -# specifies, contains, represents, a, an and the. - -ABBREVIATE_BRIEF = - -# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then -# doxygen will generate a detailed section even if there is only a brief -# description. -# The default value is: NO. - -ALWAYS_DETAILED_SEC = NO - -# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all -# inherited members of a class in the documentation of that class as if those -# members were ordinary class members. Constructors, destructors and assignment -# operators of the base classes will not be shown. -# The default value is: NO. - -INLINE_INHERITED_MEMB = NO - -# If the FULL_PATH_NAMES tag is set to YES doxygen will prepend the full path -# before files name in the file list and in the header files. If set to NO the -# shortest path that makes the file name unique will be used -# The default value is: YES. - -FULL_PATH_NAMES = YES - -# The STRIP_FROM_PATH tag can be used to strip a user-defined part of the path. -# Stripping is only done if one of the specified strings matches the left-hand -# part of the path. The tag can be used to show relative paths in the file list. -# If left blank the directory from which doxygen is run is used as the path to -# strip. -# -# Note that you can specify absolute paths here, but also relative paths, which -# will be relative from the directory where doxygen is started. -# This tag requires that the tag FULL_PATH_NAMES is set to YES. - -STRIP_FROM_PATH = - -# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of the -# path mentioned in the documentation of a class, which tells the reader which -# header file to include in order to use a class. If left blank only the name of -# the header file containing the class definition is used. Otherwise one should -# specify the list of include paths that are normally passed to the compiler -# using the -I flag. - -STRIP_FROM_INC_PATH = - -# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter (but -# less readable) file names. This can be useful is your file systems doesn't -# support long names like on DOS, Mac, or CD-ROM. -# The default value is: NO. - -SHORT_NAMES = NO - -# If the JAVADOC_AUTOBRIEF tag is set to YES then doxygen will interpret the -# first line (until the first dot) of a Javadoc-style comment as the brief -# description. If set to NO, the Javadoc-style will behave just like regular Qt- -# style comments (thus requiring an explicit @brief command for a brief -# description.) -# The default value is: NO. - -JAVADOC_AUTOBRIEF = NO - -# If the QT_AUTOBRIEF tag is set to YES then doxygen will interpret the first -# line (until the first dot) of a Qt-style comment as the brief description. If -# set to NO, the Qt-style will behave just like regular Qt-style comments (thus -# requiring an explicit \brief command for a brief description.) -# The default value is: NO. - -QT_AUTOBRIEF = NO - -# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make doxygen treat a -# multi-line C++ special comment block (i.e. a block of //! or /// comments) as -# a brief description. This used to be the default behavior. The new default is -# to treat a multi-line C++ comment block as a detailed description. Set this -# tag to YES if you prefer the old behavior instead. -# -# Note that setting this tag to YES also means that rational rose comments are -# not recognized any more. -# The default value is: NO. - -MULTILINE_CPP_IS_BRIEF = NO - -# If the INHERIT_DOCS tag is set to YES then an undocumented member inherits the -# documentation from any documented member that it re-implements. -# The default value is: YES. - -INHERIT_DOCS = YES - -# If the SEPARATE_MEMBER_PAGES tag is set to YES, then doxygen will produce a -# new page for each member. If set to NO, the documentation of a member will be -# part of the file/class/namespace that contains it. -# The default value is: NO. - -SEPARATE_MEMBER_PAGES = NO - -# The TAB_SIZE tag can be used to set the number of spaces in a tab. Doxygen -# uses this value to replace tabs by spaces in code fragments. -# Minimum value: 1, maximum value: 16, default value: 4. - -TAB_SIZE = 4 - -# This tag can be used to specify a number of aliases that act as commands in -# the documentation. An alias has the form: -# name=value -# For example adding -# "sideeffect=@par Side Effects:\n" -# will allow you to put the command \sideeffect (or @sideeffect) in the -# documentation, which will result in a user-defined paragraph with heading -# "Side Effects:". You can put \n's in the value part of an alias to insert -# newlines. - -ALIASES = - -# This tag can be used to specify a number of word-keyword mappings (TCL only). -# A mapping has the form "name=value". For example adding "class=itcl::class" -# will allow you to use the command class in the itcl::class meaning. - -TCL_SUBST = - -# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C sources -# only. Doxygen will then generate output that is more tailored for C. For -# instance, some of the names that are used will be different. The list of all -# members will be omitted, etc. -# The default value is: NO. - -OPTIMIZE_OUTPUT_FOR_C = YES - -# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java or -# Python sources only. Doxygen will then generate output that is more tailored -# for that language. For instance, namespaces will be presented as packages, -# qualified scopes will look different, etc. -# The default value is: NO. - -OPTIMIZE_OUTPUT_JAVA = NO - -# Set the OPTIMIZE_FOR_FORTRAN tag to YES if your project consists of Fortran -# sources. Doxygen will then generate output that is tailored for Fortran. -# The default value is: NO. - -OPTIMIZE_FOR_FORTRAN = NO - -# Set the OPTIMIZE_OUTPUT_VHDL tag to YES if your project consists of VHDL -# sources. Doxygen will then generate output that is tailored for VHDL. -# The default value is: NO. - -OPTIMIZE_OUTPUT_VHDL = NO - -# Doxygen selects the parser to use depending on the extension of the files it -# parses. With this tag you can assign which parser to use for a given -# extension. Doxygen has a built-in mapping, but you can override or extend it -# using this tag. The format is ext=language, where ext is a file extension, and -# language is one of the parsers supported by doxygen: IDL, Java, Javascript, -# C#, C, C++, D, PHP, Objective-C, Python, Fortran, VHDL. For instance to make -# doxygen treat .inc files as Fortran files (default is PHP), and .f files as C -# (default is Fortran), use: inc=Fortran f=C. -# -# Note For files without extension you can use no_extension as a placeholder. -# -# Note that for custom extensions you also need to set FILE_PATTERNS otherwise -# the files are not read by doxygen. - -EXTENSION_MAPPING = - -# If the MARKDOWN_SUPPORT tag is enabled then doxygen pre-processes all comments -# according to the Markdown format, which allows for more readable -# documentation. See http://daringfireball.net/projects/markdown/ for details. -# The output of markdown processing is further processed by doxygen, so you can -# mix doxygen, HTML, and XML commands with Markdown formatting. Disable only in -# case of backward compatibilities issues. -# The default value is: YES. - -MARKDOWN_SUPPORT = YES - -# When enabled doxygen tries to link words that correspond to documented -# classes, or namespaces to their corresponding documentation. Such a link can -# be prevented in individual cases by by putting a % sign in front of the word -# or globally by setting AUTOLINK_SUPPORT to NO. -# The default value is: YES. - -AUTOLINK_SUPPORT = YES - -# If you use STL classes (i.e. std::string, std::vector, etc.) but do not want -# to include (a tag file for) the STL sources as input, then you should set this -# tag to YES in order to let doxygen match functions declarations and -# definitions whose arguments contain STL classes (e.g. func(std::string); -# versus func(std::string) {}). This also make the inheritance and collaboration -# diagrams that involve STL classes more complete and accurate. -# The default value is: NO. - -BUILTIN_STL_SUPPORT = NO - -# If you use Microsoft's C++/CLI language, you should set this option to YES to -# enable parsing support. -# The default value is: NO. - -CPP_CLI_SUPPORT = NO - -# Set the SIP_SUPPORT tag to YES if your project consists of sip (see: -# http://www.riverbankcomputing.co.uk/software/sip/intro) sources only. Doxygen -# will parse them like normal C++ but will assume all classes use public instead -# of private inheritance when no explicit protection keyword is present. -# The default value is: NO. - -SIP_SUPPORT = NO - -# For Microsoft's IDL there are propget and propput attributes to indicate -# getter and setter methods for a property. Setting this option to YES will make -# doxygen to replace the get and set methods by a property in the documentation. -# This will only work if the methods are indeed getting or setting a simple -# type. If this is not the case, or you want to show the methods anyway, you -# should set this option to NO. -# The default value is: YES. - -IDL_PROPERTY_SUPPORT = YES - -# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC -# tag is set to YES, then doxygen will reuse the documentation of the first -# member in the group (if any) for the other members of the group. By default -# all members of a group must be documented explicitly. -# The default value is: NO. - -DISTRIBUTE_GROUP_DOC = NO - -# Set the SUBGROUPING tag to YES to allow class member groups of the same type -# (for instance a group of public functions) to be put as a subgroup of that -# type (e.g. under the Public Functions section). Set it to NO to prevent -# subgrouping. Alternatively, this can be done per class using the -# \nosubgrouping command. -# The default value is: YES. - -SUBGROUPING = YES - -# When the INLINE_GROUPED_CLASSES tag is set to YES, classes, structs and unions -# are shown inside the group in which they are included (e.g. using \ingroup) -# instead of on a separate page (for HTML and Man pages) or section (for LaTeX -# and RTF). -# -# Note that this feature does not work in combination with -# SEPARATE_MEMBER_PAGES. -# The default value is: NO. - -INLINE_GROUPED_CLASSES = NO - -# When the INLINE_SIMPLE_STRUCTS tag is set to YES, structs, classes, and unions -# with only public data fields or simple typedef fields will be shown inline in -# the documentation of the scope in which they are defined (i.e. file, -# namespace, or group documentation), provided this scope is documented. If set -# to NO, structs, classes, and unions are shown on a separate page (for HTML and -# Man pages) or section (for LaTeX and RTF). -# The default value is: NO. - -INLINE_SIMPLE_STRUCTS = NO - -# When TYPEDEF_HIDES_STRUCT tag is enabled, a typedef of a struct, union, or -# enum is documented as struct, union, or enum with the name of the typedef. So -# typedef struct TypeS {} TypeT, will appear in the documentation as a struct -# with name TypeT. When disabled the typedef will appear as a member of a file, -# namespace, or class. And the struct will be named TypeS. This can typically be -# useful for C code in case the coding convention dictates that all compound -# types are typedef'ed and only the typedef is referenced, never the tag name. -# The default value is: NO. - -TYPEDEF_HIDES_STRUCT = NO - -# The size of the symbol lookup cache can be set using LOOKUP_CACHE_SIZE. This -# cache is used to resolve symbols given their name and scope. Since this can be -# an expensive process and often the same symbol appears multiple times in the -# code, doxygen keeps a cache of pre-resolved symbols. If the cache is too small -# doxygen will become slower. If the cache is too large, memory is wasted. The -# cache size is given by this formula: 2^(16+LOOKUP_CACHE_SIZE). The valid range -# is 0..9, the default is 0, corresponding to a cache size of 2^16=65536 -# symbols. At the end of a run doxygen will report the cache usage and suggest -# the optimal cache size from a speed point of view. -# Minimum value: 0, maximum value: 9, default value: 0. - -LOOKUP_CACHE_SIZE = 0 - -#--------------------------------------------------------------------------- -# Build related configuration options -#--------------------------------------------------------------------------- - -# If the EXTRACT_ALL tag is set to YES doxygen will assume all entities in -# documentation are documented, even if no documentation was available. Private -# class members and static file members will be hidden unless the -# EXTRACT_PRIVATE respectively EXTRACT_STATIC tags are set to YES. -# Note: This will also disable the warnings about undocumented members that are -# normally produced when WARNINGS is set to YES. -# The default value is: NO. - -EXTRACT_ALL = YES - -# If the EXTRACT_PRIVATE tag is set to YES all private members of a class will -# be included in the documentation. -# The default value is: NO. - -EXTRACT_PRIVATE = NO - -# If the EXTRACT_PACKAGE tag is set to YES all members with package or internal -# scope will be included in the documentation. -# The default value is: NO. - -EXTRACT_PACKAGE = NO - -# If the EXTRACT_STATIC tag is set to YES all static members of a file will be -# included in the documentation. -# The default value is: NO. - -EXTRACT_STATIC = NO - -# If the EXTRACT_LOCAL_CLASSES tag is set to YES classes (and structs) defined -# locally in source files will be included in the documentation. If set to NO -# only classes defined in header files are included. Does not have any effect -# for Java sources. -# The default value is: YES. - -EXTRACT_LOCAL_CLASSES = YES - -# This flag is only useful for Objective-C code. When set to YES local methods, -# which are defined in the implementation section but not in the interface are -# included in the documentation. If set to NO only methods in the interface are -# included. -# The default value is: NO. - -EXTRACT_LOCAL_METHODS = NO - -# If this flag is set to YES, the members of anonymous namespaces will be -# extracted and appear in the documentation as a namespace called -# 'anonymous_namespace{file}', where file will be replaced with the base name of -# the file that contains the anonymous namespace. By default anonymous namespace -# are hidden. -# The default value is: NO. - -EXTRACT_ANON_NSPACES = NO - -# If the HIDE_UNDOC_MEMBERS tag is set to YES, doxygen will hide all -# undocumented members inside documented classes or files. If set to NO these -# members will be included in the various overviews, but no documentation -# section is generated. This option has no effect if EXTRACT_ALL is enabled. -# The default value is: NO. - -HIDE_UNDOC_MEMBERS = NO - -# If the HIDE_UNDOC_CLASSES tag is set to YES, doxygen will hide all -# undocumented classes that are normally visible in the class hierarchy. If set -# to NO these classes will be included in the various overviews. This option has -# no effect if EXTRACT_ALL is enabled. -# The default value is: NO. - -HIDE_UNDOC_CLASSES = NO - -# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, doxygen will hide all friend -# (class|struct|union) declarations. If set to NO these declarations will be -# included in the documentation. -# The default value is: NO. - -HIDE_FRIEND_COMPOUNDS = NO - -# If the HIDE_IN_BODY_DOCS tag is set to YES, doxygen will hide any -# documentation blocks found inside the body of a function. If set to NO these -# blocks will be appended to the function's detailed documentation block. -# The default value is: NO. - -HIDE_IN_BODY_DOCS = NO - -# The INTERNAL_DOCS tag determines if documentation that is typed after a -# \internal command is included. If the tag is set to NO then the documentation -# will be excluded. Set it to YES to include the internal documentation. -# The default value is: NO. - -INTERNAL_DOCS = NO - -# If the CASE_SENSE_NAMES tag is set to NO then doxygen will only generate file -# names in lower-case letters. If set to YES upper-case letters are also -# allowed. This is useful if you have classes or files whose names only differ -# in case and if your file system supports case sensitive file names. Windows -# and Mac users are advised to set this option to NO. -# The default value is: system dependent. - -CASE_SENSE_NAMES = YES - -# If the HIDE_SCOPE_NAMES tag is set to NO then doxygen will show members with -# their full class and namespace scopes in the documentation. If set to YES the -# scope will be hidden. -# The default value is: NO. - -HIDE_SCOPE_NAMES = NO - -# If the SHOW_INCLUDE_FILES tag is set to YES then doxygen will put a list of -# the files that are included by a file in the documentation of that file. -# The default value is: YES. - -SHOW_INCLUDE_FILES = YES - -# If the SHOW_GROUPED_MEMB_INC tag is set to YES then Doxygen will add for each -# grouped member an include statement to the documentation, telling the reader -# which file to include in order to use the member. -# The default value is: NO. - -SHOW_GROUPED_MEMB_INC = NO - -# If the FORCE_LOCAL_INCLUDES tag is set to YES then doxygen will list include -# files with double quotes in the documentation rather than with sharp brackets. -# The default value is: NO. - -FORCE_LOCAL_INCLUDES = NO - -# If the INLINE_INFO tag is set to YES then a tag [inline] is inserted in the -# documentation for inline members. -# The default value is: YES. - -INLINE_INFO = YES - -# If the SORT_MEMBER_DOCS tag is set to YES then doxygen will sort the -# (detailed) documentation of file and class members alphabetically by member -# name. If set to NO the members will appear in declaration order. -# The default value is: YES. - -SORT_MEMBER_DOCS = YES - -# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the brief -# descriptions of file, namespace and class members alphabetically by member -# name. If set to NO the members will appear in declaration order. Note that -# this will also influence the order of the classes in the class list. -# The default value is: NO. - -SORT_BRIEF_DOCS = NO - -# If the SORT_MEMBERS_CTORS_1ST tag is set to YES then doxygen will sort the -# (brief and detailed) documentation of class members so that constructors and -# destructors are listed first. If set to NO the constructors will appear in the -# respective orders defined by SORT_BRIEF_DOCS and SORT_MEMBER_DOCS. -# Note: If SORT_BRIEF_DOCS is set to NO this option is ignored for sorting brief -# member documentation. -# Note: If SORT_MEMBER_DOCS is set to NO this option is ignored for sorting -# detailed member documentation. -# The default value is: NO. - -SORT_MEMBERS_CTORS_1ST = NO - -# If the SORT_GROUP_NAMES tag is set to YES then doxygen will sort the hierarchy -# of group names into alphabetical order. If set to NO the group names will -# appear in their defined order. -# The default value is: NO. - -SORT_GROUP_NAMES = NO - -# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be sorted by -# fully-qualified names, including namespaces. If set to NO, the class list will -# be sorted only by class name, not including the namespace part. -# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES. -# Note: This option applies only to the class list, not to the alphabetical -# list. -# The default value is: NO. - -SORT_BY_SCOPE_NAME = NO - -# If the STRICT_PROTO_MATCHING option is enabled and doxygen fails to do proper -# type resolution of all parameters of a function it will reject a match between -# the prototype and the implementation of a member function even if there is -# only one candidate or it is obvious which candidate to choose by doing a -# simple string match. By disabling STRICT_PROTO_MATCHING doxygen will still -# accept a match between prototype and implementation in such cases. -# The default value is: NO. - -STRICT_PROTO_MATCHING = NO - -# The GENERATE_TODOLIST tag can be used to enable ( YES) or disable ( NO) the -# todo list. This list is created by putting \todo commands in the -# documentation. -# The default value is: YES. - -GENERATE_TODOLIST = YES - -# The GENERATE_TESTLIST tag can be used to enable ( YES) or disable ( NO) the -# test list. This list is created by putting \test commands in the -# documentation. -# The default value is: YES. - -GENERATE_TESTLIST = YES - -# The GENERATE_BUGLIST tag can be used to enable ( YES) or disable ( NO) the bug -# list. This list is created by putting \bug commands in the documentation. -# The default value is: YES. - -GENERATE_BUGLIST = YES - -# The GENERATE_DEPRECATEDLIST tag can be used to enable ( YES) or disable ( NO) -# the deprecated list. This list is created by putting \deprecated commands in -# the documentation. -# The default value is: YES. - -GENERATE_DEPRECATEDLIST= YES - -# The ENABLED_SECTIONS tag can be used to enable conditional documentation -# sections, marked by \if ... \endif and \cond -# ... \endcond blocks. - -ENABLED_SECTIONS = - -# The MAX_INITIALIZER_LINES tag determines the maximum number of lines that the -# initial value of a variable or macro / define can have for it to appear in the -# documentation. If the initializer consists of more lines than specified here -# it will be hidden. Use a value of 0 to hide initializers completely. The -# appearance of the value of individual variables and macros / defines can be -# controlled using \showinitializer or \hideinitializer command in the -# documentation regardless of this setting. -# Minimum value: 0, maximum value: 10000, default value: 30. - -MAX_INITIALIZER_LINES = 30 - -# Set the SHOW_USED_FILES tag to NO to disable the list of files generated at -# the bottom of the documentation of classes and structs. If set to YES the list -# will mention the files that were used to generate the documentation. -# The default value is: YES. - -SHOW_USED_FILES = YES - -# Set the SHOW_FILES tag to NO to disable the generation of the Files page. This -# will remove the Files entry from the Quick Index and from the Folder Tree View -# (if specified). -# The default value is: YES. - -SHOW_FILES = YES - -# Set the SHOW_NAMESPACES tag to NO to disable the generation of the Namespaces -# page. This will remove the Namespaces entry from the Quick Index and from the -# Folder Tree View (if specified). -# The default value is: YES. - -SHOW_NAMESPACES = YES - -# The FILE_VERSION_FILTER tag can be used to specify a program or script that -# doxygen should invoke to get the current version for each file (typically from -# the version control system). Doxygen will invoke the program by executing (via -# popen()) the command command input-file, where command is the value of the -# FILE_VERSION_FILTER tag, and input-file is the name of an input file provided -# by doxygen. Whatever the program writes to standard output is used as the file -# version. For an example see the documentation. - -FILE_VERSION_FILTER = - -# The LAYOUT_FILE tag can be used to specify a layout file which will be parsed -# by doxygen. The layout file controls the global structure of the generated -# output files in an output format independent way. To create the layout file -# that represents doxygen's defaults, run doxygen with the -l option. You can -# optionally specify a file name after the option, if omitted DoxygenLayout.xml -# will be used as the name of the layout file. -# -# Note that if you run doxygen from a directory containing a file called -# DoxygenLayout.xml, doxygen will parse it automatically even if the LAYOUT_FILE -# tag is left empty. - -LAYOUT_FILE = - -# The CITE_BIB_FILES tag can be used to specify one or more bib files containing -# the reference definitions. This must be a list of .bib files. The .bib -# extension is automatically appended if omitted. This requires the bibtex tool -# to be installed. See also http://en.wikipedia.org/wiki/BibTeX for more info. -# For LaTeX the style of the bibliography can be controlled using -# LATEX_BIB_STYLE. To use this feature you need bibtex and perl available in the -# search path. Do not use file names with spaces, bibtex cannot handle them. See -# also \cite for info how to create references. - -CITE_BIB_FILES = - -#--------------------------------------------------------------------------- -# Configuration options related to warning and progress messages -#--------------------------------------------------------------------------- - -# The QUIET tag can be used to turn on/off the messages that are generated to -# standard output by doxygen. If QUIET is set to YES this implies that the -# messages are off. -# The default value is: NO. - -QUIET = NO - -# The WARNINGS tag can be used to turn on/off the warning messages that are -# generated to standard error ( stderr) by doxygen. If WARNINGS is set to YES -# this implies that the warnings are on. -# -# Tip: Turn warnings on while writing the documentation. -# The default value is: YES. - -WARNINGS = YES - -# If the WARN_IF_UNDOCUMENTED tag is set to YES, then doxygen will generate -# warnings for undocumented members. If EXTRACT_ALL is set to YES then this flag -# will automatically be disabled. -# The default value is: YES. - -WARN_IF_UNDOCUMENTED = YES - -# If the WARN_IF_DOC_ERROR tag is set to YES, doxygen will generate warnings for -# potential errors in the documentation, such as not documenting some parameters -# in a documented function, or documenting parameters that don't exist or using -# markup commands wrongly. -# The default value is: YES. - -WARN_IF_DOC_ERROR = YES - -# This WARN_NO_PARAMDOC option can be enabled to get warnings for functions that -# are documented, but have no documentation for their parameters or return -# value. If set to NO doxygen will only warn about wrong or incomplete parameter -# documentation, but not about the absence of documentation. -# The default value is: NO. - -WARN_NO_PARAMDOC = NO - -# The WARN_FORMAT tag determines the format of the warning messages that doxygen -# can produce. The string should contain the $file, $line, and $text tags, which -# will be replaced by the file and line number from which the warning originated -# and the warning text. Optionally the format may contain $version, which will -# be replaced by the version of the file (if it could be obtained via -# FILE_VERSION_FILTER) -# The default value is: $file:$line: $text. - -WARN_FORMAT = "$file:$line: $text" - -# The WARN_LOGFILE tag can be used to specify a file to which warning and error -# messages should be written. If left blank the output is written to standard -# error (stderr). - -WARN_LOGFILE = - -#--------------------------------------------------------------------------- -# Configuration options related to the input files -#--------------------------------------------------------------------------- - -# The INPUT tag is used to specify the files and/or directories that contain -# documented source files. You may enter file names like myfile.cpp or -# directories like /usr/src/myproject. Separate the files or directories with -# spaces. -# Note: If this tag is empty the current directory is searched. - -INPUT = - -# This tag can be used to specify the character encoding of the source files -# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses -# libiconv (or the iconv built into libc) for the transcoding. See the libiconv -# documentation (see: http://www.gnu.org/software/libiconv) for the list of -# possible encodings. -# The default value is: UTF-8. - -INPUT_ENCODING = UTF-8 - -# If the value of the INPUT tag contains directories, you can use the -# FILE_PATTERNS tag to specify one or more wildcard patterns (like *.cpp and -# *.h) to filter out the source-files in the directories. If left blank the -# following patterns are tested:*.c, *.cc, *.cxx, *.cpp, *.c++, *.java, *.ii, -# *.ixx, *.ipp, *.i++, *.inl, *.idl, *.ddl, *.odl, *.h, *.hh, *.hxx, *.hpp, -# *.h++, *.cs, *.d, *.php, *.php4, *.php5, *.phtml, *.inc, *.m, *.markdown, -# *.md, *.mm, *.dox, *.py, *.f90, *.f, *.for, *.tcl, *.vhd, *.vhdl, *.ucf, -# *.qsf, *.as and *.js. - -FILE_PATTERNS = - -# The RECURSIVE tag can be used to specify whether or not subdirectories should -# be searched for input files as well. -# The default value is: NO. - -RECURSIVE = NO - -# The EXCLUDE tag can be used to specify files and/or directories that should be -# excluded from the INPUT source files. This way you can easily exclude a -# subdirectory from a directory tree whose root is specified with the INPUT tag. -# -# Note that relative paths are relative to the directory from which doxygen is -# run. - -EXCLUDE = - -# The EXCLUDE_SYMLINKS tag can be used to select whether or not files or -# directories that are symbolic links (a Unix file system feature) are excluded -# from the input. -# The default value is: NO. - -EXCLUDE_SYMLINKS = NO - -# If the value of the INPUT tag contains directories, you can use the -# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude -# certain files from those directories. -# -# Note that the wildcards are matched against the file with absolute path, so to -# exclude all test directories for example use the pattern */test/* - -EXCLUDE_PATTERNS = - -# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names -# (namespaces, classes, functions, etc.) that should be excluded from the -# output. The symbol name can be a fully qualified name, a word, or if the -# wildcard * is used, a substring. Examples: ANamespace, AClass, -# AClass::ANamespace, ANamespace::*Test -# -# Note that the wildcards are matched against the file with absolute path, so to -# exclude all test directories use the pattern */test/* - -EXCLUDE_SYMBOLS = - -# The EXAMPLE_PATH tag can be used to specify one or more files or directories -# that contain example code fragments that are included (see the \include -# command). - -EXAMPLE_PATH = - -# If the value of the EXAMPLE_PATH tag contains directories, you can use the -# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp and -# *.h) to filter out the source-files in the directories. If left blank all -# files are included. - -EXAMPLE_PATTERNS = - -# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be -# searched for input files to be used with the \include or \dontinclude commands -# irrespective of the value of the RECURSIVE tag. -# The default value is: NO. - -EXAMPLE_RECURSIVE = NO - -# The IMAGE_PATH tag can be used to specify one or more files or directories -# that contain images that are to be included in the documentation (see the -# \image command). - -IMAGE_PATH = - -# The INPUT_FILTER tag can be used to specify a program that doxygen should -# invoke to filter for each input file. Doxygen will invoke the filter program -# by executing (via popen()) the command: -# -# -# -# where is the value of the INPUT_FILTER tag, and is the -# name of an input file. Doxygen will then use the output that the filter -# program writes to standard output. If FILTER_PATTERNS is specified, this tag -# will be ignored. -# -# Note that the filter must not add or remove lines; it is applied before the -# code is scanned, but not when the output code is generated. If lines are added -# or removed, the anchors will not be placed correctly. - -INPUT_FILTER = - -# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern -# basis. Doxygen will compare the file name with each pattern and apply the -# filter if there is a match. The filters are a list of the form: pattern=filter -# (like *.cpp=my_cpp_filter). See INPUT_FILTER for further information on how -# filters are used. If the FILTER_PATTERNS tag is empty or if none of the -# patterns match the file name, INPUT_FILTER is applied. - -FILTER_PATTERNS = - -# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using -# INPUT_FILTER ) will also be used to filter the input files that are used for -# producing the source files to browse (i.e. when SOURCE_BROWSER is set to YES). -# The default value is: NO. - -FILTER_SOURCE_FILES = NO - -# The FILTER_SOURCE_PATTERNS tag can be used to specify source filters per file -# pattern. A pattern will override the setting for FILTER_PATTERN (if any) and -# it is also possible to disable source filtering for a specific pattern using -# *.ext= (so without naming a filter). -# This tag requires that the tag FILTER_SOURCE_FILES is set to YES. - -FILTER_SOURCE_PATTERNS = - -# If the USE_MDFILE_AS_MAINPAGE tag refers to the name of a markdown file that -# is part of the input, its contents will be placed on the main page -# (index.html). This can be useful if you have a project on for instance GitHub -# and want to reuse the introduction page also for the doxygen output. - -USE_MDFILE_AS_MAINPAGE = - -#--------------------------------------------------------------------------- -# Configuration options related to source browsing -#--------------------------------------------------------------------------- - -# If the SOURCE_BROWSER tag is set to YES then a list of source files will be -# generated. Documented entities will be cross-referenced with these sources. -# -# Note: To get rid of all source code in the generated output, make sure that -# also VERBATIM_HEADERS is set to NO. -# The default value is: NO. - -SOURCE_BROWSER = NO - -# Setting the INLINE_SOURCES tag to YES will include the body of functions, -# classes and enums directly into the documentation. -# The default value is: NO. - -INLINE_SOURCES = NO - -# Setting the STRIP_CODE_COMMENTS tag to YES will instruct doxygen to hide any -# special comment blocks from generated source code fragments. Normal C, C++ and -# Fortran comments will always remain visible. -# The default value is: YES. - -STRIP_CODE_COMMENTS = YES - -# If the REFERENCED_BY_RELATION tag is set to YES then for each documented -# function all documented functions referencing it will be listed. -# The default value is: NO. - -REFERENCED_BY_RELATION = NO - -# If the REFERENCES_RELATION tag is set to YES then for each documented function -# all documented entities called/used by that function will be listed. -# The default value is: NO. - -REFERENCES_RELATION = NO - -# If the REFERENCES_LINK_SOURCE tag is set to YES and SOURCE_BROWSER tag is set -# to YES, then the hyperlinks from functions in REFERENCES_RELATION and -# REFERENCED_BY_RELATION lists will link to the source code. Otherwise they will -# link to the documentation. -# The default value is: YES. - -REFERENCES_LINK_SOURCE = YES - -# If SOURCE_TOOLTIPS is enabled (the default) then hovering a hyperlink in the -# source code will show a tooltip with additional information such as prototype, -# brief description and links to the definition and documentation. Since this -# will make the HTML file larger and loading of large files a bit slower, you -# can opt to disable this feature. -# The default value is: YES. -# This tag requires that the tag SOURCE_BROWSER is set to YES. - -SOURCE_TOOLTIPS = YES - -# If the USE_HTAGS tag is set to YES then the references to source code will -# point to the HTML generated by the htags(1) tool instead of doxygen built-in -# source browser. The htags tool is part of GNU's global source tagging system -# (see http://www.gnu.org/software/global/global.html). You will need version -# 4.8.6 or higher. -# -# To use it do the following: -# - Install the latest version of global -# - Enable SOURCE_BROWSER and USE_HTAGS in the config file -# - Make sure the INPUT points to the root of the source tree -# - Run doxygen as normal -# -# Doxygen will invoke htags (and that will in turn invoke gtags), so these -# tools must be available from the command line (i.e. in the search path). -# -# The result: instead of the source browser generated by doxygen, the links to -# source code will now point to the output of htags. -# The default value is: NO. -# This tag requires that the tag SOURCE_BROWSER is set to YES. - -USE_HTAGS = NO - -# If the VERBATIM_HEADERS tag is set the YES then doxygen will generate a -# verbatim copy of the header file for each class for which an include is -# specified. Set to NO to disable this. -# See also: Section \class. -# The default value is: YES. - -VERBATIM_HEADERS = YES - -#--------------------------------------------------------------------------- -# Configuration options related to the alphabetical class index -#--------------------------------------------------------------------------- - -# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index of all -# compounds will be generated. Enable this if the project contains a lot of -# classes, structs, unions or interfaces. -# The default value is: YES. - -ALPHABETICAL_INDEX = YES - -# The COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns in -# which the alphabetical index list will be split. -# Minimum value: 1, maximum value: 20, default value: 5. -# This tag requires that the tag ALPHABETICAL_INDEX is set to YES. - -COLS_IN_ALPHA_INDEX = 5 - -# In case all classes in a project start with a common prefix, all classes will -# be put under the same header in the alphabetical index. The IGNORE_PREFIX tag -# can be used to specify a prefix (or a list of prefixes) that should be ignored -# while generating the index headers. -# This tag requires that the tag ALPHABETICAL_INDEX is set to YES. - -IGNORE_PREFIX = - -#--------------------------------------------------------------------------- -# Configuration options related to the HTML output -#--------------------------------------------------------------------------- - -# If the GENERATE_HTML tag is set to YES doxygen will generate HTML output -# The default value is: YES. - -GENERATE_HTML = YES - -# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. If a -# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of -# it. -# The default directory is: html. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_OUTPUT = html - -# The HTML_FILE_EXTENSION tag can be used to specify the file extension for each -# generated HTML page (for example: .htm, .php, .asp). -# The default value is: .html. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_FILE_EXTENSION = .html - -# The HTML_HEADER tag can be used to specify a user-defined HTML header file for -# each generated HTML page. If the tag is left blank doxygen will generate a -# standard header. -# -# To get valid HTML the header file that includes any scripts and style sheets -# that doxygen needs, which is dependent on the configuration options used (e.g. -# the setting GENERATE_TREEVIEW). It is highly recommended to start with a -# default header using -# doxygen -w html new_header.html new_footer.html new_stylesheet.css -# YourConfigFile -# and then modify the file new_header.html. See also section "Doxygen usage" -# for information on how to generate the default header that doxygen normally -# uses. -# Note: The header is subject to change so you typically have to regenerate the -# default header when upgrading to a newer version of doxygen. For a description -# of the possible markers and block names see the documentation. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_HEADER = - -# The HTML_FOOTER tag can be used to specify a user-defined HTML footer for each -# generated HTML page. If the tag is left blank doxygen will generate a standard -# footer. See HTML_HEADER for more information on how to generate a default -# footer and what special commands can be used inside the footer. See also -# section "Doxygen usage" for information on how to generate the default footer -# that doxygen normally uses. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_FOOTER = - -# The HTML_STYLESHEET tag can be used to specify a user-defined cascading style -# sheet that is used by each HTML page. It can be used to fine-tune the look of -# the HTML output. If left blank doxygen will generate a default style sheet. -# See also section "Doxygen usage" for information on how to generate the style -# sheet that doxygen normally uses. -# Note: It is recommended to use HTML_EXTRA_STYLESHEET instead of this tag, as -# it is more robust and this tag (HTML_STYLESHEET) will in the future become -# obsolete. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_STYLESHEET = - -# The HTML_EXTRA_STYLESHEET tag can be used to specify an additional user- -# defined cascading style sheet that is included after the standard style sheets -# created by doxygen. Using this option one can overrule certain style aspects. -# This is preferred over using HTML_STYLESHEET since it does not replace the -# standard style sheet and is therefor more robust against future updates. -# Doxygen will copy the style sheet file to the output directory. For an example -# see the documentation. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_EXTRA_STYLESHEET = - -# The HTML_EXTRA_FILES tag can be used to specify one or more extra images or -# other source files which should be copied to the HTML output directory. Note -# that these files will be copied to the base HTML output directory. Use the -# $relpath^ marker in the HTML_HEADER and/or HTML_FOOTER files to load these -# files. In the HTML_STYLESHEET file, use the file name only. Also note that the -# files will be copied as-is; there are no commands or markers available. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_EXTRA_FILES = - -# The HTML_COLORSTYLE_HUE tag controls the color of the HTML output. Doxygen -# will adjust the colors in the stylesheet and background images according to -# this color. Hue is specified as an angle on a colorwheel, see -# http://en.wikipedia.org/wiki/Hue for more information. For instance the value -# 0 represents red, 60 is yellow, 120 is green, 180 is cyan, 240 is blue, 300 -# purple, and 360 is red again. -# Minimum value: 0, maximum value: 359, default value: 220. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_COLORSTYLE_HUE = 220 - -# The HTML_COLORSTYLE_SAT tag controls the purity (or saturation) of the colors -# in the HTML output. For a value of 0 the output will use grayscales only. A -# value of 255 will produce the most vivid colors. -# Minimum value: 0, maximum value: 255, default value: 100. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_COLORSTYLE_SAT = 100 - -# The HTML_COLORSTYLE_GAMMA tag controls the gamma correction applied to the -# luminance component of the colors in the HTML output. Values below 100 -# gradually make the output lighter, whereas values above 100 make the output -# darker. The value divided by 100 is the actual gamma applied, so 80 represents -# a gamma of 0.8, The value 220 represents a gamma of 2.2, and 100 does not -# change the gamma. -# Minimum value: 40, maximum value: 240, default value: 80. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_COLORSTYLE_GAMMA = 80 - -# If the HTML_TIMESTAMP tag is set to YES then the footer of each generated HTML -# page will contain the date and time when the page was generated. Setting this -# to NO can help when comparing the output of multiple runs. -# The default value is: YES. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_TIMESTAMP = NO - -# If the HTML_DYNAMIC_SECTIONS tag is set to YES then the generated HTML -# documentation will contain sections that can be hidden and shown after the -# page has loaded. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_DYNAMIC_SECTIONS = NO - -# With HTML_INDEX_NUM_ENTRIES one can control the preferred number of entries -# shown in the various tree structured indices initially; the user can expand -# and collapse entries dynamically later on. Doxygen will expand the tree to -# such a level that at most the specified number of entries are visible (unless -# a fully collapsed tree already exceeds this amount). So setting the number of -# entries 1 will produce a full collapsed tree by default. 0 is a special value -# representing an infinite number of entries and will result in a full expanded -# tree by default. -# Minimum value: 0, maximum value: 9999, default value: 100. -# This tag requires that the tag GENERATE_HTML is set to YES. - -HTML_INDEX_NUM_ENTRIES = 100 - -# If the GENERATE_DOCSET tag is set to YES, additional index files will be -# generated that can be used as input for Apple's Xcode 3 integrated development -# environment (see: http://developer.apple.com/tools/xcode/), introduced with -# OSX 10.5 (Leopard). To create a documentation set, doxygen will generate a -# Makefile in the HTML output directory. Running make will produce the docset in -# that directory and running make install will install the docset in -# ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find it at -# startup. See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html -# for more information. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -GENERATE_DOCSET = NO - -# This tag determines the name of the docset feed. A documentation feed provides -# an umbrella under which multiple documentation sets from a single provider -# (such as a company or product suite) can be grouped. -# The default value is: Doxygen generated docs. -# This tag requires that the tag GENERATE_DOCSET is set to YES. - -DOCSET_FEEDNAME = "Doxygen generated docs" - -# This tag specifies a string that should uniquely identify the documentation -# set bundle. This should be a reverse domain-name style string, e.g. -# com.mycompany.MyDocSet. Doxygen will append .docset to the name. -# The default value is: org.doxygen.Project. -# This tag requires that the tag GENERATE_DOCSET is set to YES. - -DOCSET_BUNDLE_ID = org.doxygen.Project - -# The DOCSET_PUBLISHER_ID tag specifies a string that should uniquely identify -# the documentation publisher. This should be a reverse domain-name style -# string, e.g. com.mycompany.MyDocSet.documentation. -# The default value is: org.doxygen.Publisher. -# This tag requires that the tag GENERATE_DOCSET is set to YES. - -DOCSET_PUBLISHER_ID = org.doxygen.Publisher - -# The DOCSET_PUBLISHER_NAME tag identifies the documentation publisher. -# The default value is: Publisher. -# This tag requires that the tag GENERATE_DOCSET is set to YES. - -DOCSET_PUBLISHER_NAME = Publisher - -# If the GENERATE_HTMLHELP tag is set to YES then doxygen generates three -# additional HTML index files: index.hhp, index.hhc, and index.hhk. The -# index.hhp is a project file that can be read by Microsoft's HTML Help Workshop -# (see: http://www.microsoft.com/en-us/download/details.aspx?id=21138) on -# Windows. -# -# The HTML Help Workshop contains a compiler that can convert all HTML output -# generated by doxygen into a single compiled HTML file (.chm). Compiled HTML -# files are now used as the Windows 98 help format, and will replace the old -# Windows help format (.hlp) on all Windows platforms in the future. Compressed -# HTML files also contain an index, a table of contents, and you can search for -# words in the documentation. The HTML workshop also contains a viewer for -# compressed HTML files. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -GENERATE_HTMLHELP = NO - -# The CHM_FILE tag can be used to specify the file name of the resulting .chm -# file. You can add a path in front of the file if the result should not be -# written to the html output directory. -# This tag requires that the tag GENERATE_HTMLHELP is set to YES. - -CHM_FILE = - -# The HHC_LOCATION tag can be used to specify the location (absolute path -# including file name) of the HTML help compiler ( hhc.exe). If non-empty -# doxygen will try to run the HTML help compiler on the generated index.hhp. -# The file has to be specified with full path. -# This tag requires that the tag GENERATE_HTMLHELP is set to YES. - -HHC_LOCATION = - -# The GENERATE_CHI flag controls if a separate .chi index file is generated ( -# YES) or that it should be included in the master .chm file ( NO). -# The default value is: NO. -# This tag requires that the tag GENERATE_HTMLHELP is set to YES. - -GENERATE_CHI = NO - -# The CHM_INDEX_ENCODING is used to encode HtmlHelp index ( hhk), content ( hhc) -# and project file content. -# This tag requires that the tag GENERATE_HTMLHELP is set to YES. - -CHM_INDEX_ENCODING = - -# The BINARY_TOC flag controls whether a binary table of contents is generated ( -# YES) or a normal table of contents ( NO) in the .chm file. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTMLHELP is set to YES. - -BINARY_TOC = NO - -# The TOC_EXPAND flag can be set to YES to add extra items for group members to -# the table of contents of the HTML help documentation and to the tree view. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTMLHELP is set to YES. - -TOC_EXPAND = NO - -# If the GENERATE_QHP tag is set to YES and both QHP_NAMESPACE and -# QHP_VIRTUAL_FOLDER are set, an additional index file will be generated that -# can be used as input for Qt's qhelpgenerator to generate a Qt Compressed Help -# (.qch) of the generated HTML documentation. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -GENERATE_QHP = NO - -# If the QHG_LOCATION tag is specified, the QCH_FILE tag can be used to specify -# the file name of the resulting .qch file. The path specified is relative to -# the HTML output folder. -# This tag requires that the tag GENERATE_QHP is set to YES. - -QCH_FILE = - -# The QHP_NAMESPACE tag specifies the namespace to use when generating Qt Help -# Project output. For more information please see Qt Help Project / Namespace -# (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#namespace). -# The default value is: org.doxygen.Project. -# This tag requires that the tag GENERATE_QHP is set to YES. - -QHP_NAMESPACE = org.doxygen.Project - -# The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating Qt -# Help Project output. For more information please see Qt Help Project / Virtual -# Folders (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#virtual- -# folders). -# The default value is: doc. -# This tag requires that the tag GENERATE_QHP is set to YES. - -QHP_VIRTUAL_FOLDER = doc - -# If the QHP_CUST_FILTER_NAME tag is set, it specifies the name of a custom -# filter to add. For more information please see Qt Help Project / Custom -# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom- -# filters). -# This tag requires that the tag GENERATE_QHP is set to YES. - -QHP_CUST_FILTER_NAME = - -# The QHP_CUST_FILTER_ATTRS tag specifies the list of the attributes of the -# custom filter to add. For more information please see Qt Help Project / Custom -# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom- -# filters). -# This tag requires that the tag GENERATE_QHP is set to YES. - -QHP_CUST_FILTER_ATTRS = - -# The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this -# project's filter section matches. Qt Help Project / Filter Attributes (see: -# http://qt-project.org/doc/qt-4.8/qthelpproject.html#filter-attributes). -# This tag requires that the tag GENERATE_QHP is set to YES. - -QHP_SECT_FILTER_ATTRS = - -# The QHG_LOCATION tag can be used to specify the location of Qt's -# qhelpgenerator. If non-empty doxygen will try to run qhelpgenerator on the -# generated .qhp file. -# This tag requires that the tag GENERATE_QHP is set to YES. - -QHG_LOCATION = - -# If the GENERATE_ECLIPSEHELP tag is set to YES, additional index files will be -# generated, together with the HTML files, they form an Eclipse help plugin. To -# install this plugin and make it available under the help contents menu in -# Eclipse, the contents of the directory containing the HTML and XML files needs -# to be copied into the plugins directory of eclipse. The name of the directory -# within the plugins directory should be the same as the ECLIPSE_DOC_ID value. -# After copying Eclipse needs to be restarted before the help appears. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -GENERATE_ECLIPSEHELP = NO - -# A unique identifier for the Eclipse help plugin. When installing the plugin -# the directory name containing the HTML and XML files should also have this -# name. Each documentation set should have its own identifier. -# The default value is: org.doxygen.Project. -# This tag requires that the tag GENERATE_ECLIPSEHELP is set to YES. - -ECLIPSE_DOC_ID = org.doxygen.Project - -# If you want full control over the layout of the generated HTML pages it might -# be necessary to disable the index and replace it with your own. The -# DISABLE_INDEX tag can be used to turn on/off the condensed index (tabs) at top -# of each HTML page. A value of NO enables the index and the value YES disables -# it. Since the tabs in the index contain the same information as the navigation -# tree, you can set this option to YES if you also set GENERATE_TREEVIEW to YES. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -DISABLE_INDEX = NO - -# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index -# structure should be generated to display hierarchical information. If the tag -# value is set to YES, a side panel will be generated containing a tree-like -# index structure (just like the one that is generated for HTML Help). For this -# to work a browser that supports JavaScript, DHTML, CSS and frames is required -# (i.e. any modern browser). Windows users are probably better off using the -# HTML help feature. Via custom stylesheets (see HTML_EXTRA_STYLESHEET) one can -# further fine-tune the look of the index. As an example, the default style -# sheet generated by doxygen has an example that shows how to put an image at -# the root of the tree instead of the PROJECT_NAME. Since the tree basically has -# the same information as the tab index, you could consider setting -# DISABLE_INDEX to YES when enabling this option. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -GENERATE_TREEVIEW = NO - -# The ENUM_VALUES_PER_LINE tag can be used to set the number of enum values that -# doxygen will group on one line in the generated HTML documentation. -# -# Note that a value of 0 will completely suppress the enum values from appearing -# in the overview section. -# Minimum value: 0, maximum value: 20, default value: 4. -# This tag requires that the tag GENERATE_HTML is set to YES. - -ENUM_VALUES_PER_LINE = 4 - -# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be used -# to set the initial width (in pixels) of the frame in which the tree is shown. -# Minimum value: 0, maximum value: 1500, default value: 250. -# This tag requires that the tag GENERATE_HTML is set to YES. - -TREEVIEW_WIDTH = 250 - -# When the EXT_LINKS_IN_WINDOW option is set to YES doxygen will open links to -# external symbols imported via tag files in a separate window. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -EXT_LINKS_IN_WINDOW = NO - -# Use this tag to change the font size of LaTeX formulas included as images in -# the HTML documentation. When you change the font size after a successful -# doxygen run you need to manually remove any form_*.png images from the HTML -# output directory to force them to be regenerated. -# Minimum value: 8, maximum value: 50, default value: 10. -# This tag requires that the tag GENERATE_HTML is set to YES. - -FORMULA_FONTSIZE = 10 - -# Use the FORMULA_TRANPARENT tag to determine whether or not the images -# generated for formulas are transparent PNGs. Transparent PNGs are not -# supported properly for IE 6.0, but are supported on all modern browsers. -# -# Note that when changing this option you need to delete any form_*.png files in -# the HTML output directory before the changes have effect. -# The default value is: YES. -# This tag requires that the tag GENERATE_HTML is set to YES. - -FORMULA_TRANSPARENT = YES - -# Enable the USE_MATHJAX option to render LaTeX formulas using MathJax (see -# http://www.mathjax.org) which uses client side Javascript for the rendering -# instead of using prerendered bitmaps. Use this if you do not have LaTeX -# installed or if you want to formulas look prettier in the HTML output. When -# enabled you may also need to install MathJax separately and configure the path -# to it using the MATHJAX_RELPATH option. -# The default value is: NO. -# This tag requires that the tag GENERATE_HTML is set to YES. - -USE_MATHJAX = NO - -# When MathJax is enabled you can set the default output format to be used for -# the MathJax output. See the MathJax site (see: -# http://docs.mathjax.org/en/latest/output.html) for more details. -# Possible values are: HTML-CSS (which is slower, but has the best -# compatibility), NativeMML (i.e. MathML) and SVG. -# The default value is: HTML-CSS. -# This tag requires that the tag USE_MATHJAX is set to YES. - -MATHJAX_FORMAT = HTML-CSS - -# When MathJax is enabled you need to specify the location relative to the HTML -# output directory using the MATHJAX_RELPATH option. The destination directory -# should contain the MathJax.js script. For instance, if the mathjax directory -# is located at the same level as the HTML output directory, then -# MATHJAX_RELPATH should be ../mathjax. The default value points to the MathJax -# Content Delivery Network so you can quickly see the result without installing -# MathJax. However, it is strongly recommended to install a local copy of -# MathJax from http://www.mathjax.org before deployment. -# The default value is: http://cdn.mathjax.org/mathjax/latest. -# This tag requires that the tag USE_MATHJAX is set to YES. - -MATHJAX_RELPATH = http://cdn.mathjax.org/mathjax/latest - -# The MATHJAX_EXTENSIONS tag can be used to specify one or more MathJax -# extension names that should be enabled during MathJax rendering. For example -# MATHJAX_EXTENSIONS = TeX/AMSmath TeX/AMSsymbols -# This tag requires that the tag USE_MATHJAX is set to YES. - -MATHJAX_EXTENSIONS = - -# The MATHJAX_CODEFILE tag can be used to specify a file with javascript pieces -# of code that will be used on startup of the MathJax code. See the MathJax site -# (see: http://docs.mathjax.org/en/latest/output.html) for more details. For an -# example see the documentation. -# This tag requires that the tag USE_MATHJAX is set to YES. - -MATHJAX_CODEFILE = - -# When the SEARCHENGINE tag is enabled doxygen will generate a search box for -# the HTML output. The underlying search engine uses javascript and DHTML and -# should work on any modern browser. Note that when using HTML help -# (GENERATE_HTMLHELP), Qt help (GENERATE_QHP), or docsets (GENERATE_DOCSET) -# there is already a search function so this one should typically be disabled. -# For large projects the javascript based search engine can be slow, then -# enabling SERVER_BASED_SEARCH may provide a better solution. It is possible to -# search using the keyboard; to jump to the search box use + S -# (what the is depends on the OS and browser, but it is typically -# , /