Merge branch 'develop' into release-1.1.1

Documentation/Changelog.md

@@ -2,15 +2,15 @@
 
 These are the changes introduced in the next MaxScale version. This is not the official change log and the latest changelog can always be found in here: [MaxScale 1.1 Release Notes](Release-Notes/MaxScale-1.1-Release-Notes.md)
 
+## MaxScale 1.2
+* Logfiles have been renamed. The log names are now named error.log, messages.log, trace.log and debug.log.
+
 ## MaxScale 1.1.1
 
 * Schemarouter now also allows for an upper limit to session commans.
 * Schemarouter correctly handles SHOW DATABASES responses that span multiple buffers.
-* Schemarouter now allows disabling of the session command history.
-* Readwritesplit now allows disabling of the session command history.
-* Logfiles have been renamed. The log names are now named error.log, messages.log, trace.log and debug.log.
+* Readwritesplit and Schemarouter now allow disabling of the session command history.
 
-	
 ## MaxScale 1.1
 
 **NOTE:** MaxScale default installation directory has changed to `/usr/local/mariadb-maxscale` and the default password for MaxAdmin is now ´mariadb´.

Documentation/Documentation-Contents.md

@@ -34,7 +34,7 @@
  - [Filter Tutorial](Tutorials/Filter-Tutorial.md)
  - [Galera Cluster Connection Routing Tutorial](Tutorials/Galera-Cluster-Connection-Routing-Tutorial.md)
  - [Galera Cluster Read-Write Splitting Tutorial](Tutorials/Galera-Cluster-Read-Write-Splitting-Tutorial.md)
- - [MaxScale Information Schema Tutorial](Tutorials/MaxScale-Informaton-Schema.md)
+ - [MaxScale Information Schema Tutorial](Tutorials/MaxScale-Information-Schema.md)
  - [MySQL Replication Connection Routing Tutorial](Tutorials/MySQL-Replication-Connection-Routing-Tutorial.md)
  - [MySQL Replication Read-Write Splitting Tutorial](Tutorials/MySQL-Replication-Read-Write-Splitting-Tutorial.md)
  - [MySQL Cluster Setup](Tutorials/MySQL-Cluster-Setup.md)

Documentation/Release-Notes/MaxScale-1.1.1-Release-Notes.md

@@ -0,0 +1,93 @@
+# MaxScale Release Notes
+
+## 1.1 GA
+
+This document details the changes in version 1.1.1 since the release of the 1.1 GA Release of the MaxScale product.
+
+## New Features
+
+### Improved memory management options
+
+Readwritesplit and schemarouter now both support upper limits to session state modifying commands. They both also allow the complete disabling of the history, making the sessions consume the smallest amount of memory while still making sure all slaves keep identical session states.
+
+### Improved trace logging
+
+The process of the user authentication data retrieval is logged into the trace log and the readconnroute router also outputs more information into the trace log. This allows for easier problem detection and configuration tuning.
+
+### More informative output from maxkeys and maxpasswd
+
+Using the password functionality in MaxScale is now a lot easier. Both programs now produce verbose and exact error messages.
+
+## Bug Fixes
+
+Here is a list of bugs fixed since the release of the 1.1 version of MaxScale. The bug IDs are from the **[Mariadb Jira](https://mariadb.atlassian.net/)**.
+
+|Bug|Description|
+|---|-----------|
+|[MXS-99](https://mariadb.atlassian.net/browse/MXS-99)|/etc/init.d/maxscale reload doesn't do anything|
+|[MXS-83](https://mariadb.atlassian.net/browse/MXS-83)|linkage fails when system pcre library is recent|
+|[MXS-112](https://mariadb.atlassian.net/browse/MXS-112)|Disable saving of session commands in the readwritesplit and schemarouter modules|
+|[MXS-114](https://mariadb.atlassian.net/browse/MXS-114)|Disable recovery of disconnected slaves|
+|[MXS-73](https://mariadb.atlassian.net/browse/MXS-73)|MaxScale uses nearly 100% CPU |
+|[MXS-36](https://mariadb.atlassian.net/browse/MXS-36)|bugzillaId-671: wrong message if SHOW DATABASES privilege is missing|
+|[MXS-39](https://mariadb.atlassian.net/browse/MXS-39)|bugzillaId-731:Boolean configuration parameters accept inconsistent parameters|
+|[MXS-64](https://mariadb.atlassian.net/browse/MXS-64)|maxkeys and Maxpasswd do not produce informative error output|
+|[MXS-25](https://mariadb.atlassian.net/browse/MXS-25)|bugzillaId-656: MySQL Monitor: claims that Master is available after master failure|
+|[MXS-82](https://mariadb.atlassian.net/browse/MXS-82)|cmake warns when mariadb is compiled without mysql_release|
+|[MXS-69](https://mariadb.atlassian.net/browse/MXS-69)|dbfwfilter should be pessimistic about rule syntax errors|
+|[MXS-98](https://mariadb.atlassian.net/browse/MXS-98)|regexfilter log|
+|[MXS-28](https://mariadb.atlassian.net/browse/MXS-28)|bugzillaId-433: Logging don't include assert information|
+|[MXS-75](https://mariadb.atlassian.net/browse/MXS-75)|"wildcard" rule also blocks COUNT(*)|
+|[MXS-118](https://mariadb.atlassian.net/browse/MXS-118)|Two monitors loaded at the same time result into not working installation|
+|[MXS-33](https://mariadb.atlassian.net/browse/MXS-33)|bugzillaId-702: CLI: list services command shows negative values for the number of users of a service (Read Service).|
+|[MXS-17](https://mariadb.atlassian.net/browse/MXS-17)|bugzillaId-736: Memory leak while doing read/write splitting|
+|[MXS-30](https://mariadb.atlassian.net/browse/MXS-30)|bugzillaId-487: Buffer manager should not use pointer arithmetic on void*|
+
+## Known Issues
+
+There are a number bugs and known limitations within this version of MaxScale, the most serious of this are listed below.
+
+* The Read/Write Splitter is a little too strict when it receives errors from slave servers during execution of session commands. This can result in sessions being terminated in situation in which MaxScale could recover without terminating the sessions.
+
+* MaxScale can not manage authentication that uses wildcard matching in hostnames in the mysql.user table of the backend database. The only wildcards that can be used are in IP address entries.
+
+* When users have different passwords based on the host from which they connect MaxScale is unable to determine which password it should use to connect to the backend database. This results in failed connections and unusable usernames in MaxScale.
+
+* Binlog Router Plugin is compatible with MySQL 5.6  
+  Binlog Router Plugin currently does not work for MariaDB 5.5 and MariaDB 10.0 
+
+* LONGBLOG are currently not supported.
+
+* Galera Cluster variables, such as @@wsrep_node_name, are not resolved by the embedded MariaDB parser.
+
+* The Database Firewall filter does not support multi-statements. Using them will result in an error being sent to the client.   
+
+## Packaging
+
+Both RPM and Debian packages are available for MaxScale in addition to the tar based releases previously distributed we now provide
+
+* CentOS/RedHat 5
+
+* CentOS/RedHat 6
+
+* CentOS/RedHat 7
+
+* Debian 6
+
+* Debian 7
+
+* Ubuntu 12.04 LTS
+
+* Ubuntu 13.10
+
+* Ubuntu 14.04 LTS
+
+* Fedora 19
+
+* Fedora 20
+
+* OpenSuSE 13
+
+* SuSE Linux Enterprise 11
+
+* SuSE Linux Enterprise 12

Documentation/Tutorials/Administration-Tutorial.md

@@ -48,9 +48,9 @@ Options may be passed to the MaxScale binary that alter this default behavior, t
     <td>Use the filename passed as an argument instead of looking in $MAXSCALE_HOME/etc/MaxScale.cnf</td>
   </tr>
   <tr>
-    <td>-l<file>|<shm></td>
+    <td>-l<file>|<shm>|<stdout></td>
     <td>--log=</td>
-    <td>Control where logs are written for the debug and trace level log messages. the default is to write these to a shared memory device, however using the -lfile or --log=file option will forced these to be written to regular files.</td>
+    <td>Control where logs are written for the debug and trace level log messages. the default is to write these to a shared memory device, however using the -lfile or --log=file option will forced these to be written to regular files. Using -lstdout or --log=stdout will use the standard output for all enabled logs.</td>
   </tr>
   <tr>
     <td>-v</td>

log_manager/CMakeLists.txt

@@ -2,8 +2,8 @@ if(LOG_DEBUG)
   add_definitions(-DSS_LOG_DEBUG)
 endif()
 add_library(log_manager SHARED log_manager.cc)
-target_link_libraries(log_manager pthread aio stdc++)
+target_link_libraries(log_manager pthread aio stdc++ utils)
 install(TARGETS log_manager DESTINATION lib)
 if(BUILD_TESTS)
   add_subdirectory(test)
-endif()
+endif()

log_manager/log_manager.cc

@@ -1705,7 +1705,8 @@ static bool fnames_conf_init(
                 "-j <log path>       ............(\"/tmp\")\n"
                 "-l <syslog log file ids> .......(no default)\n"
                 "-m <syslog ident>   ............(argv[0])\n"
-                "-s <shmem log file ids>  .......(no default)\n";
+                "-s <shmem log file ids>  .......(no default)\n"
+                "-o                       .......(write logs to stdout)\n";
 
         /**
          * When init_started is set, clean must be done for it.
@@ -2599,14 +2600,14 @@ static bool logfile_init(
         }
 
 #if defined(SS_DEBUG)
-        if (store_shmem)
+        if (store_shmem && !use_stdout)
 	{
 		fprintf(stderr, "%s\t: %s->%s\n", 
 			STRLOGNAME(logfile_id),
 			logfile->lf_full_link_name,
 			logfile->lf_full_file_name);
 	}
-	else
+	else if(!use_stdout)
 	{
 		fprintf(stderr, "%s\t: %s\n", 
 			STRLOGNAME(logfile_id),
@@ -3153,7 +3154,7 @@ void flushall_logfiles(bool flush)
  */
 void skygw_log_sync_all(void)
 {
-	skygw_log_write(LOGFILE_TRACE,"Starting log flushing to disk.");
+	if(!use_stdout)skygw_log_write(LOGFILE_TRACE,"Starting log flushing to disk.");
 	flushall_logfiles(true);
 	skygw_message_send(lm->lm_logmes);
 	skygw_message_wait(lm->lm_clientmes);

macros.cmake

@@ -65,6 +65,9 @@ macro(set_variables)
   # Use tcmalloc as the memory allocator
   set(WITH_TCMALLOC FALSE CACHE BOOL "Use tcmalloc as the memory allocator")
 
+  # Use jemalloc as the memory allocator
+  set(WITH_JEMALLOC FALSE CACHE BOOL "Use jemalloc as the memory allocator")
+
   # Build tests
   set(BUILD_TESTS FALSE CACHE BOOL "Build tests")
 
@@ -259,20 +262,22 @@ debugmsg("Search returned: ${MYSQL_DIR_LOC}")
 
 
   # Check which init.d script to install
-  find_file(RPM_FNC functions PATHS /etc/rc.d/init.d)
-  if(${RPM_FNC} MATCHES "RPM_FNC-NOTFOUND")
-	find_file(DEB_FNC init-functions PATHS /lib/lsb)
-	if(${DEB_FNC} MATCHES "DEB_FNC-NOTFOUND")
-	  set(DEPS_OK FALSE CACHE BOOL "If all the dependencies were found.")
-	  message(FATAL_ERROR "Cannot find required init-functions in /lib/lsb/ or /etc/rc.d/init.d/, please confirm that your system files are OK.")
-	else()
-	  set(DEB_BASED TRUE CACHE BOOL "If init.d script uses /lib/lsb/init-functions instead of /etc/rc.d/init.d/functions.")
-	endif()
-  else()
-	set(DEB_BASED FALSE CACHE BOOL "If init.d script uses /lib/lsb/init-functions instead of /etc/rc.d/init.d/functions.")
+  if(WITH_SCRIPTS)
+    find_file(RPM_FNC functions PATHS /etc/rc.d/init.d)
+    if(${RPM_FNC} MATCHES "RPM_FNC-NOTFOUND")
+      find_file(DEB_FNC init-functions PATHS /lib/lsb)
+      if(${DEB_FNC} MATCHES "DEB_FNC-NOTFOUND")
+	set(DEPS_OK FALSE CACHE BOOL "If all the dependencies were found.")
+	message(FATAL_ERROR "Cannot find required init-functions in /lib/lsb/ or /etc/rc.d/init.d/, please confirm that your system files are OK.")
+      else()
+	set(DEB_BASED TRUE CACHE BOOL "If init.d script uses /lib/lsb/init-functions instead of /etc/rc.d/init.d/functions.")
+      endif()
+    else()
+      set(DEB_BASED FALSE CACHE BOOL "If init.d script uses /lib/lsb/init-functions instead of /etc/rc.d/init.d/functions.")
+    endif()
+    unset(DEB_FNC)
+    unset(RPM_FNC)
   endif()
-  unset(DEB_FNC)
-  unset(RPM_FNC)
 
   #Check RabbitMQ headers and libraries
   if(BUILD_RABBITMQ)

server/core/gateway.c

@@ -996,8 +996,8 @@ static void usage(void)
                 "  -c|--homedir=...  relative|absolute MaxScale home directory\n"
                 "  -f|--config=...   relative|absolute pathname of MaxScale configuration file\n"
 		"                    (default: $MAXSCALE_HOME/etc/MaxScale.cnf)\n"
-		"  -l|--log=...      log to file or shared memory\n"
-		"                    -lfile or -lshm - defaults to shared memory\n"
+		"  -l|--log=...      log to file shared memory or stdout\n"
+		"                    -lfile, -lshm or -lstdout - defaults to shared memory\n"
 		"  -s|--syslog=	     log messages to syslog."
 		" True or false - defaults to true\n"
 		"  -S|--maxscalelog= log messages to MaxScale log."
@@ -1064,6 +1064,7 @@ int main(int argc, char **argv)
         void*    log_flush_thr = NULL;
 	int      option_index;
 	int	 logtofile = 0;	      	      /* Use shared memory or file */
+	int	 logtostdout = 0;	      	      /* Use stdout for log output */
 	int	 syslog_enabled = 1; /** Log to syslog */
 	int	 maxscalelog_enabled = 1; /** Log with MaxScale */
         ssize_t  log_flush_timeout_ms = 0;
@@ -1198,6 +1199,8 @@ int main(int argc, char **argv)
 				logtofile = 1;
 			else if (strncasecmp(optarg, "shm", PATH_MAX) == 0)
 				logtofile = 0;
+			else if (strncasecmp(optarg, "stdout", PATH_MAX) == 0)
+				logtostdout = 1;
 			else
 			{
                                 char* logerr = "Configuration file argument "
@@ -1575,7 +1578,7 @@ int main(int argc, char **argv)
          */
         {
                 char buf[1024];
-                char *argv[8];
+                char *argv[9];
 		bool succp;
 		/** Set log directory under $MAXSCALE_HOME/log */
                 sprintf(buf, "%s/log", home_dir);
@@ -1605,8 +1608,17 @@ int main(int argc, char **argv)
 		}
 		logmanager_enable_syslog(syslog_enabled);
 		logmanager_enable_maxscalelog(maxscalelog_enabled);
-
-		if (logtofile)
+		if (logtostdout)
+		{
+		    			argv[3] = "-s"; /*< store to shared memory */
+			argv[4] = "LOGFILE_DEBUG,LOGFILE_TRACE"; /*< to shm */
+			argv[5] = "-l"; /*< write to syslog */
+			argv[6] = "LOGFILE_MESSAGE,LOGFILE_ERROR"; /*< to syslog */
+			argv[7] = "-o";
+			argv[8] = NULL;
+			succp = skygw_logmanager_init(8, argv);
+		}
+		else if (logtofile)
 		{
 			argv[3] = "-l"; /*< write to syslog */
 			/** Logs that should be syslogged */

server/core/maxkeys.c

@@ -33,18 +33,18 @@
 #include <log_manager.h>
 int main(int argc, char **argv)
 {
-    int arg_count = 4;
+    int arg_count = 6;
     char *home;
     char** arg_vector;
-    
+    int rval = 0;
 
 	if (argc != 2)
 	{
 		fprintf(stderr, "Usage: %s <filename>\n", argv[0]);
-		exit(1);
+		return 1;
 	}
 
-	arg_vector = malloc(sizeof(char*)*5);
+	arg_vector = malloc(sizeof(char*)*(arg_count + 1));
 
 	if(arg_vector == NULL)
 	{
@@ -52,8 +52,8 @@ int main(int argc, char **argv)
 	    return 1;
 	}
 
-	arg_vector[0] = strdup("logmanager");
-	arg_vector[1] = strdup("-j");
+	arg_vector[0] = "logmanager";
+	arg_vector[1] = "-j";
 
 	if ((home = getenv("MAXSCALE_HOME")) != NULL)
 	{
@@ -65,12 +65,12 @@ int main(int argc, char **argv)
 	    arg_vector[2] = strdup("/usr/local/mariadb-maxscale/log");
 	}
     arg_vector[3] = "-o";
-	arg_vector[4] = NULL;
+    arg_vector[4] = "-l";
+    arg_vector[5] = "LOGFILE_ERROR";
+	arg_vector[6] = NULL;
 	skygw_logmanager_init(arg_count,arg_vector);
 	skygw_log_enable(LOGFILE_TRACE);
 	skygw_log_enable(LOGFILE_DEBUG);
-	free(arg_vector[0]);
-	free(arg_vector[1]);
 	free(arg_vector[2]);
 	free(arg_vector);
 	
@@ -78,11 +78,11 @@ int main(int argc, char **argv)
 	if (secrets_writeKeys(argv[1]))
 	{
 		fprintf(stderr, "Failed to encode the password\n");
-		exit(1);
+		rval = 1;
 	}
 
 	skygw_log_sync_all();
 	skygw_logmanager_done();
 
-    return 0;
+    return rval;
 }

server/core/maxpasswd.c

@@ -41,18 +41,18 @@ int
 main(int argc, char **argv)
 {
 	char	*enc, *pw;
-	int	arg_count = 4;
+	int	arg_count = 6;
 	char	*home;
     char** arg_vector;
-	
+    int rval = 0;
 
 	if (argc != 2)
 	{
 		fprintf(stderr, "Usage: %s <password>\n", argv[0]);
-		exit(1);
+		return 1;
 	}
 
-    arg_vector = malloc(sizeof(char*)*5);
+    arg_vector = malloc(sizeof(char*)*(arg_count + 1));
 
 	if(arg_vector == NULL)
 	{
@@ -60,8 +60,8 @@ main(int argc, char **argv)
 	    return 1;
 	}
 
-	arg_vector[0] = strdup("logmanager");
-	arg_vector[1] = strdup("-j");
+	arg_vector[0] = "logmanager";
+	arg_vector[1] = "-j";
 
 	if ((home = getenv("MAXSCALE_HOME")) != NULL)
 	{
@@ -74,12 +74,12 @@ main(int argc, char **argv)
 	}
 
 	arg_vector[3] = "-o";
-	arg_vector[4] = NULL;
+	arg_vector[4] = "-l";
+	arg_vector[5] = "LOGFILE_ERROR";
+	arg_vector[6] = NULL;
 	skygw_logmanager_init(arg_count,arg_vector);
 	skygw_log_enable(LOGFILE_TRACE);
 	skygw_log_enable(LOGFILE_DEBUG);
-	free(arg_vector[0]);
-	free(arg_vector[1]);
 	free(arg_vector[2]);
 	free(arg_vector);
 	
@@ -87,7 +87,7 @@ main(int argc, char **argv)
 
 	if(pw == NULL){
 		fprintf(stderr, "Error: cannot allocate enough memory.");
-		exit(1);
+		return 1;
 	}
 
 	strncpy(pw,argv[1],80);
@@ -96,10 +96,11 @@ main(int argc, char **argv)
 		printf("%s\n", enc);
 	}else{
 		fprintf(stderr, "Failed to encode the password\n");
+		rval = 1;
 	}
 
 	free(pw);
 	skygw_log_sync_all();
 	skygw_logmanager_done();
-	return 0;
+	return rval;
 }

server/core/test/CMakeLists.txt

@@ -21,7 +21,7 @@ target_link_libraries(test_spinlock fullcore log_manager)
 target_link_libraries(test_filter fullcore)
 target_link_libraries(test_buffer fullcore log_manager)
 target_link_libraries(test_dcb fullcore)
-target_link_libraries(test_modutil fullcore)
+target_link_libraries(test_modutil fullcore utils log_manager)
 target_link_libraries(test_poll fullcore)
 target_link_libraries(test_service fullcore)
 target_link_libraries(test_server fullcore)

server/include/test_utils.h

@@ -8,11 +8,13 @@
 
 void init_test_env(char *path)
 {
-    int argc = 3;
+    int argc = 5;
     
     char* argv[] =
         {
             "log_manager",
+            "-l",
+            "LOGFILE_ERROR",
             "-j",
             path? path:TEST_LOG_DIR,
             NULL

server/modules/filter/regexfilter.c

@@ -172,7 +172,11 @@ char		*logfile = NULL;
 			else if (!strcmp(params[i]->name, "log_trace"))
 			    my_instance->log_trace = config_truth_value(params[i]->value);
 			else if (!strcmp(params[i]->name, "log_file"))
+			{
+			    if(logfile)
+				free(logfile);
 			    logfile = strdup(params[i]->value);
+			}
 			else if (!filter_standard_parameter(params[i]->name))
 			{
 				LOGIF(LE, (skygw_log_write_flush(
@@ -207,6 +211,7 @@ char		*logfile = NULL;
 		if (my_instance->match == NULL || my_instance->replace == NULL)
 		{
 			free(my_instance);
+			free(logfile);
 			return NULL;
 		}
 
@@ -218,6 +223,7 @@ char		*logfile = NULL;
 			free(my_instance->match);
 			free(my_instance->replace);
 			free(my_instance);
+			free(logfile);
 			return NULL;
 		}
 

server/modules/monitor/galera_mon.c

@@ -377,6 +377,8 @@ char 			*server_string;
 		int read_timeout = handle->read_timeout;
 		int write_timeout = handle->write_timeout;
 
+		if(database->con)
+		    mysql_close(database->con);
 		database->con = mysql_init(NULL);
 
 		rc = mysql_options(database->con, MYSQL_OPT_CONNECT_TIMEOUT, (void *)&connect_timeout);

server/modules/monitor/mm_mon.c

@@ -360,7 +360,8 @@ char 		  *server_string;
 		char *dpwd = decryptPassword(passwd);
                 int  rc;
                 int  read_timeout = 1;
-
+		if(database->con)
+		    mysql_close(database->con);
                 database->con = mysql_init(NULL);
 
                 rc = mysql_options(database->con, MYSQL_OPT_READ_TIMEOUT, (void *)&read_timeout);

server/modules/monitor/mysql_mon.c

@@ -401,7 +401,8 @@ char 		  *server_string;
                 int  connect_timeout = handle->connect_timeout;
                 int  read_timeout = handle->read_timeout;
                 int  write_timeout = handle->write_timeout;
-
+		if(database->con)
+		    mysql_close(database->con);
                 database->con = mysql_init(NULL);
 
                 rc = mysql_options(database->con, MYSQL_OPT_CONNECT_TIMEOUT, (void *)&connect_timeout);

server/modules/monitor/ndbcluster_mon.c

@@ -338,6 +338,8 @@ char 			*server_string;
                 int read_timeout = handle->read_timeout;
                 int write_timeout = handle->write_timeout;
 
+		if(database->con)
+		    mysql_close(database->con);
                 database->con = mysql_init(NULL);
 
                 rc = mysql_options(database->con, MYSQL_OPT_CONNECT_TIMEOUT, (void *)&connect_timeout);

server/modules/routing/readwritesplit/readwritesplit.c

@@ -4384,7 +4384,7 @@ static bool route_session_write(
 		    if(BREF_IS_IN_USE(bref))
 		    {
 
-			if(bref->bref_sescmd_cur.position <= prop->rses_prop_data.sescmd.position)
+			if(bref->bref_sescmd_cur.position <= prop->rses_prop_data.sescmd.position + 1)
 			{
 			    conflict = true;
 			    break;

utils/CMakeLists.txt

@@ -1,3 +1,2 @@
 add_library(utils skygw_utils.cc)
-add_dependencies(utils log_manager)
-target_link_libraries(utils stdc++ log_manager)
+target_link_libraries(utils stdc++)

utils/skygw_utils.cc

@@ -1978,8 +1978,11 @@ return_file:
 
 void skygw_file_free(skygw_file_t* file)
 {
-    free(file->sf_fname);
-    free(file);
+    if(file)
+    {
+        free(file->sf_fname);
+        free(file);
+    }
 }
 
 void skygw_file_close(