Showing posts with label mysql. Show all posts
Showing posts with label mysql. Show all posts

Wednesday, February 9, 2011

Exploring timeout variables in Mysql

Few days back I was checking one of my mysql server's settings. I found that there are a number of timeout settings. Since I am using mysql for last few years, I am well aware of wait_timeout and connect_timeout variables. As if you are using mysql in production then in most of the cases you have to tune these two variables. But Mysql provides a number of other timeout variables also. 
 
If you run show variables like '%timeout' query then you will get a number of different timeout variables, in my case I got 9 such variables.

mysql> show variables like '%timeout';
+----------------------------+-------+
| Variable_name              | Value |
+----------------------------+-------+
| connect_timeout            | 10    |
| delayed_insert_timeout     | 300   |
| innodb_lock_wait_timeout   | 50    |
| interactive_timeout            | 28800 |
| net_read_timeout               | 30    |
| net_write_timeout          | 60    |
| slave_net_timeout          | 3600  |
| table_lock_wait_timeout    | 50    |
| wait_timeout               | 5     |
+----------------------------+-------+
9 rows in set (0.00 sec)

Why are there so many timeout variables, and what are their purpose?

MySQL uses different timeout variables at different stages. When a connection is just being established connect_timeout is used. When server waits for another query to be sent to it wait_timeout (or interactive_timeout for applications which specified they are interactive during connection). If query is being read or result set is being sent back, net_read_timeout and net_write_timeout are used. innodb_lock_wait_timeout is used with Innodb tables in case getting locks on table rows. delayed_insert_timeout is used in case you are using delayed insert queries. slave_net_timeout is used in case of replication when a slave is reading data from the master.

Lets look into some more details about these variables:

connect_timeout:
The number of seconds the mysqld server waits for a connect packet before responding with Bad handshake. As of MySQL 5.1.23 the default value is 10 seconds and before that it was 5 seconds. Increasing the connect_timeout value might help if clients frequently encounter errors of the form Lost connection to MySQL server.

delayed_insert_timeout:
You can delay insert queries from happening until the table is free by using the delayed hint in your SQL statement. For example:
INSERT DELAYED INTO table (id) VALUES (123);

The above SQL statement will return quickly, and mysql server will store the insert statement in a memory queue until the table you are inserting into is free from reads. The downside to this is that you don't really know how long its going to take for your INSERT to happen. INSERT DELAYED handler thread in Mysql server will wait for delayed_insert_timeout seconds before terminating.

innodb_lock_wait_timeout:
The timeout in seconds an InnoDB transaction may wait for a row lock before giving up. innodb_lock_wait_timeout applies to InnoDB row locks only. A MySQL table lock does not happen inside InnoDB and this timeout does not apply to waits for table locks. InnoDB does detect transaction deadlocks in its own lock table immediately and rolls back one transaction. The default value is 50 seconds. A transaction that tries to access a row that is locked by another InnoDB transaction will hang for at most this many seconds before issuing the following error:
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

interactive_timeout:
The number of seconds the server waits for activity on an interactive connection before closing it. An interactive client is defined as a client that uses the CLIENT_INTERACTIVE option to mysql_real_connect(). interactive_timeout is the amount of seconds during inactivity that MySQL will wait before it will close a connection for a interactive connection.

net_read_timeout:
The number of seconds to wait for more data from a connection before aborting the read. Before MySQL 5.1.41, this timeout applies only to TCP/IP connections, not to connections made through Unix socket files, named pipes, or shared memory. When the server is reading from the client, net_read_timeout is the timeout value controlling when to abort. net_read_timeout rarely becomes the problem unless you have extremely poor network, because in most cases query is generated and sent as single packet to the server and application can’t switch doing something else and leaving server with partial query received.

net_write_timeout:
When the server is writing to the client, net_write_timeout is the timeout value controlling when to abort. It defines the number of seconds to wait for a block to be written to a connection before aborting the write. Before MySQL 5.1.41, this timeout applies only to TCP/IP connections, not to connections made using Unix socket files, named pipes, or shared memory. If you do not fetch data for long enough MySQL Server may think client is dead and close connection. This well may happen if you need long processing for each row or have long periodic data flushes. Also, result set comes back in multiple pieces and if you're using mysql_use_result you can do any work between fetches, which potentially could take a lot of time.

slave_net_timeout:
The number of seconds to wait for more data from the master before the slave considers the connection broken, aborts the read, and tries to reconnect. The first retry occurs immediately after the timeout. The interval between retries is controlled by the MASTER_CONNECT_RETRY option for the CHANGE MASTER TO statement or --master-connect-retry option, and the number of reconnection attempts is limited by the --master-retry-count option. The default is 3600 seconds.

table_lock_wait_timeout:
As per the mysql manual this variable is not used.

wait_timeout:
The number of seconds the server waits for activity on a noninteractive connection before closing it. This timeout applies only to TCP/IP and Unix socket file connections, not to connections made using named pipes, or shared memory. On thread startup, the session wait_timeout value is initialized from the global wait_timeout value or from the global interactive_timeout value, depending on the type of client (as defined by the CLIENT_INTERACTIVE connect option to mysql_real_connect()). Setting a value too low may cause connections to drop unexpectedly. Setting a value too high may cause stale connections to remain open, preventing new access to the database. For wait_timeout, this value should be set as low as possible without affecting availability and performance.


Monday, February 7, 2011

Understanding MySQL Persistent Connections with mysql_pconnect()

What is mysql_pconnect():

The main purpose of using mysql_pconnect() function is to maintain a persistent connection to the mysql server. A persistent connection is a connection that do not get closed even after the excecution of the script is over which opened the connection. Even function mysql_close() can't close a persisitent connection. In contrast a normal connection opened using mysql_connect() gets closed either by mysql_close() or at the end of the scripts execution.

Why mysql_pconnect():

In one word the answer is "Efficiency". Persistent connections are good if the overhead to create a link to your db server is high. This overhead may be high due to various reasons. Persistent connections can help you considerably if the connection overhead is high.

How mysql_pconnect() works:

The most popular method to run PHP is to run it as a module in a multiprocess web server like Apache. A multiprocess server typically has one parent process which coordinates with a set of child processes. These child processes actually do the work of serving up web pages. When a request comes in from a client, it is handed over to one of the children that is free. This means that when the same client makes a second request to the server, it may be served by a different child process than the first time.

When opening a persistent connection, the function would first try to find a persistent link that is already open with the same host, username and password combination. If one is found, instead of opening a new connection an identifier for it will be returned. It causes the child process to simply connect only once for its entire lifespan, instead of every time it processes a page that requires connecting to the same db server. Every child that opened a persistent connection will have its own open persistent connection to the db server. For example, if you had 20 different child processes that ran a script that made a persistent connection to your db server, you would have 20 different connections to the db server, one from each child.

Issues with mysql_pconnect():
  • Persistent database connections don't necessarily reflect subsequent privilege changes.
  • Be very careful when using persistent connections and temporary tables on MySQL. With normal connections temporary tables are visible only to the current connection, but if you have a persistent connection the temporary tables will be visible to everybody sharing the same persistent connection. This can lead to major trouble.
  • Don't use mysql_pconnect() in situations where multiple MySQL servers are running on multiple ports of the same host. The connection pooling algo in php apparently only checks for the host, username and password combination but not the port. Therefore, if you use the same host, username and password but a different port, you might get a connection that is connected to a different port than the one you asked for.
  • Do not use transactions with persistent connections.  If your script stops or exits for any reason, your transaction will be left open and your locks will be left on.  You have to reset MySQL to release them. They won't rollback automatically on error, like they ought to. When you restart the script, you'll get a new connection, so you can't rollback or commit for the previous script.
  • You should be very careful when using LOCK TABLES with persistent connections. If the script terminates before the UNLOCK TABLES is executed, the the table(s) will stay locked, and very likely hang future scripts.

Things to remember:
  • Persistent connections were designed to have one-to-one mapping to regular connections. That means that you should always be able to replace persistent connections with non-persistent connections, and it won't change the way your script behaves. It may change the efficiency of the script, but not its behavior.
  • Any script with a start transaction, rollback, or commit SQL statement should use regular, not persistent connections.
  • Use totally random temporary table names when using persistent connections to avoid major problems.
  • Make damn sure that max connections limit in your my.cnf has a limit with a few more connections than your httpd.conf has for number of apache children. Less MySQL connections than apache children means some apache children will be starved for db.
  • Leave a few extra mysql connections, so that in case of a problem it can leave you with the ability to log in from shell to diagnose/fix it. Otherwise, you will have to bring down all of apache to get into your database.
  • You can also use register_shutdown_function() to register a simple cleanup function to unlock your tables or roll back your transactions. But it's better to avoid the problem entirely by not using persistent connections in scripts which use table locks or transactions.
  • Instead of use wait_timeout, you can set interactive_timeout to short period of time (for ex. 20 sec.) this is a lot better solution in apache + mysql environment than wait_timeout.

Wednesday, January 26, 2011

MongoDB vs MySQL: speed test part 2, Select queries

In the first part of this post I compared the performance of Insert operations for mongoDB and mysql. In this part I tried to compare the performance for different select operations. Test setup is same as used in the part 1.

1. Selects on an indexed column with different limit clauses

To check the performance of selects on an indexed column with limit clause, a number of queries with different limit clauses were executed on both the databases.

sample mysql query:
SELECT ID, NAME, BIRTH_DT, CONTACT_ADDRESS, CITY, TOTAL_EXP, ENTRY_DT, PROFILE, SUMMARY from db.resume where ID > 1000 limit 100000, 1000

sample MongoDB query:
$collection->find(array('ID' => array(':gt'=>1000)))->skip(100000)->limit(1000);













Start LimitTotal Records FetchedMySQLMongoDB
0 10000.846 ms0.0710ms
10000010000.903 ms0.0391ms
20000010000.969 ms0.0209ms
30000010001.029 ms0.0889ms
40000010001.058 ms0.0488ms
50000010001.149 ms0.0482ms
60000010001.214 ms0.0460ms
70000010001.170 ms0.0469ms
80000010001.196 ms0.0450ms
90000010001.216 ms0.0460ms

2. Selects on a non indexed column with different limit clauses

sample mysql query:
SELECT ID, NAME, BIRTH_DT, CONTACT_ADDRESS, CITY, TOTAL_EXP, ENTRY_DT, PROFILE, SUMMARY from db.resume where TOTAL_EXP > 5 limit 100000, 1000

sample MongoDB query:
$collection->find(array('TOTAL_EXP' => array(':gt'=>5)))->skip(100000)->limit(1000);








Start LimitTotal Records FetchedMySQLMongoDB
0 10001.133 ms0.0679 ms
10000010001.166 ms0.0469 ms
20000010001.334 ms0.0469 ms
30000010001.293 ms0.0438 ms
40000010002.047 ms0.0450 ms


3. Selects on an indexed column with sorting

sample mysql query:

SELECT ID, NAME, BIRTH_DT, CONTACT_ADDRESS, CITY, TOTAL_EXP, ENTRY_DT, PROFILE, SUMMARY from db.resume where ID > 1000 order by USERNAME asc

sample MongoDB query:

$collection->find(array('ID' => array(':gt'=>1000)))->sort(array("USERNAME"=>1));

Avg time in Mysql : 1.973 sec
Avg time in MongoDB : 0.138 ms

4. Selects with IN clause on an Indexed Column

To check the performance of select queries on an indexed key with IN clause, a number of queries were executed on both the databases and an avearge is taken. Each query had 100 random ID values in the IN clause.

Avg time in Mysql : 4.865 ms
Avg time in MongoDB : 1.570 ms

Its clear from the above results that MongoDB outperformed mysql in each case by a large margin.


MongoDB vs MySQL: speed test part 1, Insert queries

Recently I started exploring NoSQL databases as an alternative for some of our high traffic mysql tables. After going through a number of articles on net, I decided to explore MongoDB. I tried to compare the performance of different database operations (inserts/ different types of selects) in mongoDB and in MySQL. Performance comparison of insert operations are given here.

Test setup:
For testing I used a 3.16 GHz, Intel Xeon CPU with 2 GB of memory and 350 GB of disk.

MySQL:

key_buffer = 128M
sort_buffer_size = 512K
read_buffer_size = 256K
max_allowed_packet = 1M

Table schema:

ID int(11)
NAME varchar(35)
BIRTH_DT date
CONTACT_ADDRESS varchar(150)
CITY int(11)
TOTAL_EXP varchar(5)
ENTRY_DT date
PROFILE varchar(250)
SUMMARY varchar(250)

MongoDB:

For mongo two shard servers, one config server and one mongos were satarted on the same machine with chunk size set to 10.

Sample document:

{"_id" : ObjectId("4ca6cca6a87305c90b000000"),
"ID" : "5839427",
"NAME" : "Gaurav Asthana",
"BIRTH_DT" : "1981-06-29",
"CONTACT_ADDRESS" : "Noida, India",
"CITY" : "19",
"TOTAL_EXP" : "06.10",
"ENTRY_DT" : "2010-11-26",
"PROFILE" : "zxzzzzz zzzzzzzzzz zzzzzzzzzzzzzzzzzz zzzzzzzz",
"SUMMARY" : "abcfsf fsdfs gdgdfg gdfgdh dfghdh dfhdh" }

An index is also created on ID field.

I have created a simple php script to perform the benchmark. This script inserted a total of 15 Lac records both in the mysql and mongodb. I have recorded time for each batch of 100 records that were inserted. So, in total I recorded 15000 readings. The average time taken by both the databases is given below.

Average time per batch of 100 records :

Mysql : 18.77 ms
MongoDB : 5.53 ms

Size on disk:

Mysql : 390 MB
MongoDB : 1.6 GB

In my benchmark, MongoDB came out three times faster that mysql in case of insert queries. But it occupied four times more disk than mysql.

Saturday, November 29, 2008

PostgreSQL vs MySQL

We have been using Mysql for last serveral years. Bur from some time now we are facing scalability issues with Mysql as our tables are growing fast and no. of queries on these tables are increasing day by day. While looking the possible solutions many suggested to use PostgreSql. So first thing that came to my mind was in what ways Pgsql is better than Mysql. I searched the web, asked this question to my network and here is some of the findings...

Pgsql is a much more feature complete SQL engine.

PostgreSQL does not have an unsigned integer data type, but it has a much richer data type support (including BOOLEAN, IP addresses, UUIDs, and such), user-defined data types mechanism, built-in and contributed data types.

Both PostgreSQL and MySQL support Not-Null, Unique, Primary Key and Foreign Key constraints. MySQL doesn't support the Check constraint while PostgreSQL has supported it for a long time. PostgreSQL's base data types are much more consistent about enforcing data integrity, even in the absence of constraints and foreign keys. Thus, "NOT NULL" really means that NULLs are forbidden, you can't have "February 29th" in other than leap years, "blank" isn't automagically transmogrified into a "zero", and such.

PostgreSQL can compress and decompress its data on the fly with a fast compression scheme to fit more data in an allotted disk space. MySQL's high performance storage engines do not support on the fly compression as of 5.1. MySQL 6.0 will support on the fly compression with its Falcon storage engine

MySQL's MyISAM engine performs faster than PostgreSQL on simple queries and when concurrency is low. MyISAM's speed comes at the cost of not supporting transactions, foreign keys, and not offering guaranteed data durability.

MySQL's count(*) is really fast. PostgreSQL count(*) is very slow because instead of counting rows using an index scan, it goes through the entire table sequentially.

MySQL supports INSERT IGNORE and REPLACE statements. PostgreSQL supports neither of these statements and suggests using stored procedures to get around the lack of these statements.

PostgreSQL's speed advantage over MySQL can be seen drastically in a large multi-core/processor environment. PostgreSQL scales much better, both in terms of using up scale hardware, and dealing with concurrency. MySQL, on the other hand, focuses on scale out technologies and the use of off the shelf commodity hardware.

PostgreSQL is fully ACID-compliant, while MySQL's InnoDB storage engine provides engine-level ACID-compliance.

PostgreSQL supports Partial and Bitmap Indices. MySQL has no bitmap indices (but achieves similar functionality using its "index_merge" feature) and partial indices (MySQL supports partial indexing using the InnoDB engine, but not with the MyISAM engine).

A PostgreSQL trigger can execute any user defined function from any of its procedural languages, not just PL/pgsql. PostgreSQL also supports "rules" which allow operating on the query syntax tree, and can do some operations more simply that are traditionally done by triggers.

MySQL has built-in replication, PostgreSQL is modular by design, and replication is not in the core. There are several packages that allow replication in PostgreSQL.

PostgreSQL makes a much better impression, from administration perspective. Backup and replication features are more advanced, many features that MySQL is going to have (?) like point in time recovery , multiple replication slaves , better support for foreign keys, cursors and stored procedures are all available in PostgreSQL already.

MySQL is an open-source product. Postgres is an open-source project. MySQL community is more active and enthusiastic than PostgreSQL's and the MySQL documentation (books, blogs etc) are way more and up to date.