Comments on: scale-out: notes on sharding, unique keys, foreign keys… http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/ yet another MySQL DBA blog Thu, 10 Sep 2009 22:56:30 +0000 http://wordpress.com/ hourly 1 By: jurriaanpersyn.com — Database Sharding at Netlog, with MySQL and PHP http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-275 jurriaanpersyn.com — Database Sharding at Netlog, with MySQL and PHP Thu, 12 Feb 2009 13:04:32 +0000 http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-275 [...] oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys [...] [...] oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys [...]

]]> By: Karuna http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-271 Karuna Wed, 22 Oct 2008 10:00:32 +0000 http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-271 Please suggest on any specific sharding algorithm to implement the database sharding in mysql. Please suggest on any specific sharding algorithm to implement the database sharding in mysql.

]]> By: Sami http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-270 Sami Wed, 08 Oct 2008 20:47:55 +0000 http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-270 Ben, Could you please elaborate on how to make auto-incremented field globally unique? We are planning to implement sharding as a scaling solution for our database and having auto-incremented field globally unique is one of the requirements. Based on my understanding of your approach, It seems like I need to know ahead of time how many shards I want to add and then skip the count by that number. Is this correct? Or I missed something? Thanks, Sami Ben,

Could you please elaborate on how to make auto-incremented field globally unique? We are planning to implement sharding as a scaling solution for our database and having auto-incremented field globally unique is one of the requirements.

Based on my understanding of your approach, It seems like I need to know ahead of time how many shards I want to add and then skip the count by that number. Is this correct? Or I missed something?

Thanks,
Sami

]]> By: Andrew Mangold http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-261 Andrew Mangold Thu, 31 Jan 2008 20:53:23 +0000 http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-261 Ben, I have the following configuration: Two co-master InnoDB databases (Local number portability LNP) 175M rows, 50GB data 300-400K updates/day V5.1.22 using statement level replication and slaved to each other, only one database taking updates at any given time. I have a MySQL Cluster 5.1.23 housing data for an SCP. The SCP needs a subset of the LNP data (~10GB). I had planned on slaving the MySQL Cluster to the co-master InnoDB databases and using a Stored Trigger to drop unneeded columns as LNP replication transactions are applied to the MySQL Cluster. But...the carrier customer has decided to use MS SQL Server 2005 to house the database of record and requires that LNP transactions be feed from the InnoDB database to the MS SQL Server 2005 database and for the MS SQL Server to send update transactions to the MySQL Cluster. All of which needs to happen in real time. I was thinking about combining FreeTDS and the source for mysqlbinlog to build a utility to read the replication logs from InnoDB and apply them to MS SQL Server 2005. Not sure how to get replication transactions from Ms SQL 2005 server and apply them to MySQL Cluster. Thoughts??? Ben,
I have the following configuration: Two co-master InnoDB databases (Local number portability LNP) 175M rows, 50GB data 300-400K updates/day V5.1.22 using statement level replication and slaved to each other, only one database taking updates at any given time. I have a MySQL Cluster 5.1.23 housing data for an SCP. The SCP needs a subset of the LNP data (~10GB). I had planned on slaving the MySQL Cluster to the co-master InnoDB databases and using a Stored Trigger to drop unneeded columns as LNP replication transactions are applied to the MySQL Cluster. But…the carrier customer has decided to use MS SQL Server 2005 to house the database of record and requires that LNP transactions be feed from the InnoDB database to the MS SQL Server 2005 database and for the MS SQL Server to send update transactions to the MySQL Cluster. All of which needs to happen in real time. I was thinking about combining FreeTDS and the source for mysqlbinlog to build a utility to read the replication logs from InnoDB and apply them to MS SQL Server 2005. Not sure how to get replication transactions from Ms SQL 2005 server and apply them to MySQL Cluster. Thoughts???

]]> By: ben http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-239 ben Tue, 08 Jan 2008 05:43:06 +0000 http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-239 dsheardown - In our case, yes, we had a table in a small "global" database that contained accountid and shardids. So you'd look up an accountid and find out which shard that account was on. Of course, it's important to have this table's database backed up and probably replicated. If it is unavailable, that would make things difficult! We were able to move accounts to new shards by adding their data to a new shard then simply changing that one entry in the accountid/shardid table to point to the new shard. Thanks for reading! Ben dsheardown -

In our case, yes, we had a table in a small “global” database that contained accountid and shardids. So you’d look up an accountid and find out which shard that account was on. Of course, it’s important to have this table’s database backed up and probably replicated. If it is unavailable, that would make things difficult!

We were able to move accounts to new shards by adding their data to a new shard then simply changing that one entry in the accountid/shardid table to point to the new shard.

Thanks for reading!

Ben

]]> By: dsheardown http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-237 dsheardown Mon, 07 Jan 2008 07:40:09 +0000 http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-237 Hey, I am not really a DBA by any standard, although going through some design decisions about sharding with a new project. I have read countless discussions on how to effectively "scale out" and not "scale up". If I may ask a rather junior question: In the global database, do you simply have say a table with the userid/accountid and then some identify to say which shard DB/server to look at? Might sound rather rudimentary, but I guess you don't know if you don't ask! Hey, I am not really a DBA by any standard, although going through some design decisions about sharding with a new project. I have read countless discussions on how to effectively “scale out” and not “scale up”.

If I may ask a rather junior question:

In the global database, do you simply have say a table with the userid/accountid and then some identify to say which shard DB/server to look at?

Might sound rather rudimentary, but I guess you don’t know if you don’t ask!

]]> By: Scalable web architectures » Blog Archive » Sharding: Different from Partitioning and Federation ? http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-45 Scalable web architectures » Blog Archive » Sharding: Different from Partitioning and Federation ? Sun, 09 Sep 2007 20:39:53 +0000 http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-45 [...] Notes on sharding, unique keys, foreign keys [...] [...] Notes on sharding, unique keys, foreign keys [...]

]]> By: ben http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-17 ben Fri, 24 Aug 2007 18:36:19 +0000 http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-17 Joel - We're doing what you described. We have our lightweight database (we call it the global database) that associates users to shards. We're working on scripts to move users from one shard to another if it becomes desirable. Joel – We’re doing what you described. We have our lightweight database (we call it the global database) that associates users to shards. We’re working on scripts to move users from one shard to another if it becomes desirable.

]]> By: joel http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-16 joel Fri, 24 Aug 2007 14:39:06 +0000 http://oracle2mysql.wordpress.com/2007/08/23/scale-out-notes-on-sharding-unique-keys-foreign-keys/#comment-16 How are you partitioning your data? Just using a regular scheme to uniformly distribute the data? You might want to consider deciding how to partition stuff yourself. What happens if your ten busiest users are all on the same partition? You want a way to move users around from partition to partition to distribute the load. For that you'd want to keep a lightweight database which associates users to shards and just do a quick look up each time so that you know which shard to go to or use mysql proxy to do that automatically for you. How are you partitioning your data? Just using a regular scheme to uniformly distribute the data? You might want to consider deciding how to partition stuff yourself. What happens if your ten busiest users are all on the same partition? You want a way to move users around from partition to partition to distribute the load. For that you’d want to keep a lightweight database which associates users to shards and just do a quick look up each time so that you know which shard to go to or use mysql proxy to do that automatically for you.

]]>