Categoria: MySQLfs

MySQLfs 0.4.1

Hello everybody.

I’m pleased to announce the release 0.4.1 of MySQLfs. This is the first new official version after a long inactivity so please handle it with care. Furthermore this is my first release so, although I have double checked everything, yet I may have done some tremendous mistake.

These are main improvements in this version:

  • InnoDB usage instead of MyISAM
  • Basic transaction support
  • Upgrade to FUSE API 2.6
  • Enabled support for “big_writes” to speed up FS operations
  • New datablock size
  • FreeBSD (FUSE4BSD) support “out-of-the-box”
  • Support for new FUSE 2.6 API functions:
    • fuse::create – create a file
    • fuse::statfs – returns stats about file system usage (needed for df and such)
    • corrected used block count (needed for du and such)
  • Fixed command line issues: now you can use -obig_writes, -oallow_other (to allow other users to read the mounted filesystem) and -odefault_permissions (which, per this version, is mandatory when using mysqlfs under FreeBSD)

Please note that this is not a production-ready version (yet), but I ask you to test it wildly and please report all the issues that you may have. I’ll try to fix them.

You can download the package here: mysqlfs-0.4.1.tar (232kb).

PLEASE NOTE THAT THE DATABASE SCHEMA HAS BEEN CHANGED FROM 0.4.0 TO 0.4.1!

If your plan is to upgrade from a previous installation my suggestion is to compile the new version alongside the old one, create a new, separated FS, mount the new FS and then copying the datas from the old FS to the new one.

If you really need to do a live upgrade of an 0.4.0 database please take a look at the (unrecommended and incomplete!) upgrade script in the sql subdir.

Installation

To install mysqlfs just make sure you have installed fuse and all it’s libs, plus mysql and all his devel libraries, unpack the tar.gz and just run

./configure
make
make install (as root)

Then create a database with proper permissions and use the file schema.sql in the sql dir to create the database definitions.

Run mysqlfs –help to see al the available options.

You’re done. Have fun.

MySQLfs “AB” public repo

So, actually it was easier than I thought. Here’s a public repo for my updated MySQLfs: https://github.com/skeyby/mysqlfs

It’s a work in progress and I wrote my first C statement 10 days ago, so please be kind 😉

Contributors are welcome. As you see I cloned an already existing repository in order to have all the history imported but didn’t start from the latest commit as I found the new added function a bit confused as of now (well, maybe it’s just me who can’t really understand them…), but I plan to merge them back later as I fix the things that need more attention in my environment right now.

Have fun.

MySQLfs updates

I already spent some talk on this previously but I wrote in Italian, so let’s do a little recap for English readers.

Just recently I became involved in a project where a cluster of machine had to replicate their datas constantly in an active-active fashion and with geographical distribution.

We checked different kinds of solutions based either on drbd or zfs or hast or coda or… Well there’s a lengthy post on this issue just a few posts before this one, so I suggest you checking that.

At the end of our comparison we found the solution that suited us best to be mysqlfs. So I started investigating on that and quickly found some issues that could be improved.

Main points were:

  • performances, as mysqlfs turned out to be pretty slow under certain circumstances
  • transaction awareness
  • better integration with mysql replication

As I digged in the code, I found a pretty good general infrastructure but quite frankly I don’t think mysqlfs was really ever used in a production environment.
Apart from that, the project was quite young (latest official version was 0.4.0) and also pretty “static”, with it’s latest release dating back in 2009.

So, without any knowledge of C or Fuse whatsoever I took the sources of the latest “stable” release and began experimentating with it.

A few weeks has passed and I think I reached a very interesting point. Those are the goal that I reached as of now:

  • mysqlfs is now using InnoDB instead of MyIsam
  • all the writing operations are now enclosed into a transaction that gets rolled back if something bad happens halfway
  • using transactions also means better replication interoperability, since innodb and the binary log don’t fight for the drives. Innodb first, bin log after.
  • mysqlfs now uses fuse API version 2.6 instead of the old… Mmh… 2.5 I think.
  • using fuse’s API version allowed switching on fuse “big writes”, the switch that allow the kernel and the file system to exchange data blocks bigger than standard 4k
  • mysqlfs internal data block was changed from 4k to 128k either to reduce the blocks fragmentation, to reduce the rows in the “data block” table, to reduce the inserts, and finally to match the big write setting. Receiving 128kb of datas and then pushing 4k at time in the db wouldn’t really make any sense
  • moving to the 2.6 API allowed mysqlfs to be FreeBSD compliant. I don’t know (yet) about Mac OS or anything else, but having the devil in the party is a good plus for me.
  • today I started working on the new file system function that got introduced in e latest versions of fuse, that should to even better to speed and such

Next steps are…

  • implement file and inode locking
  • implement some kind of mechanism to do somewhat write-thru cache. Many basic system commands (cp, tar, gzip) use very small writing buffers (8/16k) so the impact of the big write switch get lost and the performances degrade a lot. I’m a bit afraid this task is above my (null) C knowledge but I have some interesting ideas in mind…
  • implement the missing functions from fuse
  • introduce some kind of referential integrity in the db, although I have to understand the performance downsize
  • introduce some kind of internal data checksum, although everything comes at a price in terms of CPU time
  • introduce some kind of external, let’s say php for example, API, to allow direct php applications to access directly the file system, while it’s mounted, maybe in different machines at the same time, without having to use the file system-functions… Wouldn’t be cool to have a web app, a Linux server and a FreeBSD server all working on the same file system at the same time? Yes I know, I’m insane 🙂
  • improve replication interoperability introducing server signatures in stored datas
  • introduce some kind of command line tools to interact with the db and check total size usage and such
  • in a long future introduce some kind of versioning algorithm for the stored datas

Right now the modification I made to mysqlfs aren’t public yet, as I couldn’t really understand the status of the project on sourceforge. Furthermore it’s code base on sourceforge is not aligned to the version that was on the installer (the one I started working on), and it’s stored in svn that I don’t absolutely know how to use.

I know the latter ones aren’t big issue, but my spare time is very thin, and I definitely can’t waste time in learning svn or manually merging the “new” code that’s in the svn and that is different from the tgz I started from. If any of you is willing to do it I’ll be glad to help.

My internal git repository isn’t public yet because of laziness and because I’d like to publish something that’s at least usable and I have to fix a couple of problems before it could be defined “idiot proof”.

In the meantime if any of you is willing to try I’d be very glad to share the modified code, or if any of you is willing to contribute then I’d be twice as glad as long as we try to keep development a bit aligned.

Cheers