« Post 2 | This is the 3rd post in my MoFo Futures 2009 blog series.

"As for the future, your task is not to foresee it, but to enable it." — Antoine de Saint-Exupéry

I like beginnings. This is a constant. In primary school, I tried to re-invent arithmetic. My D&D characters were be obsessively re-rolled and scrapped. On various computers, I'd start my games of Civilization over and over. I'd hope that if I could just get the beginning right that the rest wouldn't be so hard or boring.

There were three other constants in my life during my teen years: a loving family, a long-running D&D group and near-complete loathing of school. Before I hit twenty, these three constants had dwindled down to one. While my family remained (and remains) wonderful, but my D&D group evaporated - a dozen or so friends each fumbling their separate ways into life - and I had managed to drop out of school not once, but twice.

It wasn't much change but it wasn't much of a life and I still found myself adrift. I wandered from job to job, holding six different positions in the space of two years. Each time I'd quit in frustration and go looking for a new beginning to try and get right.

In early '94, financial pressures mounting, I landed a job at a dental lab and clung to it for dear life. I liked the work. I made friends. I even survived my first evaluation. Thankfully, I had a kind boss who turned me into a project. It was a good job. Some years later, I even met my lovely wife there. So, by late '94, I was pretty sure that I would be a dental technician. Full stop. For the foreseeable future, I would spend my nights playing D&D and my days crafting false teeth while wearing a turquoise smock. Life would be good.

Then the province of Alberta drastically cut funding for senior's dental care and, with it, much of the profit involved in running a dental lab. Waves of layoff swept the industry and, as lifelong staffers at the lab were let go, I was sure that I would be next.

One night in January, while I moped over my coming unemployment, my mother saw a segment on the local news about the Information Superhighway. The idea of the Net set a fire in her which quickly spread to the rest of the family.

I hardly remember the six months that followed — they are still a blur of days at the dental lab (I never did get laid off) and of nights exploring the Net with my family. As best I can recall, it went like this:

Dream. Dream a lot. Get quotes on web page development. Beat a hasty retreat from $10,000/page. Scrape some cash together. Buy a second-hand 286. Buy a modem. Find cheap Web access via a local BBS. Spend three days figuring out how to actually get to a website. Surf the awesomeness of the textual web with Lynx. Fail to understand the HTML spec. Buy a second-hand 386. Get Netscape Navigator. Freak out at the total awesomeness of the graphical Web. Discover View Source. Lose mind. Build hideously malformed web pages stitched together from the HTML sources of Netscape's site, razorfish.com and hotwired. Work ourselves into a bleary-eyed, sore-fingered, sweaty, sleep-deprived lather. Repeat several times. Finally, in June of 1995, publish an online travel magazine with 20-something articles and, somewhat oddly, a satirical paperdoll of Canadian politician Preston Manning.

The first few weeks after publishing, we'd refresh our log files every few minutes to see who had visited, cheering as people worked their way from article to article. Good times.

I hold much fondness for this year. It still feels like we just wandered up to the side of the information superhighway and stuck out our thumbs, waiting in the bright sunshine for the future to pull up in a fast car. And when she pulled up, we were so naïve that we weren't even surprised — with a nod of our heads we just got in.

« Post 2 | This is the 3rd post in my MoFo Futures 2009 blog series.

Sometimes one has weird ideas, or am I the only one? - This specific one is at least a year old, now, during the Christmas days, waiting for New Year's Eve I had the time and mood to finally try it out: MySQL 5.1 has a plugin interface to easily add storage engines. PHP can easily embedded into other applications.  So why not combine these two things? - Writing a MySQL Storage Engine which reads data by calling a PHP script.

Let's start with a simple example first:

<?phpfunction create_table($table, $data) {
    return true;
}

function open_table($table) {
    return new ArrayIterator(array(
        array('id' => 1, 'dat' => 'foo'),
        array('id' => 2, 'a' => 'bar')
    ));
}
?>

This is the bare minimum storage engine my plugin supports. create_table() is called for creating the table, open_table() to access it, the later one then returns an iterator which is used for a full table scan. This example uses an ArrayIterator, which implements the SeekableIterator and the Countable interfaces, the first one provides a seek() method, which is called to read specific rows after sorting for instance, the later provides a method count() which gives the optimizer a hint.

Let's use this table:

mysql> CREATE TABLE php_test (id int, val CHAR(3)) ENGINE=PHP;
Query OK, 0 rows affected (0.04 sec)

mysql> SELECT * FROM php_test;
+------+------+
| id | val |
+------+------+
| 1 | foo |
| 2 | bar |
+------+------+
2 rows in set (0.00 sec)

Ok, of course that's nice and shiny but well, it's read only. To solve that you can implement a few interfaces provided by the plugin to handle writes:

<?php
class Test extends ArrayIterator 
implements MySQLStorage_Writable, MySQLStorage_Updatable, MySQLStorage_Deletable {
    public function write($data) {
        $this[] = $data;
    }

    public function update($data) {
        $this[$this->key()] = $data;
    }

    public function delete() {
        unset($this[$this->key()]);
    }
}

function create_table($table, $data) {
    return true;
}

function open_table($table) {
    return new Test(array(
        array('id' => 1, 'dat' => 'foo'),
        array('id' => 2, 'a' => 'bar')
    ));
}
?>

Again, we can test it:

mysql> UPDATE php_test SET val = 'baz' WHERE id = 1;
Query OK, 1 row affected (0.02 sec)
Rows matched: 1 Changed: 1 Warnings: 0

mysql> DELETE FROM php_test WHERE id = 2;
Query OK, 1 row affected (0.00 sec)

mysql> INSERT INTO php_test VALUES(3, 'bar');
Query OK, 1 row affected (0.00 sec)

mysql> SELECT * FROM php_write;
+------+------+
| id | val |
+------+------+
| 1 | baz |
| 3 | bar |
+------+------+
2 rows in set (0.00 sec)

As a reminder: This is calling PHP for all these operations.

So what might real life use cases be, once the major issues in the code are fixed? I have a few ideas like

• A live-logfile query tool, not sure that's really need, see the primitive Apache httpd access_log parser which is provided with the code as an example
• Combine it with the embedded MySQL server and use this storage engine for your unit tests, "mock tables" ...

Any other ideas? - Leave a comment

Oh, and like most MySQL stuff nowadays: There's a launchpad project for this plugin.

I've created a Twitter for Drizzle at http://twitter.com/drizzlebugs

Follow it for announcements, bugs reports, and other Drizzle related stuff

Ok this isn't very timely reporting, but about two weeks ago Batched Key Access feature has been pushed into MySQL 6.0. You can get it from the bazaar repo now (bzr branch lp:mysql-server/6.0), or wait several more weeks till MySQL 6.0.9 is released and get it from there.

Batched Key Access in a nutshell

BKA is about accessing tables in batches when running nested loop joins. The benefits of batching table accesses are that

• "Remote" engines save on number of roundtrips
• Disk-based engines do reads in disk order instead of randomly probing the table, which allows to be easier on disk cache and take advantage of prefetching

Batched Key Access only works if the used storage engine supports it. At the moment there is support for MyISAM, InnoDB, Maria, Falcon (these are disk-based) and NDB (this one is remote) engines.

Documentation

At the moment there's no manual chapter yet. There is a short introduction at Batched_Key_Access page on the forge and there are MySQL Conference 2008 session slides. The slides cover some benchmarking and give an idea about what kind of queries and dataset you need to get speedups with MyISAM/InnoDB. We've seen great speedup with NDB also but didn't publish anything so far.

Observation

With Batched Key Access and condition pushdown, it is now feasible to create a remote table engine with decent performance. We have a remote engine, ha_federated and it doesn't support BKA or condition pushdown and is a death by latency if you have queries that do not match the
SELECT * FROM table WHERE primary_key=const

pattern. I have a strong temptation to code a performance version of ha_federated myself, but have to resist it as there is subquery optimization work to be finished and optimizer "bugs" to be addressed.

This is now a rather low-hanging fruit, any takers?

One of our customers gave me a chance to run some benchmarks on 24-core (intel cpu based) server, and I could not miss it and ran few CPU-bound tasks there.

The goal of benchmarks was investigation of InnoDB-plugin and XtraDB scalability in CPU-bound load.

CPU specification:

PLAIN TEXT CODE:

1. processor       : 23
2. vendor_id       : GenuineIntel
3. cpu family      : 6
4. model           : 29
5. model name      : Intel(R) Xeon(R) CPU           E7450  @ 2.40GHz
6. stepping        : 1
7. cpu MHz         : 2394.011
8. cache size      : 12288 KB
9. physical id     : 3
10. siblings        : 6
11. core id         : 5
12. cpu cores       : 6
13. fpu             : yes
14. fpu_exception   : yes
15. cpuid level     : 11
16. wp              : yes
17. flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
18. bogomips        : 4788.59
19. clflush size    : 64
20. cache_alignment : 64
21. address sizes   : 40 bits physical, 48 bits virtual
22. power management:

I tested MySQL-5.1.30 with InnoDB plugin, Xtradb-1.0.2-1, and XtraDB-1.0.2-2 (rel2). XtraDB-rel2 has not been released yet, we still are doing internal QA for, but it will be ready soon. Main difference XtraDB-rel2 it contains additional scalability fixes for buffer_pool (split_buffer_pool_mutex patch).

my.cnf is

PLAIN TEXT CODE:

1. [mysqld]
2. user=root
3.  
4. binlog_cache_size=1M
5. default_table_type=MYISAM
6. ft_min_word_len=4
7.  
8. innodb_additional_mem_pool_size=16M
9. innodb_buffer_pool_size=15G
10. innodb_data_file_path=ibdata1:10M:autoextend
11. innodb_file_per_table=1
12. innodb_flush_log_at_trx_commit=2
13. innodb_log_buffer_size=8M
14. innodb_log_files_in_group=2
15. innodb_log_file_size=512M
16. innodb_status_file=0
17. innodb_thread_concurrency=0
18.  
19. innodb_io_capacity=1000
20. innodb_write_io_threads = 16
21. innodb_read_io_threads = 16
22.  
23.  
24. join_buffer_size=1M
25. max_allowed_packet=1M
26. max_connections=3000
27. max_heap_table_size=64M
28. max_prepared_stmt_count=1000000
29. query_cache_size=0
30. skip_grant_tables
31. skip_locking
32. sort_buffer_size=64K
33. table_cache=2048
34. thread_cache_size=16
35. thread_concurrency=16
36. thread_stack=196K
37. tmp_table_size=64M
38. transaction_isolation=REPEATABLE-READ
39. local-infile=1

At first I tried sysbench oltp read-only with 10mil rows (the datasize is about 2.5GB), uniform distribution.
The results you can see there: (Results are removed for additional checking)

In the next run I tested sysbench oltp read-write load, and the results are:

Here starting 16 threads the result is dropping down with the same speed as it grew, and with 128 connections we have the same TPS as with 1 connection (and it is on 24-cores box!). XtraDB is slightly better there than InnoDB, but nothing special to be proud of. We definitely we will look how to fix it as next step, read detailed investigation what is the reason of performance drop in next post.

And last one workload I tried is TPCC-like benchmark (you can get it on https://launchpad.net/perconatools), with 100 Warehouses (about 9.5GB datasize).
The result is:

Here the result grows up to 16 connections, but after that InnoDB-plugin is dropping down. XtraDB and XtraDB-rel2 seem quite better, I guess this is mostly because fixes to rw_locks and to buffer_pool mutex (in rel2).

Conclusion: As read-only workload seems fine, read-write cases is something to worry about in 16+ cores boxes.
Intel Based 24 Core Servers are not mainstream these days but as number of cores is increasing now at the same pace as CPU frequency before we believe they are very soon to come. Also in real production there may be not a lot database fits "in-memory" cases - but on other hand 64-128GB RAM per box is not something rare already and recommendation to fit at least active dataset in memory is one we use for our customers.

Entry posted by Vadim | No comment

Add to: | | | |

The other day, a client mentioned they were getting strange results when running ALTER TABLE. The episode involved modifying an existing primary key to add an auto_increment primary key: it was “shifting” values. Say what?!

As it turns out, it was a very special value getting changed: zero. Some fiddling revealed the underlying reason. Care to join me?

To understand what’s going on, follow the example below as we start with an empty database, create a table and insert a few rows:

mysql> use test; Database changed mysql> show tables; Empty set (0.00 sec) mysql> create table test_table (id int not null primary key) engine=innodb; Query OK, 0 rows affected (0.01 sec) mysql> desc test_table; +-------+---------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+---------+------+-----+---------+-------+ | id | int(11) | NO | PRI | | | +-------+---------+------+-----+---------+-------+ 1 row in set (0.00 sec) mysql> insert into test_table (id) values (1); Query OK, 1 row affected (0.00 sec) mysql> insert into test_table (id) values (2); Query OK, 1 row affected (0.00 sec) mysql> insert into test_table (id) values (0); Query OK, 1 row affected (0.00 sec) mysql> select * from test_table; +----+ | id | +----+ | 0 | | 1 | | 2 | +----+ 3 rows in set (0.00 sec) mysql>

Now let’s change our PK and make it auto_increment (more…)

By now you have been looking at the UC’s upcoming schedule. On Tuesday @ 2pm Yves and I are currently scheduled to present our presentation on Waffle Grid entitled: “Distributed Innodb Caching with memcached “ . When we submitted this topic, we had not yet come up with a name for the project, So we are really hoping one of the conference gods allows us to change it to something like “The Waffle Grid Project: Distributed Innodb Caching with memcached”. Speaking of changes something, my other accepted proposal a talk on Solid State Disk scheduled for Wednesday @ 2pm Entitled “SAN Performance on a Internal Disk Budget: The coming Solid State Disk revolution” (http://en.oreilly.com/mysql2009/public/schedule/detail/5991) went horribly wrong in the formatting department when I got lazy and copied the outline of a presentation I gave earlier this year on that topic. I really need to figure out who can change that to something more legible. Or something like:

 

“We will spend 45 minutes discussing the past, present, and future of Solid State Disk technologies. Specifically we will look at the performance of drives from Mtron, Memoright, and Intel. Paying special attention to DBT2, Sysbench, Orion and other disk based benchmarks. I will also touch on other technologies from Fusion IO, Texas Memory systems, Violin memory and others to compare and contrast various companies approaches to speeding up the disk IO layer. “

Oracle Database 10g offered two new Flashback features that allowed an authorized user to see all versions of any changes made to one or more rows in a table. Database 11gR1 provides the ability to back out one or more independent or dependent transactions with Flashback Transaction Backout.

Laurel Ruma (@laurelatoreilly) just did a quick census of the number of O'Reilly employees on twitter. She came up with 74 twitter accounts out of about 300 employees worldwide, plus five people who were controlling departmental or project-based O'Reilly twitter accounts like the following:

Official O'Reilly account:
@oreillymedia: The top level O'Reilly Media site.


@oreilly_verlag: O'Reilly Germany

Number of O'Reilly products or divisions on Twitter: 8


@make: Make: Magazine and makezine blog

@craft: Craft: Magazine and craftzine blog

@hacks: Hacks book series and hackszine blog

@insideria: Our Inside RIA microsite sponsored by Adobe.

@missingmanuals: The Missing Manuals

@headfirstlabs: Head First book series

@tocTools of Change for Publishing conference and blog

@radar: The O'Reilly Radar blog

Number of O'Reilly conferences on Twitter: 12

@oscon: The O'Reilly Open Source Convention

@etech: The O'Reilly Emerging Technology Conference

@moneytech: Money:Tech

@foundconf: Found: The Search Acquisition and Architecture Conference

@where20: Where 2.0

@railsconf: RailsConf

@MySQLconf: The MySQL User Conference

@web2summit: The Web 2.0 Summit

@RailsConfEU: RailsConf EU

@w2esf09: Web 2.0 Expo SF

@w2e09: Web 2.0 Expo NY

@velocityconf: Velocity

Many of you have probably seen some or all of these accounts in my retweet stream. For better or worse, my personal account (@timoreilly) has garnered the most followers, and so I've become a switchboard for passing on the best of the news from others in the company.

I do find this to be an interesting exercise in managing corporate social media. I don't follow every O'Reilly employee, as we have no formal method for tracking them, but often, people who have posted something they want to bring to my attention send me an email requesting a retweet. (So do lots of outsiders. My habit of retweeting has ended up building a great extended information network!)

The fact that I don't automatically pass on company propaganda, but require it to be interesting, makes for a great teaching opportunity with employees. As I explain to them what I consider retweetable and why, and how to write tweets that make me want to share them, we improve the overall social media marketing IQ of the company.

(This is a purely technical or at least semi-technical post about a database and Web20 architecural issue.)

I just noticed about 5-10 status updates from my friends on Facebook are duplicated. Reading from the top I get to "17 hours ago" and then it restarts with duplicate status update messages from "5 hours ago".

read more

Every hour, we saw the diskspace on our root partition go up from 45 %  to 90 % and come back to 45 %. Initially thought mysql was processing some temp tables and in the next hour we kept watching the temp location. In terms of files / diskspace usage using “du / ls” nothing changed. But lsof provided an interesting output.

Output 1 :

mysqld    31719   mysql    6u   REG                8,1          0   149240 /var/tmp/ibOLLuRa (deleted)
mysqld    31719   mysql    7u   REG                8,1        102   149243 /var/tmp/ibNuSvI9 (deleted)
mysqld    31719   mysql    8u   REG                8,1          0   149245 /var/tmp/ibqq2wz8 (deleted)
mysqld    31719   mysql    9u   REG                8,1          0   149255 /var/tmp/ibVN66Rh (deleted)
mysqld    31719   mysql   13u   REG                8,1          0   149256 /var/tmp/ibslSJJA (deleted)
mysqld    31719   mysql   15u  unix 0×000001085e445680            19044143 /tmp/mysql.sock
mysqld    31719   mysql   18u   REG                8,1 1128267776   146656 /var/tmp/MLHbDVOq (deleted)

Output 2 :

mysqld    31719   mysql    6u   REG                8,1          0   149240 /var/tmp/ibOLLuRa (deleted)
mysqld    31719   mysql    7u   REG                8,1        102   149243 /var/tmp/ibNuSvI9 (deleted)
mysqld    31719   mysql    8u   REG                8,1          0   149245 /var/tmp/ibqq2wz8 (deleted)
mysqld    31719   mysql    9u   REG                8,1          0   149255 /var/tmp/ibVN66Rh (deleted)
mysqld    31719   mysql   13u   REG                8,1          0   149256 /var/tmp/ibslSJJA (deleted)
mysqld    31719   mysql   15u  unix 0×000001085e445680            19044143 /tmp/mysql.sock
mysqld    31719   mysql   18u   REG                8,1 1396703232   146656 /var/tmp/MLHbDVOq (deleted)

The size occupied by a file marked deleted kept increasing Strange but it seems to be happening on every dataload. The size change in this file corroborates perfectly with the space usage reported by “df” but not with “du”.

We are setting up the tmp location to a filesystem with more space for now before we investigate further.

   Tagged: disk, mysql   

Lots of blogs list a bunch of stuff that happened in the year just past, and I have done a year-in-review post before, but in looking back at posts on this blog and elsewhere, what strikes me most is not the big achievements that took place in technology in 2008, but rather the questions that remain unanswered. So much got started in 2008 — I’m really excited to see what happens with it all in 2009!

Cloud Computing

Technically, the various utility or ‘cloud’ computing initiatives started prior to 2008, but in my observation, they gained more traction in 2008 than at any other time. At the beginning of 2008, I was using Amazon’s S3, and testing to expand into more wide use of EC2 during my time as Technology Director for AddThis.com (pre-buyout). I was also investigating tons of other technologies that take different approaches to the higher-level problem these things all try to solve: owning, and housing (and cooling… and powering…) equipment. Professionally, I’ve used or tested heavily AppLogic, GoGrid, and all of the Amazon services. Personally, I’ve also tried Google App Engine.

2008 was a banner year for getting people to start tinkering with these technologies, and we’ve seen the launch of ‘helper’ services like RightScale, which puts a very pretty (and quite powerful) face on the Amazon services. The question now is whether the cost-benefit analyses, and the security and availability story is going to be compeling enough to lure in more and bigger users. I think 2009 is going to be the year that makes or breaks some of these initiatives.

The other question I have about cloud computing, which I’ve been asking since the last half of 2007, is “where does all of this leave the sysadmin?” It seems to me that a great many of the services being trotted out for users to play with seek to provide either user-level GUI interfaces, or low-level developer-centric interfaces to solve problems that historically have been the purview of system administrators. I’ve been wondering if it will force sysadmins to become more dev-centric, developers to become more system-savvy, if it will force more interaction between the two camps, or if it means death to sysadmins on some level, to some degree, or for some purposes.

I really think there’s a lot of hype surrounding the services, but I also think there’s enough good work being done here that 2009 could begin to reveal a sea change in how services are delivered and deployed on the web.

Drizzle

If you’re working in the web 2.0, uber-scaling space, and you’re using MySQL, chances are your relationship with your database is less ideal than it was when you were using it to run your blog or your recipe database. As you try to scale MySQL through various means, you find that there are lots of things that could be handled better to make MySQL scale more gracefully. Some extra internal accounting and instrumentation would also be nice. In many cases, it would also be nice to just cut out all of the crap you know you’re not going to use. If you’re looking to sharding, it would be good if there was a database that was born after the notion of sharding became widely understood.

Drizzle is a project started by some MySQL gurus to take a great experimental leap toward what could become a beacon in the dark sea of high scalability. At the very least, it will serve as a foundation for future work in creating databases that are more flexible, more manageable, and, more easily scaled. Of course, it’s also likely that Drizzle will be tied more closely to a slightly narrower audience, but I can say from experience that had the ideals of the Drizzle team been fully realized in an open source product prior to 2008, I may not have even installed MySQL in the first place. I had at least a passing familiarity with what I was getting myself into, and pulled the trigger to use MySQL based on criteria that deviated somewhat from pure technological merit.

I don’t believe Drizzle has announced any kind of timeline for releases. I wouldn’t expect them to. Instead, the first release will probably be announced on blogs in various places with links to downloads or something. The Cirrus Milestone for the project seems to focus quite a bit on cleanup, standardization, and things that, to prospective deployers, are relatively uninteresting. But I think 2009 will at least see Drizzle getting to the point where it can support more developers, and make more progress, more quickly. In 2009, I think we’ll see people doing testing with Drizzle with more serious goals in mind than just tinkering, and I think in 2010 we’ll see production employments. Call me crazy - it’s my prediction.

Microsoft

Windows market share on the desktop, it was recently reported by IDC, has dropped below 90% for the first time in something like 15 years, to 89.6%. Mac users now represent 9.1% of the market, and the rest is owned by Linux, at a paltry 0.9%.

It would seem that OS X has eaten away a few percentage points from Windows, and done perhaps more damage to the Linux space. I have no data to back that up at the moment - I’m going by the enormous shift from Linux to OS X between OSCON 2006 and OSCON 2008. I’ll let you know what I see at LISA 2009, which I plan to attend.

But what about Microsoft? Sure, they’re the company IT wonks love to hate, but the question of how their apparent (marketed) direction will affect their products and business is one that truly fascinates me. Microsoft has become the Herbert Hoover of American software companies, while Apple is FDR, perceived as having saved many of us from the utter depression and despair of the Hoover years (insert joke about sucking here).

Microsoft is enormous. It moves horribly slowly. It has shown a stubborness in the past that would seem difficult for something so large to shake off. Their products reflect this big, slow, obstinacy. What end users need is a software company that is going to lead its users in the direction they’re all moving in already on their own. It can no longer be about “allowing users” to do things (Ballmer has used such phrasing in the past). It needs to be about enabling and empowering, and getting the hell out of the user’s way.

The big question I think 2009 will answer is whether or not Ray Ozzie can affect change to either the culture, or the mechanics of how Microsoft does business (either one is likely to have a drastic effect on the other).

Python 3.0

It’s here already. I, for one, am quite excited about it. I think that GvR, Alex Martelli, Steve Holden, and others have put forth a very admirable effort to communicate with users and developers about what changes are imminent, what they mean, and how to prepare to move forward. I think 2009 is going to require 100% of the communication effort expended in 2008 in order to continue to rally the troops. I don’t know, but would imagine that the powers that be can see that as well, and so it will be. Assuming I’m right there, adoption will increase in the community, and the community buzz resulting from the wider adoption will begin to take some of the pressure off of the really big names, who quite honestly have craploads of other things to work on!

I believe that by summer 2009 we’ll see Python 2.6 migrations happening more rapidly, and a year out from that point we’ll start to see the wave of 3.0 migrations building to more tsunami-like proportions.

Another question: is there sufficien new adoption of Python going on to register 3.0 on the usage scale? Probably not now, but hopefully in 2009…

USA Gets a CTO

I’ve read a few articles about this, but all I’ve read really just amounts to noise and speculation. What, exactly, will the CTO be charged with? I’ve seen Ed Felten floated as a candidate for the position, but he’s not a person who’s going to want to run in and try to herd cats to try to standardize their desktop computing platform. I think if the CTO position is going to take charge of the things Felten has already shown a keen interest in (namely, high-level IT policy, the effect of technology on society, privacy and security… as it relates to the former two items, etc), then there could be nobody better for the job. Princeton’s Center for Information Technology Policy is one of the few places (maybe the only place) I’d actually take a pay cut to join ;-P

I imagine that 2009 will answer the questions surrounding the nation’s very first CTO.

It’s The Economy!

I’m a freelance technology consultant and trainer. Anyone who is making a living freelancing is probably wondering about the state of the economy, no matter where they live (incidentally, I live in the US). The numbers aren’t good. The S&P is down something like 41% this year - the largest drop on record. The state of the markets in general, along with the failing of the banks and their subsequent appearance in Senate committee hearings, as well as the deflationary spiral in the housing market (and predicted more general deflationary spiral) invoke images of bread lines and soup kitchens… or at least very little work for freelancers.

Personally, I have a lot to lose if things *really* go south to the degree that they did in the 1930’s, but I have to say that I don’t think it’ll happen. If you’re worried about this becoming the next Great Depression and are really losing sleep over it, I recommend you read a book called “The Great Depression” by Robert S. McElvaine. There are probably tons of books you can read, but this is one I happen to like. It’s full of both fact and opinion, but the opinions are well-reasoned, and loudly advertised as being opinions (you’re not likely to find a book about any topic relating to economics that isn’t full of opinions anyway).

What I think you’ll find is that, while there are a lot of parallels between now and then, there are lots of things that *aren’t* parallel as well (partly as a result of the depression - for example, the US is no longer on the gold standard, and both banks and securities trading are infinitely more regulated now). Also, not all of the parallels are bad. For example, things began to improve (though slightly at first) almost the day a new Democratic leader replaced the outgoing Republican regime.

My advice (which I hope I can follow myself): If the market numbers bother you, don’t look. Service your customers, don’t burn any bridges, rebuild the ones you can, build new ones where you can, and above all, Do Good Work. When you don’t have work, market, volunteer, and build your network and friendships. Don’t eat lunch alone, as they say.

What are you wondering about?

My list is necessarily one-sided. A person can be into only so many things at once. What kinds of tech-related questions are you searching for answers on as we enter the new year?

2008 is nearly over, and it's time to take a look at what happened over the year, as well as to take a peek at the the coming 2009. A year ago I made a guess that social networking services would open up and start sharing their profiles – well, practically everyone but Facebook are doing some of that, and Facebook is trying to get everyone to depend on them – not that “create dependency” isn't a part of Google's and MySpace's plan, too. Unfortunately, we haven't yet found a meaningful way for Habbo to participate in this festival, due to differences in demographies, interest areas, and the priority of running a profitable business, instead. Still looking for that solution, though.

I also guessed that productivity applications would seriously move to the cloud – and was a bit too optimistic on that one. Sure, the applications are there, but I don't really see any of them having replaced the desktop-based counterparts – nor do I see that happening next year, either. People are, rightly so, focused somewhere else, and while over the long run moving off to the cloud will make sense from both productivity and cost standpoint, it's still too much of a jump, and too expensive to make.

The increasing popularity of netbooks, Internet access via 3G networks, etc, will have an impact on that, though. Perhaps we'll all move out to the net in a completely different way: not via our old productivity apps, but via entirely new class of applications. Something else than Facebook and Twitter though, I hope.

What else? MySQL was acquired by Sun, and we're all still waiting for the next step. The Register (I can't believe I keep reading it) has somehow gotten the impression that Sun has slowed MySQL down – nah, it's been this slow for at least three times that long. Fortunately, the acquisition may have been a catalyst for the MySQL developer community to start doing something else instead of waiting, and I'm really looking forward to the improvements Percona and Drizzle are making to keep MySQL competitive. As for Sun – time to stop confusing a good thing with dubious business models and bad release engineering before you lose all your customers, I'd say. At the same time, I'm also super-interested in the stuff Sun is doing on the hardware side of database storage with SSD-optimized solutions. Can't say I paid much attention to Sun there for a while, but they're making what seems like an unlikely comeback.

For Habbo, we've continued making progress on the track chosen late 2007 – revolutionary changes made incrementally. Biggest one this year, the free second currency of Pixels, was just launched a month ago. Several improvements are coming up for that, of course, and a whole lot of other stuff is in the works, or at least being thought of. We're trying not to hold anything longer than it absolutely needs to, so everything radically new continues to be launched sort-of unfinished and get improved along the way. It just ends up being so much better that way, as the feedback makes a significant contribution to the overall design.

This a weird time. The world is reeling from what indeed may be the worst economic crisis in 75 years (though I'm not well versed enough in history to be able to tell myself), and still (or because of it?), opportunities lie all around, ignored by most. It's never easy to tell which direction is most promising, but now I'm finding it incredibly hard to choose and prioritize between the possible things to focus on. Still, 2009 is definitely going to be a year to really focus on even fewer things than usual, and really kick ass on those.

If you made it this far, thanks for reading. I wish you a great year 2009, whatever it is you're doing.

« Post 1 | This is the 2nd post in my MoFo Futures 2009 blog series. | Post 3 »

"When a job went wrong, you went back to the beginning. And this is where we got the job. So it's the beginning, and I'm staying till Vizzini comes." — Inigo Montoya, The Princess Bride Screenplay by William Goldman

Being a part of Mozilla over the last three years has been a humbling experience.

When I started with the project, I imagined great personal success. Like a happy little worker bee flitting from Free Software flower to Free Software flower, I would cross-pollinate Mozilla with PHP's community savvy and pragmatism, MySQL's  disruptive innovation canny, and the Free Software Foundation's dedication to their mission and service ethic to Free Software developers. Mozilla would be better, stronger, faster (and, by happy side-effect, I would be a hero.)

Instead, I've experienced a mix of tepid success and equally tepid failure in my Mozilla Foundation work - the sparks I strike never seem to make it into full flame. While I'm proud of my Age of Literate Machines work, it's still just a spark.

If you had asked me why this was a year ago, I probably would have told you that it was because I worked with world-changers - people who've helped build the foundations of the net, people who've stood up to powerful governments, people who've helped build things that are the foundations of parts of my life - and that it is difficult to be a peer in this group.

Reflecting on this after the last three months of personal and organizational soul-searching, I think that the key reason being a part of Mozilla is uniquely humbling is that we have a massive constituency and want to effect global change across the breathtaking complexity of the net. To make meaningful positive change at this scale requires deep clarity of thought, a singular purpose and a lot of luck. I've had the luck, but to get to where I must be I need to go back to the beginning.

« Post 1 | This is the 2nd post in my MoFo Futures 2009 blog series. | Post 3 »

What's new, in a nutshell: http://dev.mysql.com/doc/refman/5.1/en/mysql-nutshell.html.

Release notes: http://dev.mysql.com/doc/refman/5.1/en/news-5-1-x.html (In the video, it's the page entitled "Changes in release 5.1.x").

And yes, very early on (at about 2 minutes in) I talk about my take on Monty's controversial post at http://monty-says.blogspot.com/2008/11/oops-we-did-it-again-mysql-51-released.html

read more

Comparing the performance of SQL Server 2005 to SQL Server 2000 demonstrated an improvement of 18.5% in the time to process a workload. Now that SQL Server 2008 has shipped, it's time to compare the performance of SQL Server 2005 to SQL Server 2008.

I'm a Sales Engineer, I admit it. This blog is mostly about technical stuff, but I felt I should write a not-so-technical post for once. For a techie like myself, and to someone else out there, even the Sales Engineer title may look weird. Frankly Sun doesn't have Sales Engineers, they have System Engineers, which is the same thing. Not. Not at all, it is something different. But I insist on being a Sales Engineer.

I have been an SE (as in Sales Engineer) on and off (mostly on) for a good 20+ years now, more or less always in the database industry (a short term with security software is the exception). I learnt the SE trade at another RDBMS company in the 1980's. I had joined them as Support Engineer, having previously been a developer. I learnt SQL the hard way, by using it. I learnt to use indexes by having not used indexes, realising things were slow, looked at the syntax and found the word INDEX which sounded like something that would speed things up, and it did.

Anyway, I now was working for a US corporation as a Support Engineer in Sweden. There was 8 of us. No, not 8 support engineers, 8 employees in Sweden. Only one Sales Engineer, and that was NOT me. I was not too keen on having to meet a customer, as I had been forced to be a trainer early on, and I was scared stiff by that experience. And the Sales Guys then. I hadn't dealt much with them previously, but now.. Oh my. So this Sales Guy comes into my room one day, the SE was off to something else, and a techie was needed at a customer meeting the next day. I was the only one available, so "Take your best suit, a white shirt and a tie and you are all set, it's easy", the Sales Guys said, trying to calm me down. It didn't. I didn't have a nice suit, nor a white shirt and a tie, well I did not ever want to wear one, and I didn't have one either.

But what could I do, I was the youngest employee by far, the lowest on the ladder and self-confidence wasn't something that was I too familiar with. So I got myself a suit, a shirt and a tie (in those days, if you were in Sales at all, you just HAD to wear a tie) and went along with the fast-talking salesguy. And I found out I liked it. So after some time, I went along, took some training and became an SE and was assigned the important task of carrying the Sales Guys cellphone (this is true. And cellphones were BIG in those days. And only management and salesguys had them).

Now, 20+ years later, what am I doing you ask? A consultant that isn't necessarily paid? A tech support service for sales guys and customers that Sales Guys think deserves it? Someone with the most contradicory title on the planet? Frankly, the job of an SE includes all those things. But the deal with an SE doing, say consulting or tech support isn't what we do, but why. We do engineering work in the name of Sales, and that isn't necessarily a bad thing. That's the what and the why.

As for any customers reading this, and a few of you have probably meet me out there, what does this mean? Do we fake engineering abilities so we can get paid? Nope, a Sales Engineer is always an Engineer at heart. If my customers are happy, then I am happy. We are honest guys, it's just what we do is for a different reason than, say, tech support. But that doesn't mean we are evil, at least I try to avoid it (Santa was here fo me this year, so I think I am doing OK).

Doing what is right, from a technical perspective, is what we try to achieve. For example, a prospective customer wants to start a project based on this or that idea, and he is using MySQL for it. It is not the Sales Persons job to say yes or no to that, or even to understand if this is a good thing or not. No, the sales persons job is to make sure that the customer gets a fair deal on MySQL for this project. But what if the design of the thing is completely flawed? Anyone can see that this will not work, right? Well, the customer might not have worked with MySQL ´before, and maybe not even with an RDBMS. And it's not the salespersons job to understand or question this (which doesn't necessarily mean that he doesn't understand, it's just that it isn't his job). So whose job is it then to get back to the customer with Sales person and say that, "Hey, this is not really such a good idea" or something to that effect? The SE. That is what we do. And also the SE should provide some alternative design here.

And if the customer says, "I want MySQL, but I'm not sure about the design", well an SE can answer that. So we can spend a few weeks at a site for free fixing the customers design, before any contract of any kind has been signed. Nope, wrong. There is a balance here, SEs don't do things "for free", as it seems sometimes. But we do help out, point out a direction and give advice, in the extent we can do just that.

What kind of training does an SE have, what competences to we hold? Typically more a combination of many things than in most other professions. An SE typically knows little about many things, and combine that with knowing A LOT of a few things. I'm a good C hack for example (hey, I am *old* now) and I know a few other things well too.

Is what I am saying here trade secrets of MySQL / Sun Microsystems then? Nope, if you have worked as a Sales Engineer, you know it's about the same thing everywhere. Sometimes we are called Pre-Sale consultants, sometimes SE, sometimes Field Engineers or something, but the job and what we do is similar.

We are all in all the engineering side of the Sales Process. When a sales person doesn't know if technical project has any chance of being successful, he asks us. When a prospective customer has some technical questions, we answer them or get them answered.

And then there is one other thing. When the customer has paid up, are we gone into thin air, never to be seen again. Nope, as I said, we are after all Engineers, we like to see technology work, we want to help, and as MySQL is selling subscriptions, we will see you again next year.

SE's with MySQL / Sun Microsystems has a lot of freedom, more than with any company I have worked for an an SE. If a technology or a customer interests me, I am allowed to dig into that and get my hands dirty, should I wish so. Aand are we cometing with the MySQL / Sun Microsystem Preofessions Services consultants? Nope, we try hard not to.

All in all, we are techies, the work we do is stuff that a Support Engineer or PS Consultant can do, at times we do those jobs at least (we are probably not as fast as those guys at it though). We just do these things for a slightly different reason. And we ARE a bit salesy, I admit, we have to know what is going on in the market, we have to understand how to explain something in a few words to someone who might not be a techie, but someone who is signing the checks, for a customer.

After so many years as an SE, more than most I think, I can say that being an SE at MySQL / Sun is great. And I look forward to more of it. And (blushing a bit) I a´was here easrly enough to start it (we were 2 SE's back then). I can tell you more about that, but this posting is long enough already. Next posting will be back at technology, I promise!

Yours truly
/Karlsson

Network Communications, a publisher of printed and online real-estate information in North America, has selected Omniture, a provider of online business optimization software solutions, to deliver web traffic analysis across its network of sites.

WFTO - end of 2008 edition This is the last WFTO report for 2008. Our MySQL/Falcon team is mostly on vacation and therefore there was not that much activity last week. However, Sergey Vojtovich fixed one tablespace related bug. I am looking forward to report about our WFTO many more times in 2009! Since my last report [...]

I’ve always been more of a GNOME guy, and when running Linux, I use Rhythmbox to play my music. However, Amarok 2.0 might just change that.

They’ve chosen their database - it is none other than MySQL. The release notes state:

Some features, such as the player window or support for databases other than MySQL, have been removed because either they posed insurmountable programming problems, or they didn’t fit our design decisions about how to distinguish Amarok in a saturated market of music players.

If you want to know why the decision was made, read MySQL in Amarok 2 - The Reality. It has a lot to do with the fact that MySQL can be embedded, and performs well. Its a generally useful read to see why SQLite and PostgreSQL was not chosen.

MySQL… powering the music of today!
(wonder as I may, if we will ever get any Enterprise customers, who make heavy use of Amarok over many computers, etc… - I’m thinking modern night clubs, lounges, et al)