Database Technology Trends

Back in the 90′s a group of German engineers put together the world’s first grid computing network with over 100 PCs running on the first version of the Linux operating system. It was a great success and everybody called it the dawn of a new technology that will change the computing world forever.

What these engineers didn’t understand was how database engines worked at the time and how they actually set the trends for hardware development. The database engines were calling for Massive Parallel Processing (MPP) systems that offered dozens of CPUs in one single server platform. Just when we thought the mainframe was dead – with grid computing, we witnessed the birth of the open systems mainframe.
To offset these MPP systems, software architects created a middleware layer to get away from these monsters. Now we have these MPP systems in the center of the universe and all these middleware servers dancing around them.

For years to come, companies filled their data centers with hundreds of middleware servers and dozens of MPP systems. Cooling and power supplies were running at their peak and data center managers didn’t know how to support this hardware excess into the future.

This became the birthplace of server virtualization. Within a few years, server virtualization became the main focus for every company searching for infrastructure savings. Consolidating middleware servers was an easy task and a huge success story shared with pride by the project managers.

But one area was not so successful in consolidating hardware. MPP systems didn’t go away quietly. It turned out that the database systems were too much to handle for the server virtualization frenzy. Yes, countless efforts have been undertaken to move the databases off these monsters, but the virtual world couldn’t provide the performance needed to support the databases.

Remember our grid computing story? The vision of sharing an army of small computers to produce the same computing power as the massive MPP systems seemed lost forever. Until the ugly truth about MPP systems became obvious. Running huge MPP systems are not only energy intensive, but the associated maintenance costs are also burdensome. Buying a replacement MPP system was the only option. However, spending money got increasingly tight over the past several years. This all became a Catch-22.  You needed to spend more to increase your costs!

Database vendors to the rescue: grid database technology seems to be the way out of under the massive weight of these MPP systems. Take a couple of low-cost powerful dual or quad core servers and spread the workload over multiple servers. Not only do you get instant high availability, but you gain added scalability beyond your MPP platform’s physical limitations. There are two major methodologies in achieving grid databases; shared everything and shared nothing.

The shared everything category is dominated by Oracle and Sybase. Both systems are able to instantly failover database processes should one participating server go down, aka. high availability. And both can dynamically scale their CPU power by adding more servers to the grid, and both systems can balance the workload among all participating servers. Oracle RAC and Sybase ASE-Cluster Edition are the most sophisticated systems available today. If you want to squeeze the maximum out of your existing hardware or if you are seeking to replace energy-wasting and maintenance-fee-eating MPP monsters, these two databases are the weapons of choice.

The second methodology is shared nothing. Microsoft SQL Server 2008 Federation Data Store represents the leader in this category. Unlike the shared everything technology, the shared nothing approach has a clear distinction between local and global data. The data federation approach allows combining data stored locally on multiple individual databases. It acts as an aggregator of multiple databases. This is not as sophisticated as the shared everything approach, but it gets the job done as well.

There’s a third contender, Sybase IQ Multiplex. This system uses a hybrid approach, shared data, but no shared cache. This is very unique and no other database vendor has anything like it. Sybase IQ is Sybase’s data warehouse engine. A column-vector database that set new performance benchmark records by having just one distinctive writer node in the cluster and a nearly unlimited number of reader nodes.  But there’s a caution: Never try to run an OLTP application on this system. This system is built for data warehousing and massive analytical reporting, a perfect match for data hungry BI tools.

There are a couple of other database vendors that are offering grid database technology of some flavor. This article is not meant to create a competitive analysis between all of these systems, but a starting point to get your imagination going. The bottom-line is that preserving energy becomes more and more important and software vendors are providing solutions to maximize low-cost and low-energy consuming hardware. The future belongs to grid databases.

This is not an article on how to organize your next company party or sales event. Business event management (BEM) is a method on how to detect deviations of the effectiveness in your business and pro-actively reacting to changes.

Forrester defines business event management as:

“Business event management (BEM) is the process of capturing real time business events from multiple sources and assigning them to the appropriate decision maker for resolution based on the business context of the events”

The main focus is on the terms real time and multiple sources. This is the only way to intercept business processes that are out of control or are about to miss their target. Business process management is the method to ensure that a business process has a clear definition form start to finish with checkpoints along the way. Business event management is also known as cousin to workflow. Workflow management ensures that a single business process completes successfully and in a timely fashion. Whereas business event management combines multiple business processes, monitors its progress and let’s process owners know when things are about to go wrong.

Another term that is frequently used in combination with business event management is complex event processing (CEP). As CEP implies, business event management is a complex method to detect patterns that are exceeding a certain threshold or are about to exceed them. To boost the effectiveness of BEM, the pattern detection must be done in a timely matter to stop a bad situation from getting worst.

As an example, in the restaurant industry, fraud detection is a big problem. Detecting these fraud patterns as they happen would save an immense amount of money. Without BEM the detection happens after closing the books, a long time after the potential fraud happened, and a lot of man hour is spend to scan the transactions for irregularities. This is a good example on how BEM can apply complex event processing in real time to detect situations that exceed a predefined threshold. The timeliness of the detection is the real money saving factor in this example.

BEM doesn’t need to be a messenger of bad news. Monitoring positive situations is as important to a success of a business as preventing negative situations. Identifying positive situations initiated by an employee and recognizing this employee will not only boost morale, but ultimately boost the bottom-line.

There are many vendors that are offering their implementation of BEM. Most of these products are vendor centric, meaning that they are not capable of correlating multiple sources of event data. This defeats the primary objective of Forrester’s BEM definition. Other vendors went too far with their BEM implementation and made it overly complicated and unnecessarily complex.

I came across one tool that lives up to Forrester’s BEM definition without being too complex. It is called SmartScore. With the ability to connect to most known database flavors and even monitoring ftp servers and file folders, this tool has the power to combine many data sources for an effective pattern detection, which is the primary objective of BEM; monitoring multiple sources in real time.

Here’s the link to more information about SmartScore:
http://www.doblerconsulting.com/bem.html

This is what the industry has to say about BEM:

“Through 2010, the adoption of BEM platforms will grow at least 200 percent from 2006.”

Source: Gartner Report

“BEM’s real potential is at the business level, helping to enable new strategies, reduce operating costs, and improve process performance and other tangible areas of management attention.”

Source: Gartner Report

“The Business Event Management market will grow from $1.2B in 2005 to $2.7B in 2009.”

Source: Forrester

In my opinion, BEM is absolutely crucial to detect real cost saving opportunities and then constantly and consistently execute these saving measures. Especially in a down economy it is a question of survival if your business is able to minimize the cost to do business. Sometimes the only way to do this is with the use of additional software. This additional expense is Ok as long as the ROI is fast enough to benefit the cost savings in the long run. I believe SmartScore is the tool that can deliver.