I've been working on a way to backup SQL Server 2005 databases to a compressed file recently, but without using temporary files. I recently found that it is actually pretty easy. SQL Server has an interface for creating a virtual backup device so that the data is pumped to your application, not a real device. Once the application has the data, it can compress it, encrypt it, or do whatever it likes. Maybe DBAs would like to save the data to an FTP server. There are so many options.

The application I wrote is called MSQL Compressed Backup which you can find on Sourceforge. It is currently in beta, and I don't plan to release a 1.0 version for awhile until it has had some testing by other people. If you are interested in it, please download a copy and give it a try.

Documentation is currently lacking, but here are a couple of examples. I think they are self explanatory.

• msbp.exe backup [model] gzip file:///c:\model.bak.gz
• msbp.exe restore file:///c:\model.bak.gz gzip [model]
  Here, gzip knows to uncompress since it is in "restore" mode.
• msbp.exe backup [model] bzip2(level=5) file:///c:\model.bak.bz2
  Each plugin in the pipeline can have parameters.
• msbp.exe backup [model] rate(ratemb=5.0) bzip2(level=5) file:///c:\model.bak.bz2
  You can have any number of plugins in the pipeline. The rate plugin limits the impact of a backup on your server by restricting the speed of the backup (or restore).

I had a performance problem recently with SQL Server, and I went through the standard performance checklist, however it didn't solve the problem permanently. Sometimes it would perform well, but most times it was performing poorly. I knew the next step was to mess with the execution plan. This is something I really don't like.

You do not want to force SQL Server to use a particular execution plan, because SQL Server can pick different execution plans depending on how much data will be processed. When it processes a few rows, it will choose a plan that is optimized for a few rows (and typically use nested loops). If the same script processes a lot of rows, it will use a plan that is optimized for a lot of rows (and use merge joins or hash joins). By forcing SQL Server to use single execution plan, you prevent it from using the most efficient execution plan for different scenarios.

But what happens if SQL Server estimates the wrong number of rows? The worst thing it can do is estimate few rows, use an execution plan optimized for a few rows, and actually process a large number of rows. In this scenario, you will find a very slow query.

I found an easy fix for this situation. Use the OPTION(HASH JOIN, MERGE JOIN) modifier to any SELECT, INSERT, UPDATE, or DELETE statement. For instance:

UPDATE cust
SET CustomerSourceID = th.SourceID
FROM Customer cust
INNER JOIN TransactionHeader th ON th.CustomerID = cust.CustomerID
WHERE cust.CustomerSourceID IS NULL
OPTION (HASH JOIN, MERGE JOIN)

The OPTION (HASH JOIN, MERGE JOIN) modifier does not allow SQL Server to use nested loops. Since nested loops are typically efficient for a small number of rows, this causes SQL Server to optimize your query for a large number of rows. Even if this query encounters a few rows, the plan will be moderately efficient.

The good thing about OPTION (HASH JOIN, MERGE JOIN) is

• It does not require a statement to be restructured.
• It will not likely introduce any bugs.

The bad thing about it is

• You prevent SQL Server from selecting the best execution plan for all scenarios. The plan will be optimized for a large number of rows.

I have been working on a performance problem recently, so you might see several blog entries with information that help me. Hopefully they will help you.

Getting the SQL Server execution plan from a production can be difficult, since you are not running the code within Enterprise Manager. You can still get the execution plan of any running statement and display the graphical representation in Enterprise Manager.

If your script is currently running, you can lookup its execution plan in the plan cache. The steps are:

1. Run a query to get the the plan_handle of currently running code
2. Lookup the execution plan for the plan_handle.

The first query is:

SELECT
sder.session_id AS [SPID],
sder.sql_handle as [SQL_Handle],
sder.plan_handle as [PLAN_Handle],
sdes.login_name AS [Login],
sd.name AS [DBName],
sder.start_time AS [Start Time],
sder.status AS [Status],
sder.command AS [Command],
sdet.text AS [SQL Text],
sder.percent_complete AS [Pct Cmplt],
sder.estimated_completion_time AS [Est Cmplt Time],
sder.wait_type AS [Wait],
sder.wait_time AS [Wait Time],
sder.last_wait_type AS [Last Wait],
sder.cpu_time AS [CPU Time],
sder.total_elapsed_time AS [Total Elpsd Time],
sder.reads AS [Reads],
sder.writes AS [Writes],
sder.logical_reads AS [Logical Reads]
FROM
sys.dm_exec_Requests sder
CROSS APPLY sys.dm_exec_sql_text(sql_handle) AS sdet
JOIN sys.dm_exec_sessions sdes on sder.session_id = sdes.session_id
JOIN sys.databases sd on sder.database_id = sd.database_id
WHERE
sder.session_id <> @@SPID and sder.session_id > 50

The second query is:

SELECT * FROM sys.dm_exec_query_plan ( <PLAN_Handle> )
--here <PLAN_Handle> is supplied based on
--the results from the first query.

This gives you the XML execution plan that you can copy and paste into notepad, save it with a *.sqlplan extension, and double click on the file to view the graphical version of the execution plan.

This is a second entry in a series on SQL Server Analysis Services (SSAS). To see the other blog entries on this tutorial, click on the SSAS Beginner’s Guide on the top bar.

Relational Databases

Let’s take a quick look at a relational database table of football players:

The table is structured in rows and columns. Note that the column names are fixed – they are not based on the data. This is a difference between relational databases and SSAS databases.

SSAS databases are typically loaded from the relational database to the SSAS database and a copy of the data stored in a different form in the SSAS database. Once the data has been copied, you can delete the relational database (This assumes you are using the default MOLAP option).

2 Dimensional SSAS Databases

SSAS databases are stored in multidimensional structures. Below is an example of an SSAS database with just 2 dimensions. This contains fantasy football points, so the "2" in the first row and first column is the fantasy football points that Reina received during the 11/Aug/2007 game week.

The grey area contains two dimensions: Players and Game Weeks. Each column header (and row label) is called a dimension member. Example dimension members are "Carragher" and "03/11/2007". These dimension members are similar to column names, however they are based on data.

Each number in a cell is at the intersection of two dimension members. Take for instance Gerrard’s first match where he received 10 points. It is stored at the intersection of

• "Gerrard"
• and "11/08/2007"

Gerrard and "11/08/2007" are coordinates to the cell containing 10. When you start writing MDX queries, you will try to find ways to reference these cells using coordinates.

The All Member aggregation is what is really useful in SSAS databases. It is a member that is added to each dimension when loading the data. The All Member cells are pre-calculated when loading the SSAS database data and are typically a sum. This member is physically stored and you can also query All Members just like any other member. The All Member aggregation above makes it easy to answer questions like, "What is the total number of points that Gerrard received?" The answer is found at the intersection of

• "Gerrard"
• And the Game Week’s "All Member".

You can reference the All Members using the coordinate system just like any other member of the dimension.

3 Dimensional SSAS Database

Let’s take a look at a 3 dimensional SSAS database:

The diagram above shows most of the values stored in the 3 dimensional SSAS database. There are some values in the back of the cube that you can’t see without cutting the cube open, but they still exist.

The cube is shown so that the front pane is the same data as the two dimensional SSAS database above.

Just as a side note, although the diagram has two 10s on the cell for Gerrard on 11/08/2007, on the front in grey and on the top of the cell in white, the cell block itself only contains the number once. The cells can only contain one number and the numbers shown are for the 3D block, rather than a 2D pane. So the cell contains just 17, even though 17 is shown three times.

The only difference with 3D cubes is that locating a cell requires 3 coordinates instead of 2:

• Game week
• Player
• Home/Away

The 3^rd dimension, Home/Away, gives us more information about the scores compared to the 2 dimensional database. If you wanted to know how many points these players received for all Away matches, you would go to the intersection of these three coordinates:

• Game week’s "All Member"
• Player’s "All Member"
• and Home/Away’s "Away" member

Here you will find the value "155".

The cube can answer other questions. Although you cannot see it in the diagram, the cube stores total points that Gerrard received at home. You would find it at the intersection of

• the game week’s "All Member"
• "Gerrard"
• "Home".

This value would be found on the bottom plane just to the left of the value 155.

4 or More Dimensions

There is really no limit to the number of dimensions you can create in your database even though it might be difficult to visualize. Let’s say you want to track sales for the following:

• the year 2007
• the southeast region
• the product "gizmo"
• and the "Acme" store

You may not be able to see how all the data is stored, however this scenario does have 4 dimensions.

I am going to use this blog to convince a friend to use SQL Server Analysis Services (SSAS) in his projects because it makes it easier to write reports on as compared to relational databases. Part of the challenge is that people don't realize its advantages, and part is the large learning curve to use it.

This first post is on it advantages. Next will be a whole series to make it a bit easier to learn. First there are two main advantages for using SSAS.

SSAS is fast even on a large volume of data

Well, as long as the queries return few rows, and this really gets at the heart of when you want to use SSAS and when you don't. It is good at returning a single sum for a column over a large number of rows, but is very slow at returning a lot of detailed rows. The reason for this is the totals (and the subtotals) are stored in the SSAS database. It takes more work to read all the original rows than the one single subtotal value. In fact, relational databases will be faster at returning detailed rows compared to SSAS.

Take for example the following SQL code from a relational database:

SELECT SUM(Cost)FROM Sales

In SQL Server, this query could take a very long time as it scans the Sales table, however in SSAS the result comes back instantly because the one value is already pre-calculated and stored in the SSAS database.

SSAS's calculated measures are fast execution-wise and easy reusable

In relational databases, calculation code can get messy. The main problem with SQL SELECT statements is that they make it difficult to write a calculation once, like in a function, and GROUP BY different ways. You might build a SELECT statement in a string and execute the string using sp_executesql, or you use a CASE statement to list the different SELECT statements that group differently. Both are messy compared to SSAS.

Calculated measures in SSAS give you a way to define the calculations in a straight forward manner that does not matter how you will group the data later. They also you another advantage in that they are defined centrally in the SSAS database, and the reports pick and choose the calculated measures they want. This moves formulas from many reports to the central SSAS database where they are tested once.

Between having one set of centralized "official" formulas, and having formulas written in clear straight forward manner, the reports will have fewer number of bugs.

The following query lists the tables and the space they use. This query is much faster (sub-second) than a standard SELECT COUNT(*) query since it uses the dynamic management views in SQL Server rather than scanning your data.

SELECT sum ( used_page_count ) * 8 AS SizeKB,
  sum(row_count) AS [RowCount], object_name ( object_id ) AS TableName
FROM sys.dm_db_partition_stats
WHERE index_id=0 OR index_id=1
GROUP BY object_id
ORDER BY sum ( used_page_count ) DESC;

SQL Server 2008 includes a new HierarchyID datatype!

I was reading about two of Google's internal programs, Bigtable and Google File System, and how they handle a large amounts of data so that the processing is distributed (Bigtable) and the data is replicated (Google File System). One thing they both do is exploit immutability. So once something is saved to disk, that block of data doesn't change. If you have a large volume of data, this assumption can be very useful.

Let's say we're talking about a transaction table for bank accounts. Records in the transaction table don't change. Any corrections are handled by creating a new transaction record at the tail end of the table. If you are processing the transactions, this is great! You can ignore previously processed records since you know they will never change.

There are two problems with indexes on large nvarchar columns:

• You will likely hit the 900 byte limit in your index
• Indexing large data isn't efficient anyway.

A neat feature of SQL Server is the CHECKSUM() function which hashes your varchar/nvarchar values into a 4 byte number. You can then use this value in an index. For example if you have a Site table, add a calculated column, URLChecksum.

CREATE TABLE Site (
  SiteID int NOT NULL,
  URL nvarchar(2083) NOT NULL,
  URLChecksum AS (checksum([URL])),
 CONSTRAINT [PK_Site] PRIMARY KEY CLUSTERED (SiteID)
);

Next create an index on the hash and include the URL:

CREATE INDEX IX_Site ON Site (URLChecksum) INCLUDE (URL);

This index will make the following query faster:

SELECT SiteID
FROM Site
WHERE URLChecksum = CHECKSUM(N'http://www.microsoft.com/downloads/details.aspx?familyid=9a8b005b-84e4-4f24-8d65-cb53442d9e19&displaylang=en')
AND URL = N'http://www.microsoft.com/downloads/details.aspx?familyid=9a8b005b-84e4-4f24-8d65-cb53442d9e19&displaylang=en';

This query will first "seek" the hash value in the index very quickly, since the hash values are just ints. Once it finds one or more matching hash values, it will check that the URLs match. Since the URLChecksum, URL, and SiteID values are included in the index, this query does not need to touch the Site table.

A neat feature in SQL Server 2005 is the ability to "include" columns in an index. These included columns are not in the main part of the index, but are additional information in the index.

For example, lets say you have the following SELECT statement:

SELECT Url
 
FROM Site
 
WHERE Category = 'News';

You might be tempted to create a covering index with Category as the first column, and Url as the second column. Since URLs can be 2083 characters, you can't put the URL column in an index since it would exceed 900 bytes. However, the query above would benefit from the following index where the URL is "included" in the index and therefore isn't restricted to 900 bytes.

CREATE INDEX IX_Site ON dbo.Site (Category) INCLUDE (Url);

The main part of the index only contains the category column. The index also stores the Url, but the Url can't be efficiently used for filtering. In the SELECT statement above, this is OK, since the Url is only returned, not filtered.

Another benefit is that the main part of the index is smaller, so it is faster to find records in the index.