Real Life experiences with SQL

SQL 2008 Fulltext Query Performance issue using FREETEXT and CONTAINS June 6, 2010

Of all the things that have improved in SQL 2008 Fulltext seems to have taken a step backwards. We were helping one of our clients upgrade from SQL 2005 to SQL 2008. They are a 24×7 shop with a framework that supports various websites for their customers. Fulltext is heavily used for text based searches on the websites. We had set up a test rig for them on SQL 2008 to test their applications. The testing process revealed sever performance issues with the text based searches. We investigated and identified the cause of the performance degradation to be the FREETEXT clause used in a multiple OR condition.

Issue:
A fulltext query using the FREETEXT predicate in a multiple OR combination works magnitudes slower in SQL 2008 than in SQL 2005 like for like vanilla installs. We tested this in SQL 2008/R2 and the same issue persists so it pointed to an engine issue in the way fulltext query plans were generated and handled in SQL 2008.

Eg.

SELECT A.Col1,  A.Col2, B.Col1, B.Col1….
FROM TableA AS A LEFT JOIN TableB AS B ON A.ID = B.A_ID
WHERE FREETEXT((A.Name, A.Description), ‘SomeSearchText’)
OR FREETEXT((B.Name, B.Description),’SomeSearchText’)

The fulltext catalog encapsulates multiple tables which each have a single fulltext index.

Workaround:
After thorough investigation we identified a workaround for this issue, it is to separate the query using a UNION clause which has extremely good performance on SQL 2008/R2.

Eg.

SELECT A.Col1,  A.Col2, B.Col1, B.Col1….
FROM TableA AS A LEFT JOIN TableB AS B ON A.ID = B.A_ID
WHERE FREETEXT((A.Name, A.Description), ‘SomeSearchText’)

UNION

SELECT A.Col1,  A.Col2, B.Col1, B.Col1….
FROM TableA AS A LEFT JOIN TableB AS B ON A.ID = B.A_ID
WHERE FREETEXT((B.Name, B.Description),’SomeSearchText’)

Unfortunately, this leads to code bloat and may not be the best approach to resolve this issue as it could mean many man hours to update the related code which could be hundreds of stored procedures as was in our clients case.

We raised a PSS case with Microsoft to identify the root cause and possibly provide a solution for the same.

I have attached the repro scripts I provided Microsoft to help them recreate the issue:

• FREETEXT_Issue_CreateSchema.sql => Contains the Sample Schema Creation Script
• FREETEXT_Issue_Query.sql => Contains the Sample Problem Queries and the Workarounds we have found

Instructions to Reproduce the issue(Run in SQL 2005 and SQL 2008 instances preferably similar hardware/software settings):

• Run the FREETEXT_Issue_CreateSchema.sql Script => this will create the schema and populate the tables and the fulltext catatlog
• Wait for the fulltext catalog to finish population
• In a separate window run the FREETEXT_Issue_Query.sql script, switch to the messages tab and compare the CPU times in the output.

Microsoft confirmed that this was a BUG in the SQL 2008/R2 and infact affected the CONATINS clause too. They confirmed that it would take a substantial design change to fix this issue. After much persistence they have confirmed that a fix will be released as a Cumulative update “soon”. I will update this blog with the appropriate link for the Cumulative update as soon as it is released.

UPDATE: (24th August 2010)
Microsoft has released a patch which resolves this issue in the SQL 2008 SP1 CU9 rollout. You can obtain it from here: http://support.microsoft.com/kb/2083921. I am still awaiting confirmation for SQL 2008 R2. Will update as soon as I have more news.

UPDATE: (27th January 2011)
Microsoft has released a patch which resolves this issue in SQL 2008 R2 CU4 rollout. You can obtain it from here: http://support.microsoft.com/kb/2264562. Although many readers have informed me that the workaround still works faster. Test it and let me know.