Upgrade Notes for Developers

What's In This Presentation

These slides provide a general overview of the upgrade process, specifically "internal upgrade", as it is now implemented in ASE 15. They cover:

• Some basic facts about upgrade
• Steps involved in upgrade
• Adding upgrade items
• A "how-to" cookbook for adding upgrade tasks
• Debugging hints for common upgrade problems.

This presentation is a collection of information that server developers and debuggers will need. It is not an explanation of how customers upgrade their installations. It talks about specific files that exist in the server layer, but not the user application layer that invoke them.

Some Basic Facts About Upgrade

Steps Involved in Upgrade

Adding Upgrade Items

How-To

1. Add a new system table
2. Add a column to a system table
3. Add an index to a system table
4. Everything that can be done through SQL
5. Everything else.

First, note that every upgrade item can have several status bits set:

• UPGDATTR_NOATTRIBUTES

This is the "no special status" attribute. This item applies in all databases.

• UPGDATTR_MASTER_ONLY

This status only applies when upgrading the master database.

• UPGDATTR_SYBSECURITY_ONLY

This status only applies when upgrading the sybsecurity database.

• UPGDATTR_SYBSYSTEMDB_ONLY

This status only applies when upgrading the sybsystemdb database.

• UPGDATTR_OPTIONAL

This upgrade item is optional: a failure to install it does not cause upgrade to fail, unless this item is being installed as a dependent item for another item that is not optional. This bit can be OR'ed with other status bits.

0. Steps common to ALL modifications

Before we even get started on how to do things using internal upgrade, there are steps that happen for every upgrade item. Those steps are listed here, to give us a convenient reference for them:

Edit utils/attrib.lst.

Find the end of the list of upgrade attributes. (In that file, you can search for the tag "END of UPGRADE attributes", or you can use the array Upgrade_items[] in utils/intupgd.c to help - the attribute numbers are in comments above the items. Searching attrib.lst for that number usually works.) Add a definition for your item, using the existing items as a model. Give the item a mnemonic that will help others understand what it is doing.

As required, modify the number of the "checkpoint" attribute for the version you're working on so that your item has a lower number than it does.

Edit utils/intupgd.c.

In the array Upgrade_items[], add an upgrade item to identify the new item. Choose the appropriate action ID for your item: you can select from the list in array Upgd_fns[]. (You may have to add your own ID; we'll talk about that in section 5, below.)

If your item depends on other upgrade items, add a dependency list for it; you can use item 1133 and array Ckpt_11_9_3_attr_list as models for how to create a dependency list.

Add your item as a dependency in the present server version's dependency list.

Select attributes for the new item as appropriate, using the mnemonics listed above.

The "data" fields (one integer and one string) vary widely in how they are used. If you are reusing a previously existing upgrade function, use an example of that type to see what information you must provide. If you are adding a new upgrade function, you can design the inputs as needed for your application; but design carefully: other developers may want to reuse your work.

Note that it is not possible to change the data storage design of items already in the field - they can be extended, but not changed! The data is stored in customer databases, and there is no facility to change it.

Compile intupgd.o.

This will create the required new header and source files containing your new upgrade item definition.

Those steps happen for every upgrade item. In the text below, I will refer to them by saying "perform the common steps listed in item zero, above."

1. Add a new system table

Upgrade action ID UPGD_CREATE_TABLE is a "does-everything" item that creates a system table, including all its defined indexes. Use this in preference to typing in SQL statements; it will generate the correct SQL for this table based on information you will place in demo/syscol.c. It works both for real and fake tables.

In the include directory, add a header file for the structure for your table.

This structure definition will give you the required data types and offsets for the table's column definitions. As a model, use an example showing the appropriate locking style for your table: include/syssn.h is an example for row-locked tables, and include/lock.h is an example for page-locked tables. TAKE NOTE of the required row overhead in each table style, including the row-length word at the end of the fixed-length columns!

Edit demo/syscol.c:

Add an array of SYSCOLs for your table, showing all the columns that table should have. (The SYSCOL is defined in include/catalog.h.) All fixed and all variable-length columns must be represented in this array, including any padding space in the fixed part of the row. The only parts of the row that are not represented are the row overhead:

• for page-locked tables, the attribute-count byte, row ID, and row length
• for row-locked tables, the row ID, status word, attribute-count word, and row length.

If your table has indexes:

• add one array of INDCOLs per index showing the array index and column ID of the index key columns
• add an INDDEF listing the INDCOL arrays in index order.

Add your table's information in the Systables[] array, in the slot matching the table's ID.

Each slot is tagged to show its ID. If this is a fake table, be sure to mark it as SYSTAB_ISFAKE in that array.

Edit include/catalog.h:

• add your table's ID
• if your table is master-only:
• adjust MAXMASTER to show your table's ID
• adjust SYSIND_IN_MASTERDB, adding the number of indexes your table has
• if your table appears in all databases:
• adjust MAXUSER to show your table's ID
• adjust SYSIND_IN_ALLDB, adding the number of indexes your table has
• add constants for the array indexes in demo/syscol.c of the columns code will search on.

Note that the array indexes are not the same as column IDs! The SYSCOL entries themselves hold the column ID. The definitions you provide are indexes into the array of SYSCOLs. The server uses these indexes for setting up internal scans of system tables. You are only required to provide mnemonics for the columns that code will actually search.

Perform the common steps listed in item zero, above.

2. Add a column to a system table

Upgrade action ID UPGD_ADD_COLUMN adds a column to an existing system table. Use it in preference to typing in an "alter table" SQL statement. It will generate the correct SQL for this table based on information you will place in demo/syscol.c.

Modify include/catalog.h

(This step may be unnecessary.) If you are adding a column that code will search on, add the appropriate mnemonic for its syscol.c array index. See "add a new system table", above, for more information. If you added the new syscol.c entry above any previously existing entries, adjust the mnemonics as required for the columns that have moved.

Modify demo/syscol.c

Add an array entry to the appropriate SYSCOL array for the table you are modifying. This entry need not be in any particular order; you can add it to the end of the array.

Perform the common steps listed in item zero, above.

3. Add an index to a system table

Upgrade action ID UPGD_CREATE_INDEX creates the index on a system table having a given index ID. Use it in preference to typing in a "create index" SQL statement. It will generate the correct SQL for this table based on information you will place in demo/syscol.c.

Modify include/catalog.h

Add the appropriate mnemonic for the new index's ID. This is the index ID, not the array index. It is not used by upgrade directly, only by code that will do scans on that index.

Modify demo/syscol.c

Add an array of INDCOLs for the index you are adding, listing the index columns. Place that array below the SYSCOL array for that table. Add an entry to the array of INDDEFs for that table, pointing to the new array and giving the index name.

Perform the common steps listed in item zero, above.

4. Everything that can be done through SQL

Upgrade action ID UPGD_SQL_STMT executes random SQL. Anything that can be done using SQL, that hasn't already been addressed by other functions in internal upgrade, should be done using this method.

Perform the common steps listed in item zero, above.

No, really: that's it. No other setup required. However ...

Remember that SQL statements must be idempotent!

Any SQL statement used in upgrade must first check whether the work has been done, and only do it if it hasn't been done. The only exception to this is for statements that don't have any bad side effects from being done more than once. This is required because we don't know when a crash may happen: if the server should happen to crash between executing a statement and recording that it was done, it could be executed twice.

Check for permissions failures.

Upgrade executes SQL statements with a special flag, P2_INTERNALSQL, set in the PSS. Most server code has been modified to permit random SQL when this flag is set. However, we don't guarantee to have caught all cases. If the code area you will be exercising has special permissions requirements, check your work to ensure that it permits internal SQL statements. If it does not, find the failure point and add a check for P2_INTERNALSQL.

Unfortunately, internal SQL can be fooled into thinking it failed.

Our error handling mechanism places an error code into the PSS whenever an error above a certain severity is printed. This happens because the error is printed, without regard for whether the error was raised through ex_raise() or simply printed through ex_callprint(), and without regard for whether the operation ultimately succeeds or fails.

Internal SQL inspects that error code, and presumes that if it is non-zero then there was an error. This is required because internal SQL depends on sequencer(), which always returns "success" if it exits normally; and that can mask certain error conditions that were caught by code lower down. Fixing this will require some amount of redesign within the server.

5. Everything else

Many upgrade items can't be accomplished using SQL statements: for example, anything that requires modifying page headers, allocation pages, and similar - no SQL exists to do that.

These actions require special functions to do the work. Internal upgrade establishes a special function type, UPGD_ACTION_FN, for these functions, and collects them into an array, Upgd_fns[], in function upgd__get_method(). Before you do anything else, check that array to find out whether the function you need has already been written. If it has, or if you can reasonably easily modify an existing function to also do your task, do so.

Presuming that you need a new function, write it:

• Add a function of type UPGD_ACTION_FN in utils/intupgd.c.

This function must be able to extract all the data it needs from the arguments that are provided to it in the DBTABLE and the Upgrade_items[] entry you will provide. This function may be no more than a stub that calls other functions. Make it as general-purpose as you can, so that others might also be able to use it for future upgrade tasks.

• Modify include/intupgd.h, adding a mnemonic for the new function's action ID (that is, its array entry in Upgd_fns[]).
• Add a pointer to the function in utils/intupgd.c.

Be very sure that its action ID addresses the correct array entry. Otherwise, the upgrade driver will encounter errors when it tries to invoke that method by its action ID.

• Add a phrase to identify your new function's operation, using msgdb.

Each upgrade step announces when it begins an upgrade step. The function's action is identified by a server phrase, so unless there is an already existing phrase that explains exactly what your function is doing, you will need to add one. Remember to get that phrase reviewed.

Perform the common steps listed in item zero, above.

Debugging Upgrade