Using AVS/Express

11 Shared Libraries

This chapter describes the shared libraries both in UNIX (shared libraries) and in Windows (DLLs).

This chapter includes:

11.1 Loading Methods Dynamically (UNIX)

You can build and run methods that are part of the main AVS/Express executable without having to close and restart AVS/Express. This allows you to build and test modules as you develop them without having to exit from AVS/Express. To do this you must build the modules in a library for which the "dyn_libs" property has been set.

Note: This feature is not available on the PC platforms or on IBM AIX 4.1 in this release. It will be supported on the PC and on IBM AIX 4.2 in a later release.

Writing a Dynamically Shared Method

In order to dynamically load methods while running AVS/Express, you make the following changes while writing modules:

Set the new dyn_libs property on the user library, for example Workspace 1, that will be the parent of the module containing the methods.
Set the library path variable for your platform (e.g. LD_LIBRARY_PATH) to include the directory in which you will build the external library.

You can then add the module and create the method template and parameter declarations in the library in the usual way. You compile and build the external library outside AVS/Express.

Note: Recall that if the method is a cxxmethod you will want to set the out_hdr_file and out_src_file properties on the method to generate separate files for the class header and source information. You use these files when building the external dynamically loaded library.

Setting the dyn_libs property on a library causes AVS/Express to exclude these libraries from the process build files. When the project is compiled, the methods in modules in the dyn_libs libraries appear as "unregistered." When you run a project executable in which AVS/Express encounters unregistered methods in libraries for which the dyn_libs property is set, AVS/Express uses the library path variable to look for the methods. When it finds the methods, AVS/Express loads the methods from the external library and executes them.

Note: If the module is very large there may be a noticeable delay when the library is first loaded.

Building the Shared Library

You must compose a makefile and build the shared library containing the methods outside of Express. Currently the makefile necessary to build the dynamic library (outside of the Express process build) is not generated automatically.

Note: You must write the makefile to comple the library

For platforms for which AVS/Express supports shared libraries, you can see the appropriate compile and link flags to use in your makefile by looking at the platform specific MACH_CFLAGS, MACH_FFLAGS, MACH_CXXFLAGS (for compiling) and DLFLAGS, etc. (for linking) found in include/$MACHINE/machinc.mk in the Express install directory.

For other platforms it might be necessary to look at compiler and loader help files and manual pages. It might also be helpful to look at the help files on the system dynamic loading commands themselves. The commands used by Express can be seen in the include/avs/port.h file by searching for DL_OPEN, etc.

Editing and Re-Loading a Dynamically Shared Module.

After you have created the shared modules, use the following procedure to edit and reload the methods:

1. Delete all instances of the shared module in AVS/Express.

The OM keeps a counter of all references to external libraries, and it will not reload the methods while current references exist

2. Edit the source code and rebuild the shared library outside of AVS/Express.

3. Execute the AVS/Express project again.

As above, AVS/Express will load and execute the new code

Understanding the dyn_libs Property

The dyn_libs property is a string list separated by whitespace specifying the name of the libraries that contain the method(s) to be invoked. You can set the dyn_libs property on a library using the Properties Editor (All Properties). The dyn_libs property can be set on an individual module, but the operating systems load only at the library level. In most cases, the best use of a library is as a collection of related methods that would be likely to be used together.

The dyn_libs property is referenced:

during the Compile Process phase of module development
when a method located in a dynamic library is to be run

When the process is compiled, i.e. "base -comp_proc xxx" is run (either directly or via the Add Module/Object Editor), methods that are members of libraries with the dyn_libs property effect the code generation process in the following way:

The source and object files for the methods are excluded from the process makefile. You will build them separately from the process itself.
The OMadd_named_func() calls for the methods are excluded from the BASEinit() function of the process. This causes those methods to be unregistered which, in turn, triggers the Object Manager to look for dyn_libs property when the methods are to be run.
Prototypes for the methods are excluded from process header files.

As described above, when the Object Manager tries to run a method and finds that it is unregistered, it will look for the dyn_libs property on the library containing the method template. If dyn_libs is found, the Object Manager will use the library path variable to find and load the external library specified in the dyn_libs property, look up the function address for the method within it, and execute the method.

The dyn_libs Property and C++ code

It is important that you use the dyn_libs when doing project generation for a project that uses C++ code. The name mangling performed by C++ compilers can confuse the logic that looks up functions in dynamic libraries, leading to errors about not being able to find or load functions. The dyn_libs property alters to project generation process so that this problem is avoided.

As an example, consider the WriteUCD project at the IAC. If you don't use the dyn_libs property, the generated header file will have declarations as follows (shown somewhat simplified).

void WriteUCD__free(void *obj);
OMXgroup *WriteUCD__inst(OMobj_id obj_id);
int WriteUCD__update__stub(OMXgroup *obj, OMevent_mask event_mask, int seq_num);

However, the C++ compiler will generate function names that do not exactly match the function names as shown above. AVS/Express will look for a function called, e.g., "WriteUCD__update__stub", but the name in the dynamic library will not match.

With the dyn_libs property, the generated header file will have declarations as follows.

extern "C" {
void WriteUCD__free(void *obj);
OMXgroup *WriteUCD__inst(OMobj_id obj_id);
int WriteUCD__update__stub(OMXgroup *obj, OMevent_mask event_mask, int seq_num);
}

The extra 'extern "C"' declaration instructs the C++ compiler to not use name mangling when compiling these functions. As a result AVS/Express' dynamic loading logic should be able to successfully find the function.

11.2 Dynamic Link Library - DLL (Windows)

With Release 4.0, the architecture of AVS/Express on the Windows platforms has been changed to move the core libraries from the executable express.exe, to a DLL file, express.dll.

The express.dll library includes the AVS/Express libraries that support the API's most commonly used in module methods. These include all the OM, OMX, OMF, FLD and FLDF functions, as well as the common ARR, FILE, ERR, MAT and EV utilities in the TOOL and WTOOL libraries.). In addition, some libraries on which the API libraries depend are also included in the DLL, namely COMM, XDR, and GDIF.

To take advantage of the express.dll, you build your modules as DLL programs external to AVS and reference the express.dll. Then, during development, you can modify, rebuild, and test your module dll without having to restart AVS/Express. For detailed instructions about how to develop modules using express.dll, see Writing Dynamically Loaded Modules In AVS/Express on page 11-9.

Note: For this release the Object Manager will not automatically generate the makefile needed to build the external DLL. It is assumed that AVS/Express developers on Windows platforms are reasonably familiar with this procedure, or can use tools such as those found in Microsoft's Visual Studio to automate the process.

Within AVS/Express itself, the W3C (Web) library (Read_URL, ReadWebField and ReadWebGeom macros) and the Read/Write image libraries are implemented as DLL libraries that share express.dll with the main executable. The W3C library uses the Object Manager dynamic loading property, while the Read/Write image libraries use their own dynamic loading method.

Changes in Memory Usage

AVS/Express' use of memory at start-up in Release 4.0 is larger than an equivalent statically linked version. All code within a DLL is loaded when it is first referenced, and express.exe implicitly loads express.dll and agx.dll at start up. The memory is used for the Import Address Table and other code overhead that is needed to support the DLL mechanism.

However, if you run multiple copies of the AVS/Express executable, there will be a saving in memory since the express processes can share the same instance of the express.dll.

The dynamic loading capability can also potentially lead to reduced memory usage because the DLL mechanism defers loading code for module methods until the first reference to the library.

Runtime Considerations.

If you are shipping an AVS/Express application you do not need to change your procedures for Release 4.0. AVS/Express installs the required express.dll file in runtime\bin\%MACHINE% and automatically copies it to the application directory when you save the application.

If you build your own DLLs, it is good practice to build them in the runtime\bin\%MACHINE% directory. However, you will need to copy your DLL files to the final runtime application directory. AVS/Express does not automatically copy them from the runtime directory.

Required Changes for All Users who Build Projects

If you save compiled project in AVS/Express, the following changes in your current use of AVS/Express are required by the introduction of the express.dll architecture:

Regenerate existing project makefiles and recompile source code.
Remove any extern statements used to reference AVS/Express global data.
Do not statically link projects.
Update any local copies of .v files to use Release 4.0 files.

Each of these changes is described in detail in the sections that follow.

These changes are necessary to allow the express executable in your projects to properly access the express.dll. You must make these changes even if you do not intend to write DLLs that are external to AVS/Express.

Regenerate Existing Project Makefiles and Recompile Source Code

The AVS/Express core libraries that were formerly statically linked are now implicitly linked by referencing the import library for the express.dll (express.lib). After installing Release 4.0, you must recompile your source code to ensure that AVS/Express functions and global data are being correctly referenced.

If You Use the AVS/Express Generated Makefiles

If you use the express.mk file generated by AVS/Express, you must regenerate the project makefiles for all your existing projects. For example, run the command `base -gen-proc express -exit' in each project directory.

The Object Manager generates the appropriate express.mk makefile to reflect the changes in this release. You do not need to make any other changes to the regenerated makefile.

After you regenerate the express.mk make file, you must recompile your source code to ensure that AVS/Express functions and global data are being correctly referenced.

If You Use a Makefile Not Generated by AVS/Express

If you use a makefile you have written yourself or generated with another tool such as MS Developer Studio, you must remove references to the AVS/Express static libraries.

If you hand edit your existing 3.3 or 3.4 makefile, remove the following line:

XP_LIBS = libomx.a libom.a libcomm.a libtool.a liblic.a libxdr.a

and add the following references:

XP_LIBS = wsock32.lib express.lib

If you use DevStudio, follow these steps:

n With the project open, from the Project menu on the DevStudio menu bar, select Project Settings.

n Select the Link tab and set the Category list box to Input.

n Remove the AVS/Express static libraries from Object/Library modules field: libomx.a, libom.a, libcomm.a, libtool.a, liblic.a, and libxdr.a.

n Make sure the Object/Library modules field includes references to wsock32.lib and express.lib

Important: Update any local copies of .v files to use Release 4.0 files

Some users maintain local copies of AVS/Express v files to preserve customizations they have made in the files. If your project has a local copy of the v files for the libraries included in express.dll that are not replaced when you regenerate the project, you must update your copies with the Release 4.0 version of these files. These libraries are FLD, FLDF, GDIF and WTOOL. Their respective v files are fld.v, fldf.v, gdif.v and templ.v.

In Release 4.0, these v files have been changed by the addition of a new property, "in_dll." This is an internal property, not for use by developers, that is used when AVS/Express generates the makefile express.mk. This property causes the makefile generator to exclude the libXXX.a string in the express .exe link command. If you are using an obsolete version of the v file, you will see an error from the linker when you try to build your project signalling a missing LIB*.a file.

IMPORTANT: The practice of maintaining local copies of v files should be avoided. As in this case, this practice makes updating to new versions of AVS/Express more complicated and can lead to compatibility problems between your projects and new releases of AVS/Express.

Remove Any extern Statements Used to Reference AVS/Express Global Data

If you are writing your own module methods, you should always use the include files that are supplied or generated by AVS/Express when you compile your module methods. This ensures that the compiler can resolve macro definitions, perform correct function prototype checking, and dereference global data correctly. If you use extern statements to resolve AVS/Express global dependencies, your application will not perform correctly.

Before using Release 4.0, replace any extern statements by including the appropriate AVS/Express header file. You must then recompile all your source code so that the compiler includes the correct dereferencing code for the data in the DLL. The results are unpredictable if this de-referencing is incorrect. Your program may access incorrect data, and there are no warnings from the compiler/linker that alert you to the problem.

Do Not Statically Link Projects

The AVS/Express static library archive (*.a) files for libraries now included in express.dll are not part of this release. It is not possible to statically link express.exe with these archive files and have the resulting executable function correctly.

Writing Dynamically Loaded Modules In AVS/Express

In order to dynamically load methods while running AVS/Express, you make the following changes while writing modules:

Set the dyn_libs Code Management property on the user library that will be the parent of the module containing the methods. The dyn_libs property lists the names of the external DLL files that contain the methods called in the library.

Note: Do not include the .dll extension on the library name property. The Object Manager will assume the standard system extension for the current platform when loading the library. Using library names without an extension allows you to use the same library properties for both the Windows and UNIX platforms. However, you must be sure to use the standard extension in the name of your program files.

Set the PATH environment variable to include the directory in which you will store the external DLLs, or store your DLL in the default DLL search path.

Windows looks for your library DLL in the following order:

1. The directory containing the EXE file. i.e. express

2. the process's current directory

3. The Windows system directory

4. The Windows directory

5. the directories listed in the Path environment variable.

You can then add the module and create the method template and parameter declarations in the library in the usual way. You can use the same method attributes and properties as in previous releases, and AVS/Express generates source code as expected. You compile and build the external DLL library outside AVS/Express. You must write the make file and the supporting DEF file for your module library. See, Building the External DLL on page 11-11.

Note: If the method is a cxxmethod you must set the out_hdr_file and out_src_file properties on the method to generate separate files for the class header and source information. You use these files when building the external dynamically loaded library.

For more information about using libraries and managing processes, see Chapter 10, Projects and Processes.

For more information about adding methods to AVS/Express see Chapter 8, Developing Modules.

How express.dll Works

Setting the dyn_libs property on a library causes AVS/Express to exclude these libraries from the express process build files. When you compile the project, AVS/Express marks the methods in libraries with the dyn_libs property as "unregistered".

When you run the project executable and AVS/Express encounters these unregistered methods in libraries for which the dyn_libs property is set, AVS/Express explicitly loads the library DLL file listed in the dyn_libs property, looks up the module method within the DLL, and calls the function.

Note: If the module is very large there may be a noticeable delay when the library is first loaded.

Building the External DLL

You build the DLL from the source code files generated by the AVS/Express Add Module wizard. However, you must write the makefile with which to build your DLL and a DEF file listing the definitions you want to export from the DLL. As of this release, AVS/Express does not automatically generate these files.

For a detailed example of how to build an AVS/Express module as an external DLL, see Example: Using an External DLL Method on page 11-12. The sample DEF file and make file for this example are included in the AVS/Express install directory in xp_dll/example/win32.

Editing and Re-Loading a DLL

During development, you can edit and reload the DLL while running the express process by performing the following steps:

1. Delete all instances of the DLL module in AVS/Express.

The OM keeps a counter of all references to external libraries, and it will not close the library while current references exist. Note: If you attempt to rebuild the DLL while instances of it are still current in AVS/Express, the linker recognizes that the file is still in use and generates an error similar to the following: LNK1168: Cannot open mymods.dll for writing.

2. Edit the source code and rebuild the DLL program.

3. Reinstance the module or reload the application just rebuilt.

AVS/Express executes the module by loading the new code from your external DLL program.

If you are working with a complicated network, you can remove all instances of the DLL modules, by saving your project application and then deleting the application from Express. You can then rebuild the DLLs, and then select File-> Load Application to load the application and reload the DLL modules' modified code.

Understanding the dyn_libs Property

The dyn_libs property is a string list separated by whitespace specifying the name of the DLL files that contain the method(s) to be invoked. You can set the dyn_libs property on a library using the Properties Editor (All Properties). The dyn_libs property can be set on an individual module, but the operating systems load only at the library level. In most cases, the best use of a library is as a collection of related methods that would be likely to be used together.

The dyn_libs property is referenced:

when you build the express process during module development
when AVS/Express executes a method located in a dynamic library

Generating Code for a Dynamically Loaded Module

When you build the express process, methods that are members of libraries with the dyn_libs property effect the code generation process in the following way:

The source and object files for the methods are excluded from the process makefile. You will build them separately outside of AVS/Express.
The OMadd_named_func() calls for the methods are excluded from the BASEinit() function of the process. This causes those methods to be unregistered which, in turn, triggers the Object Manager to look for dyn_libs property when the methods are to be run.
Prototypes for the methods are excluded from process header files.

You build the express process either directly by using the command line command base -gen_proc or base_comp_proc in the project directory, or indirectly by selecting the Edit Source or Compile Process buttons in the Add Module wizard or the Project-> Edit Source, Project-> Compile, or Project-> Saved Compiled Project menu commands from the Network Editor menu bar.

Executing a Dynamically Loaded Module

Example: Using an External DLL Method

The following example, outlines the procedure for created a DLL library, `mymods', containing an OMX module named `adder.' that has been created in the AVS/Express user library, Workspace 1. When the module was created, the out_hdr_file attribute was set to xp_adder.h, and the out_src_file attribute was set to xp_adder.cxx.

Samples of the files created in this example, including the DEF file, mymods.def, and the make file. mymods.mak, are included in the AVS/Express install directory in xp_dll/example/win32.

The basic steps in the example are the following:

Step 1: Create Project Workspace

1. Start AVS/Express.

2. Select the Project->Save As menu command from the Network Editor menu bar.

3. In the Select Directory dialog, specify a new directory name, `dll_proj,' and click OK.

AVS/Express creates a new express directory named `dll_proj.'

Step 2: Set dyn_libs property on the parent library for the DLL

1. Select the Library Workspaces page from the Libraries option menu.

2. Right-click on the border of the Workspace 1 sublibrary to display the pop-up menu and select Properties.

3. In the Properties Editor, select Code Management Properties from the Property Group option menu.

4. From the All Properties scrolling list, select dyn_libs, set its value to "mymods", and click OK to close the dialog.

You return to the Network Editor.

Step 3: Use the Add Module wizard to create the module and source code

1. With Workspace 1 still selected, select the Object -> Add Module . . . menu command from the Network Editor menu bar.

This launches a wizard-like dialog box.

a. Change the name of the module to adder.

2. Add a method.

a. Click the Add... button in the Add Method to Module section of the dialog box. This adds a cxxmethod object to the module.

b. Change the name of the method to update.

c. Click Done.

3. Add the parameters.

a. Click Next> so that page 2 of the module wizard is displayed.

b. Change the Type option menu of the Add parameter to module section of the wizard from boolean to int.

c. Click Add... This adds an int parameter to the module.

d. Change the name of the parameter to input1.

e. Turn on the toggles labeled Object is an exported parameter and Display input port.

f. Click Next>. The next page specifies the interaction between the input1 parameter and the update method.

g. Turn on the toggles labeled notify, read, and req.

h. Click Done.

You return to page 2 of the wizard.

i. Perform exactly the same steps to add the input2 parameter.

j. In the same way, add the result parameter, except display the output port instead of the input port, and turn on only the write toggle on the parameter/method interaction page.

k. Click Next> to go to page 3 of the wizard.

4. Specify source code file and out_src_file and out_hdr_file properties.

a. In the Source filename field, specify the filename adder.cxx.

b. In the Advanced Properties panel set the out_scr_file property to xp_adder and the out_hdr_file property to xp_adder.h

5. Edit source code.

a. Click the Edit Source button. The OM Compile message appears that asks if you want the OM to generate the method code.

b. Click Yes on this dialog.

Now a text editor appears, displaying the contents of the adder.cxx file that the OM generated in your project directory.

c. In the text editor, remove the ERRverror function call from the update function code, and add the following line instead:

result = input1 + input2;

d. Save the file and exit from the editor. You are now back at the module wizard.

6. Compile Process to generate .h and .cxx files for DLL

a. Click the Compile Process button.

AVS/Express generates the xp_adder.cxx and xp_adder.h files and attempts to compile the express process. Because you are still working in the express process, the compile cannot be completed and an error message is displayed. You can disregard the error message.

b. Click Done on Add Module wizard.

You return to the Network Editor.

Step 4: Save Project

n Select the Project -> Save menu command from the menu bar.

Step 5: Rebuild Express

1. Exit Express.

2. In the project directory: base -comp_proc express -exit.

3. Restart Express.

Step 6: Outside AVS/Express, Build the Module DLL

1. Write DEF file.

The DEF file lists the library symbols you want to export. For the current example, the DEF file, mymods.def, is as follows.

LIBRARY mymods

EXPORTS
WORKSPACE_1_adder__update__stub
WORKSPACE_1_adder__inst
WORKSPACE_1_adder__free

2. Write DLL makefile.

You must write a makefile that builds the DLL with the appropriate linking. For the current example, the makefile, mymods.mak, is as follows:

!include d:\dll_proj\include\xp_defs.mk

DLL_OBJ_FILES= xp_adder.obj adder.obj

mymods.dll: $(DLL_OBJ_FILES) mymods.exp
$(LINK) $(MACH_DLLFLAGS) /out:mymods.dll mymods.exp \
$(DLL_OBJ_FILES) $(XP_LIBS)
mymods.exp: mymods.def
$(AR) $(ARFLAGS) /def:mymods.def /out:mymods.lib

xp_adder.obj: xp_adder.cxx
$(COMPILE_CXX) /c xp_adder.cxx
adder.obj: adder.cxx
$(COMPILE_CXX) /c adder.cxx

clean:
-$(RM) $(DLL_OBJ_FILES) *.ilk *.pdb *.lib *.dll *.exp

Executing this make file, builds the program mymods.dll. When you instance the adder module in AVS/Express, it calls the exported functions in mymod.dll.

You can now instance and use the adder module from Workspace 1 in AVS/Express. When the module method is executed, AVS/Express loads the mymods.dll and executes the update method there.

The dyn_libs Property and C++ code

As an example, consider the WriteUCD project at the IAC. If you don't use the dyn_libs property, the generated header file will have declarations as follows (shown somewhat simplified).

void WriteUCD__free(void *obj);
OMXgroup *WriteUCD__inst(OMobj_id obj_id);
int WriteUCD__update__stub(OMXgroup *obj, OMevent_mask event_mask, int seq_num);

With the dyn_libs property, the generated header file will have declarations as follows.

extern "C" {

void WriteUCD__free(void *obj);

OMXgroup *WriteUCD__inst(OMobj_id obj_id);

int WriteUCD__update__stub(OMXgroup *obj, OMevent_mask event_mask, int seq_num);

}