The Decimate filter can be used to reduce the number of triangles in a triangle mesh, forming a good approximation to the original geometry. The input to vtkDecimatePro is a vtkPolyData object, and only triangles are treated. If you desire to decimate polygonal meshes, first triangulate the polygons with vtkTriangleFilter object.
The implementation of vtkDecimatePro is similar to the algorithm originally described in "Decimation of Triangle Meshes", Proc Siggraph `92, except that this algorithm does not necessarily preserve the topology of the mesh and it is guaranteed to give the a mesh reduction factor specified by the user.
The algorithm proceeds as follows. Each vertex in the mesh is classified and inserted into a priority queue. The priority is based on the error to delete the vertex and retriangulate the hole. Vertices that cannot be deleted or triangulated (at this point in the algorithm) are skipped. Then, each vertex in the priority queue is processed (i.e., deleted followed by hole triangulation using edge collapse). This continues until the priority queue is empty. Next, all remaining vertices are processed, and the mesh is split into separate pieces along sharp edges or at non-manifold attachment points and reinserted into the priority queue. Again, the priority queue is processed until empty. If the desired reduction is still not achieved, the remaining vertices are split as necessary (in a recursive fashion) so that it is possible to eliminate every triangle as necessary.
To use this filter you should specify the Target Reduction. The algorithm is guaranteed to generate a reduced mesh at this level as long as the following four conditions are met: 1) topology modification is allowed; 2) mesh splitting is enabled; 3) the algorithm is allowed to modify the boundary of the mesh; and 4) the maximum allowable error is set to VTK_LARGE_FLOAT. Other important parameters to adjust include the Feature Angle and Split Angle, since these can impact the quality of the final mesh. Also, you can set the Accumulate Error to force incremental error update and distribution to surrounding vertices as each vertex is deleted. The accumulated error is a conservative global error bounds and decimation error, but requires additional memory and time to compute.