Resource Streaming

The Resource Streaming system loads data from disk into game memory on demand, allowing a huge open world environment.

Resources

Resources must be assigned to a index to be able to take a hole in the streaming engine, the user only needs to register model indices in object definitions and the game takes care of automatically assigning indices to other resource types.

As follow is a table of the game resource types and it's index range on each game.

Name				Description
Models (.DFF)	0-5499	0-6499	0-19999	All game models are in these IDs. Internally this range is further split up into subsections of objects, vehicles and peds.
Texture Dictionaries (.TXD)	5500-6350	6500-7884	20000-24999	Contains texture dictionary slots. Texture dictionaries are containers with all game textures. The game assigns texture dictionaries to models.
Collision Sectors (.COL)	N/A	7885-7915	25000-25255	Slots holding all loaded collision sectors. Collision sectors are COLL containers that can be referenced and stretch across in-game bounds (based on buildings that use them).
Scene Sectors (.IPL)	N/A	N/A	25256-25510	Slots holding all loaded IPL sectors. Scene sectors are represented by the binary .ipl files in the img archives. The game loads them inside of its bounds, only when they are required.
Pathnode Containers (.DAT)	N/A	N/A	25511-25574	Pathnodes holds paths for AI. They are aligned in 8*8 sectors of 750*750 units. When the game references such a quad, it loads the associated pathnode files, that are numbered after the (height * rowLen + rowIndex) formula.
Animation Blocks (.IFP)	N/A	7916-7942	25575-25754	Animation blocks contain all animation sequences. Models can reference animation blocks, so they request them along.
Vehicle Recordings (.RRR)	N/A	N/A	25755-26229	Recordings hold prerecorded game motion data. This motion data is used during missions when you have to chase somebody or drive-by against enemies while a friend is driving.
Streamed Scripts (.SCM)	N/A	N/A	26230-26311	Streamed scripts holds game code. Those codes are required only during certain circumstances of the game and so doesn't need to be loaded all the time, such as tuning garages code.

Asynchronous Reading

To avoid the need to stop the game to load resources, the game reads it in a different thread from where the game logic and rendering takes place. This allows the massive open world environment of the game without constant loading screens.

Each resource (or couple of resources) can be in any of those states:

1. The resource is not loaded meaning the game is not needing this resource.
2. The resource has been asked by the game and is now in a queue to be read by the asynchronous reader.
3. The resource has been take off the queue and is being asynchronously read from disk into memory.
4. The resource has been read from the disk into memory and is being converted into an engine object.
5. The resource is fully loaded and available to be used by the game.

The conversion from the data taken from disk into memory to an engine object is not performed in the streaming thread but in the logic thread which may cause the game to halt for a moment if the object to be converted is too complex.

A double streaming buffer is used to avoid higher dependency of the streaming thread and the logic thread for object conversion. This means that while the streaming thread reads the resource into streaming buffer A the game logic thread is converting the memory data in streaming buffer B into an engine object, then the buffers are exchanged between the threads.

Optimization Techniques

The game uses several techniques to improve the performance of the resource loading, those are:

• The cd images (that's IMG files) reflects exactly how CD-ROM and disk sectors are stored.
• The cd image reader picks the from the queue of requested resources the one that's nearest to the previous resource read from the disk.
• The cd image reader is also able to read more than one resource at once if each follows the end of the other sector and all of them fits in the streaming buffer.
• As converting the data taken off the disk to actual engine object is performed in the logic thread and may be expensive, the game is able to divide this operation in two frames, although this feature is only available for texture dictionaries.

Sectors

A sector stores the world boundaries of a resource, when the player position intersects any of the sectors the data related to it will be requested to the streaming engine. Since Vice City the game uses collisions sectors and since San Andreas scene sectors.

Garbage Collector

The game maintains a counter, known as the streaming memory, that gets increased by the size of every resource that gets loaded and decreased by the size of each resource that gets unloaded. When this counter reaches a certain memory limit, the game will unload resources that are still loaded but not used by anything in the game or if none, unload buildings, especially the ones out of sight and far away, until there's enough room for the new resource in the streaming memory.

External links

• GTA:SA Resource Streaming for a more technical explanation of the streaming system.
• GTA:SA Streaming Garbage Collection for a more technical explanation of the garbage collector.