Difference between revisions of "Resource Streaming"

From GTAMods Wiki
Jump to navigation Jump to search
(Resources)
(No linked list ids)
 
(2 intermediate revisions by 2 users not shown)
Line 14: Line 14:
 
! Description
 
! Description
 
|-
 
|-
|Models ([[DFF|.DFF]])
+
|Models (.[[DFF]])
 
|0-5499
 
|0-5499
 
|0-6499
 
|0-6499
Line 20: Line 20:
 
|All game [[model]]s are in these IDs. Internally this range is further split up into subsections of objects, vehicles and peds.
 
|All game [[model]]s are in these IDs. Internally this range is further split up into subsections of objects, vehicles and peds.
 
|-
 
|-
|Texture Dictionaries ([[TXD|.TXD]])
+
|Texture Dictionaries (.[[TXD]])
 
|5500-6350
 
|5500-6350
 
|6500-7884
 
|6500-7884
Line 26: Line 26:
 
|Contains [[texture dictionary]] slots. Texture dictionaries are containers with all game textures. The game assigns texture dictionaries to models.
 
|Contains [[texture dictionary]] slots. Texture dictionaries are containers with all game textures. The game assigns texture dictionaries to models.
 
|-
 
|-
|Collision Sectors ([[COL|.COL]])
+
|Collision Sectors (.[[COL]])
 
|N/A
 
|N/A
 
|7885-7915
 
|7885-7915
Line 32: Line 32:
 
|Slots holding all loaded [[COL|collision]] sectors. Collision sectors are COLL containers that can be referenced and stretch across in-game bounds (based on buildings that use them).
 
|Slots holding all loaded [[COL|collision]] sectors. Collision sectors are COLL containers that can be referenced and stretch across in-game bounds (based on buildings that use them).
 
|-
 
|-
|Scene Sectors ([[IPL|.IPL]])
+
|Scene Sectors (.[[IPL]])
 
|N/A
 
|N/A
 
|N/A
 
|N/A
Line 38: Line 38:
 
|Slots holding all loaded [[IPL#Binary format|IPL sectors]]. Scene sectors are represented by the binary .ipl files in the [[img archive]]s. The game loads them inside of its bounds, only when they are required.
 
|Slots holding all loaded [[IPL#Binary format|IPL sectors]]. Scene sectors are represented by the binary .ipl files in the [[img archive]]s. The game loads them inside of its bounds, only when they are required.
 
|-
 
|-
|Pathnode Containers ([[DAT|.DAT]])
+
|Pathnode Containers (.[[DAT]])
 
|N/A
 
|N/A
 
|N/A
 
|N/A
Line 44: Line 44:
 
|[[Path]]nodes holds paths for AI. They are aligned in 8*8 sectors of 750*750 units. When the game references such a quad, it loads the associated pathnode files, that are numbered after the (height * rowLen + rowIndex) formula.  
 
|[[Path]]nodes holds paths for AI. They are aligned in 8*8 sectors of 750*750 units. When the game references such a quad, it loads the associated pathnode files, that are numbered after the (height * rowLen + rowIndex) formula.  
 
|-
 
|-
|Animation Blocks ([[IFP|.IFP]])
+
|Animation Blocks (.[[IFP]])
 
|N/A
 
|N/A
|7916-7942
+
|7916-7950
 
|25575-25754
 
|25575-25754
 
|[[Animation]] blocks contain all animation sequences. Models can reference animation blocks, so they request them along.
 
|[[Animation]] blocks contain all animation sequences. Models can reference animation blocks, so they request them along.
 
|-
 
|-
|Vehicle Recordings ([[RRR|.RRR]])
+
|Vehicle Recordings (.[[RRR]])
 
|N/A
 
|N/A
 
|N/A
 
|N/A
Line 56: Line 56:
 
|[[Carrec|Recordings]] hold prerecorded game motion data. This motion data is used during missions when you have to chase somebody or drive-by against enemies while a friend is driving.
 
|[[Carrec|Recordings]] hold prerecorded game motion data. This motion data is used during missions when you have to chase somebody or drive-by against enemies while a friend is driving.
 
|-
 
|-
|Streamed Scripts ([[SCM|.SCM]])
+
|Streamed Scripts (.[[SCM]])
 
|N/A
 
|N/A
 
|N/A
 
|N/A
Line 92: Line 92:
 
The game maintains a counter, known as the streaming memory, that gets increased by the size of every resource that gets loaded and decreased by the size of each resource that gets unloaded. When this counter reaches a certain memory limit, the game will unload resources that are still loaded but not used by anything in the game or if none, unload buildings, especially the ones out of sight and far away, until there's enough room for the new resource in the streaming memory.
 
The game maintains a counter, known as the streaming memory, that gets increased by the size of every resource that gets loaded and decreased by the size of each resource that gets unloaded. When this counter reaches a certain memory limit, the game will unload resources that are still loaded but not used by anything in the game or if none, unload buildings, especially the ones out of sight and far away, until there's enough room for the new resource in the streaming memory.
  
== External Links ==
+
== External links ==
 
* {{Icon|SA}} [https://wiki.multitheftauto.com/wiki/GTA:SA_Resource_Streaming GTA:SA Resource Streaming] for a more technical explanation of the streaming system.
 
* {{Icon|SA}} [https://wiki.multitheftauto.com/wiki/GTA:SA_Resource_Streaming GTA:SA Resource Streaming] for a more technical explanation of the streaming system.
 
* {{Icon|SA}} [https://wiki.multitheftauto.com/wiki/GTA:SA_Streaming_Garbage_Collection GTA:SA Streaming Garbage Collection] for a more technical explanation of the garbage collector.
 
* {{Icon|SA}} [https://wiki.multitheftauto.com/wiki/GTA:SA_Streaming_Garbage_Collection GTA:SA Streaming Garbage Collection] for a more technical explanation of the garbage collector.
  
 
{{N|SA|VC|3}}
 
{{N|SA|VC|3}}

Latest revision as of 18:35, 9 August 2016

The Resource Streaming system loads data from disk into game memory on demand, allowing a huge open world environment.

Resources

Resources must be assigned to a index to be able to take a hole in the streaming engine, the user only needs to register model indices in object definitions and the game takes care of automatically assigning indices to other resource types.

As follow is a table of the game resource types and it's index range on each game.

Name GTA III Vice City San Andreas Description
Models (.DFF) 0-5499 0-6499 0-19999 All game models are in these IDs. Internally this range is further split up into subsections of objects, vehicles and peds.
Texture Dictionaries (.TXD) 5500-6350 6500-7884 20000-24999 Contains texture dictionary slots. Texture dictionaries are containers with all game textures. The game assigns texture dictionaries to models.
Collision Sectors (.COL) N/A 7885-7915 25000-25255 Slots holding all loaded collision sectors. Collision sectors are COLL containers that can be referenced and stretch across in-game bounds (based on buildings that use them).
Scene Sectors (.IPL) N/A N/A 25256-25510 Slots holding all loaded IPL sectors. Scene sectors are represented by the binary .ipl files in the img archives. The game loads them inside of its bounds, only when they are required.
Pathnode Containers (.DAT) N/A N/A 25511-25574 Pathnodes holds paths for AI. They are aligned in 8*8 sectors of 750*750 units. When the game references such a quad, it loads the associated pathnode files, that are numbered after the (height * rowLen + rowIndex) formula.
Animation Blocks (.IFP) N/A 7916-7950 25575-25754 Animation blocks contain all animation sequences. Models can reference animation blocks, so they request them along.
Vehicle Recordings (.RRR) N/A N/A 25755-26229 Recordings hold prerecorded game motion data. This motion data is used during missions when you have to chase somebody or drive-by against enemies while a friend is driving.
Streamed Scripts (.SCM) N/A N/A 26230-26311 Streamed scripts holds game code. Those codes are required only during certain circumstances of the game and so doesn't need to be loaded all the time, such as tuning garages code.

Asynchronous Reading

To avoid the need to stop the game to load resources, the game reads it in a different thread from where the game logic and rendering takes place. This allows the massive open world environment of the game without constant loading screens.

Each resource (or couple of resources) can be in any of those states:

  1. The resource is not loaded meaning the game is not needing this resource.
  2. The resource has been asked by the game and is now in a queue to be read by the asynchronous reader.
  3. The resource has been take off the queue and is being asynchronously read from disk into memory.
  4. The resource has been read from the disk into memory and is being converted into an engine object.
  5. The resource is fully loaded and available to be used by the game.

The conversion from the data taken from disk into memory to an engine object is not performed in the streaming thread but in the logic thread which may cause the game to halt for a moment if the object to be converted is too complex.

A double streaming buffer is used to avoid higher dependency of the streaming thread and the logic thread for object conversion. This means that while the streaming thread reads the resource into streaming buffer A the game logic thread is converting the memory data in streaming buffer B into an engine object, then the buffers are exchanged between the threads.

Optimization Techniques

The game uses several techniques to improve the performance of the resource loading, those are:

  • The cd images (that's IMG files) reflects exactly how CD-ROM and disk sectors are stored.
  • The cd image reader picks the from the queue of requested resources the one that's nearest to the previous resource read from the disk.
  • The cd image reader is also able to read more than one resource at once if each follows the end of the other sector and all of them fits in the streaming buffer.
  • As converting the data taken off the disk to actual engine object is performed in the logic thread and may be expensive, the game is able to divide this operation in two frames, although this feature is only available for texture dictionaries.

Sectors

A sector stores the world boundaries of a resource, when the player position intersects any of the sectors the data related to it will be requested to the streaming engine. Since Vice City the game uses collisions sectors and since San Andreas scene sectors.

Garbage Collector

The game maintains a counter, known as the streaming memory, that gets increased by the size of every resource that gets loaded and decreased by the size of each resource that gets unloaded. When this counter reaches a certain memory limit, the game will unload resources that are still loaded but not used by anything in the game or if none, unload buildings, especially the ones out of sight and far away, until there's enough room for the new resource in the streaming memory.

External links