Difference between revisions of "Mobile textures (SA/VC)"

From GTAMods Wiki
Jump to navigation Jump to search
(Seems unlikely)
 
(13 intermediate revisions by the same user not shown)
Line 1: Line 1:
There are multiple texture formats used in the mobile version of GTA:SA, with different formats being used for different platforms in order to take advantage of the strengths of the different graphics/processing chips in devices on each platform.
+
The mobile ports of GTA: San Andreas and GTA: Vice City use texture systems different from their PC and console counterparts in order to reduce the
 +
amount of storage space required for storing textures, and to improve performance by using texture formats which take advantage of the hardware of the target device.
 +
 
 +
The information in this article is based on the formats found in the mobile version of GTA:SA, but most can also be applied to GTA VC.
 +
 
 +
Textures are grouped into separate databases based on which archive file they are in on non-mobile versions. New databases have been added to store textures exclusive
 +
to the mobile games, such as touch controls, updated splash screens and menu icons.
  
 
A single texture database is made up of several files:
 
A single texture database is made up of several files:
* <code>name.txt</code>
+
* <code>name.txt</code> – <b>t</b>e<b>xt</b>
* <code>name.x.toc</code>
+
* <code>name.x.toc</code> – <b>t</b>able <b>o</b>f <b>c</b>ontents
* <code>name.x.dat</code>
+
* <code>name.x.dat</code> – texture <b>dat</b>a
* <code>name.x.tmb</code>
+
* <code>name.x.tmb</code> – <b>t</b>hu<b>mb</b>nails
  
Textures are grouped into separate databases based on which archive file they are in on non-mobile versions, and the 4 files listed above are named
+
The four files listed above are named according to the archive they come from: <code>name</code> is <code>gta3</code> for textures from <code>gta3.img</code>, <code>gta_int</code> for textures
accordingly: <code>name</code> is <code>gta3</code> for textures from <code>gta3.img</code>, <code>gta_int</code> for textures
 
 
from <code>gta_int.img</code>, and so on.
 
from <code>gta_int.img</code>, and so on.
  
Line 79: Line 84:
 
| <code>detailtile</code>
 
| <code>detailtile</code>
 
| Integer
 
| Integer
| The number of times to tile the detail texture (unconfirmed).<br />
+
|  
 
|-
 
|-
 
| <code>alphamode</code>
 
| <code>alphamode</code>
Line 104: Line 109:
 
== .toc ==
 
== .toc ==
 
=== Overview ===
 
=== Overview ===
The <code>.toc</code> file is a table of offsets for the <code>.dat</code> file (which contains all of the texture data). This file is not
+
The <code>.toc</code> (<b>t</b>able <b>o</b>f <b>c</b>ontents) file is a table of offsets for the <code>.dat</code> file (which contains all of the texture data). This file is not
required by the game; a new offset table can be created by reading the <code>.dat</code> file. The purpose of the offset table is to accelerate
+
required by the game: a new offset table can be created by reading the <code>.dat</code> file. The purpose of the offset table is to accelerate
 
texture loading. A game running without prebuilt offset tables will likely suffer from small stutters, generally lower FPS, and textures loading
 
texture loading. A game running without prebuilt offset tables will likely suffer from small stutters, generally lower FPS, and textures loading
 
later than they should (leaving some objects untextured until the texture loads).
 
later than they should (leaving some objects untextured until the texture loads).
Line 113: Line 118:
 
read directly from the data file instead.
 
read directly from the data file instead.
  
The game is capable of generating offset table files for future use, but this code never runs in production builds.
+
The game is capable of generating offset table files for future use, but this code never runs in production builds. It is likely that to generate the
 +
offset files distributed with the game, a late build was run with the generation code enabled.
  
 
=== Format ===
 
=== Format ===
Offset files have a very basic format: one 32-bit unsigned integer (<code>uint32</code>) for the size of the data file,
+
Offset files have a very basic format: one 32-bit unsigned integer (<code>uint32</code>) for the size of the <i>data</i> file,
and then the rest of the file is consecutive <code>uint32</code> offsets in the order of the entries in the <code>.txt</code>
+
and then the rest of the file is consecutive 32-bit signed integer (<code>int32</code>) offsets in the order of the entries in  
file.
+
the <code>.txt</code> file. These offsets are absolute offsets from the beginning of the data file to the start of the texture's data.
 +
 
 +
All entries in the database are given offset values. No meaningful offsets can be given for <code>affiliate</code> entries (as they
 +
are not actually textures themselves), so for these the value -1 (<code>0xffffffff</code>) is used. No other offset is valid for
 +
affiliate entries: -1 is a special case, and everything else is treated as a real offset.
  
To read an offset file, code similar to the following pseudo-C++ may be used:
+
The stored size of the data file is used to verify that the two files match. If this step were left out, the game would
<syntaxhighlight lang="c++">
+
be able to try to read any arbitrary file as an offset file, which would lead to incorrect offsets being used. The data size
 +
also validates the offset file because in almost every case where the data size changes the offset file must be regenerated.
 +
If the offsets are not regenerated, the mismatch of size values allows the game to detect the invalid offsets and ignore them.
 +
 
 +
Below is example code for reading an offset file, which also demonstrates some of the checks the game will perform.
 +
<pre>
 
FILE *offsetFile = std::fopen(pathToOffsetFile, "rb");
 
FILE *offsetFile = std::fopen(pathToOffsetFile, "rb");
  
Line 138: Line 153:
 
}
 
}
  
uint32 *offsetArray = new uint32[database.entryCount];
+
int32 *offsetArray = new int32[database.entryCount];
  
 
std::fread(offsetArray, 4, database.entryCount, offsetFile);
 
std::fread(offsetArray, 4, database.entryCount, offsetFile);
Line 144: Line 159:
  
 
// offsetArray now contains the offsets from the file.
 
// offsetArray now contains the offsets from the file.
</syntaxhighlight>
+
</pre>
  
 
In both error cases in the above code, the game will resort to reading the offsets from the data file.
 
In both error cases in the above code, the game will resort to reading the offsets from the data file.
 +
 +
== .dat ==
 +
The <code>.dat</code> file contains an array of texture listings along with the actual texture data.
 +
 +
Textures are represented in different formats on different platforms. PVR is used for a majority of textures
 +
on iOS devices, because many iOS devices use PowerVR GPUs which have special hardware support for PVRTC.
 +
 +
=== Structure ===
 +
The <code>.dat</code> file consists entirely of listing structures, with texture data following the metadata in each.
 +
 +
The following table defines, in order, the fields of these listing structures.
 +
 +
{| class="wikitable"
 +
|-
 +
! Field
 +
! Type
 +
! Use
 +
|-
 +
| Hash
 +
| <code>u16</code>
 +
| Hash of the texture name. Used to ensure that <code>.dat</code> listings match up with <code>.txt</code> entries.
 +
|-
 +
| Encoding type
 +
| <code>u16</code>
 +
|
 +
|-
 +
| Width
 +
| <code>u16</code>
 +
| The width of the texture in pixels.
 +
|-
 +
| Height mask
 +
| <code>u16</code>
 +
| First 15 bits encode the height of the texture in pixels; last bit indicates the number of mips: if set, mipmapping is disabled for the texture (so only the full-size version is stored). If the last bit is <i>not</i> set, multiple mips are stored (so mipmapping is enabled).
 +
|-
 +
| Compressed size
 +
| <code>u32</code>
 +
| Size of the data when compressed.
 +
|-
 +
| RLE indicator
 +
| <code>u32</code><!-- Could it be u8 but with three bytes of padding? -->
 +
| Zero if the texture is not run-length-encoded. Otherwise, the indicator is used to tell the decoder that a segment is to be repeated.
 +
|}
 +
 +
==== Hash ====
 +
Since the game expects the order of the <code>.txt</code> file to match up with the order of the <code>.dat</code> file, it is important
 +
that the two never get out of sync. In order to prevent this, the <code>.dat</code> file stores a hash of the name given in the <code>.txt</code> file. This allows the game to ensure that each listing matches
 +
up with the information from the <code>.txt</code> file rather than blindly trusting that the files match, but without including the name of the texture in the <code>.dat</code> file listing. The hash is calculated
 +
as follows:
 +
 +
# Start with an unsigned 32-bit hash value of 0.
 +
# For every character byte in the string, shift the hash bits left by 5 and add the byte.
 +
# After the loop, shift the hash bits right by 5.
 +
# Take the two least significant hash bytes to obtain an unsigned 16-bit value.
 +
 +
Sample algorithm (Rust):
 +
<pre>
 +
fn hash_name(name: &str) -> u16 {
 +
    let mut hash: u32 = 0;
 +
 +
    for byte in name.as_bytes() {
 +
        hash = hash
 +
            .overflowing_add((hash << 5).overflowing_add(*byte as u32).0)
 +
            .0;
 +
    }
 +
 +
    hash.overflowing_add(hash >> 5).0 as u16
 +
}
 +
</pre>
 +
 +
==== Width & Height ====
 +
The height mask dedicates 15 bits to storing the height of the texture, but the most significant bit
 +
determines the use of mipmaps in the texture. If the bit is 1, the texture only contains a single image,
 +
but if it is 0, there are multiple mips present. To find the number of mips, the width and height values (with the height
 +
being <code>height_mask & 0x7fff</code>) must be divided by two repeatedly until neither can be divided any further. The
 +
number of mips is always 1 when <code>height_mask</code>'s uppermost bit is set.
 +
 +
{| class="wikitable" style="text-align: center;"
 +
|+ Counting mips for a 32x16 texture
 +
|-
 +
! Iteration !! Mip Width !! Mip Height
 +
|-
 +
| 1 || 32 || 16
 +
|-
 +
| 2 || 16 || 8
 +
|-
 +
| 3 || 8 || 4
 +
|-
 +
| 4 || 4 || 2
 +
|-
 +
| 5 || 2 || 1
 +
|-
 +
| 6 || 1 || 1
 +
|}
 +
 +
Notice how on the fifth iteration the mip height reaches one pixel and that this does not change for the sixth iteration (as it is not possible to divide 1 by 2 using integer division).
 +
After the sixth iteration, iteration stops because both dimensions have reached 1. For square textures, the two dimensions will reach 1 on the same iteration.
 +
 +
==== Run-Length Encoding ====
 +
The game uses additional run-length encoding as a form of lossless compression to reduce the size of many of the stored textures. This compression is only enabled on textures where the RLE indicator is not zero.
 +
 +
Decompression is very simple, and requires a segment size and an indicator byte as well as the compressed bytes. The bytes are split into groups. Each group can be either <i>segment size</i> bytes, or <i>segment size + 2</i> bytes. If the group starts with the indicator byte, the next byte is the number of repetitions. The <i>segment size</i> bytes after that constitute the segment to be repeated. If the first byte in the group is not the indicator byte, a segment is read and appended to the output.
 +
 +
<pre>
 +
decompress(segment size, indicator, compressed) do
 +
  output := new vector
 +
 
 +
  while byte := next byte from compressed do
 +
    if byte = indicator then
 +
      repetition count := next byte
 +
      segment := take next 'segment size' bytes from compressed
 +
     
 +
      'repetition count' times do
 +
        add 'segment' to output
 +
      end loop
 +
    else then
 +
      segment := take next 'segment size' bytes from compressed
 +
      add 'segment' to output
 +
    end if
 +
  end while
 +
 +
  return output
 +
end
 +
</pre>
 +
 +
[[Category:File Formats]]

Latest revision as of 11:56, 2 June 2021

The mobile ports of GTA: San Andreas and GTA: Vice City use texture systems different from their PC and console counterparts in order to reduce the amount of storage space required for storing textures, and to improve performance by using texture formats which take advantage of the hardware of the target device.

The information in this article is based on the formats found in the mobile version of GTA:SA, but most can also be applied to GTA VC.

Textures are grouped into separate databases based on which archive file they are in on non-mobile versions. New databases have been added to store textures exclusive to the mobile games, such as touch controls, updated splash screens and menu icons.

A single texture database is made up of several files:

  • name.txttext
  • name.x.toctable of contents
  • name.x.dat – texture data
  • name.x.tmbthumbnails

The four files listed above are named according to the archive they come from: name is gta3 for textures from gta3.img, gta_int for textures from gta_int.img, and so on.

In these file names, x refers to the file extension of the texture format used. For example iOS devices use the PVRTC format, which means that x is pvr.

.txt

The .txt file contains a list of all the textures in the database, along with various texture properties for each. It also contains information about the database itself.

Properties in the .txt file are set with propertyname=value where value may be an integer or string value. There are two properties – png and img – that are used with hexadecimal numbers (e.g. png=72b88910), but these properties are never read by the game. It is possible that they are some sort of checksum or identifier for texture PNGs/other images.

The affiliate property is set inside quotes – "affiliate=something".

The category line defines default values for texture properties as well as information about the database itself. It must set the cat property in order to be correctly identified as a category line. In GTA:SA, this line is the first in the file, but this is not a requirement.

Texture lines begin with the quoted texture name, such as "ahoodfence2".

See the table below for a list of properties.

Property name Type Meaning
format
Integer Specifies the format of the texture.
mipmode Boolean
hassibling Boolean
hasbias Boolean
camnorm Boolean Unknown. Used for "thin" textures (e.g. leaves, wires).
forcez Boolean Unknown. Used for some glass textures.
decalz Boolean
noalphatest Boolean Unknown. Used for some shadow textures.
hasdetail Integer
isdetail Integer
detailtile Integer
alphamode Integer 1, 2 or 3
streammode Boolean
width Integer The width of the texture, measured in pixels.
height Integer The height of the texture, measured in pixels.
affiliate String A texture that this texture "points to". If a texture is defined as an affiliate, it simply redirects to whichever texture it points to. Affiliate textures do not have any texture data and typically do not define any other properties.

.toc

Overview

The .toc (table of contents) file is a table of offsets for the .dat file (which contains all of the texture data). This file is not required by the game: a new offset table can be created by reading the .dat file. The purpose of the offset table is to accelerate texture loading. A game running without prebuilt offset tables will likely suffer from small stutters, generally lower FPS, and textures loading later than they should (leaving some objects untextured until the texture loads).

The offset table file stores the size of the .dat file it is for in order to allow the game to ensure that the two files are compatible. If the size given in the .toc file does not match the actual .dat size, the table will be discarded and the offsets will be read directly from the data file instead.

The game is capable of generating offset table files for future use, but this code never runs in production builds. It is likely that to generate the offset files distributed with the game, a late build was run with the generation code enabled.

Format

Offset files have a very basic format: one 32-bit unsigned integer (uint32) for the size of the data file, and then the rest of the file is consecutive 32-bit signed integer (int32) offsets in the order of the entries in the .txt file. These offsets are absolute offsets from the beginning of the data file to the start of the texture's data.

All entries in the database are given offset values. No meaningful offsets can be given for affiliate entries (as they are not actually textures themselves), so for these the value -1 (0xffffffff) is used. No other offset is valid for affiliate entries: -1 is a special case, and everything else is treated as a real offset.

The stored size of the data file is used to verify that the two files match. If this step were left out, the game would be able to try to read any arbitrary file as an offset file, which would lead to incorrect offsets being used. The data size also validates the offset file because in almost every case where the data size changes the offset file must be regenerated. If the offsets are not regenerated, the mismatch of size values allows the game to detect the invalid offsets and ignore them.

Below is example code for reading an offset file, which also demonstrates some of the checks the game will perform.

FILE *offsetFile = std::fopen(pathToOffsetFile, "rb");

if (!offsetFile) {
    // Error: unable to open the file.
    // ...
}

uint32 statedSize;
std::fread(&statedSize, 1, 4, offsetFile);

uint32 dataFileSize = /* the size of the data file */;
if (statedSize != dataFileSize) {
    // Error: offset and data files do not match.
    // ...
}

int32 *offsetArray = new int32[database.entryCount];

std::fread(offsetArray, 4, database.entryCount, offsetFile);
std::fclose(offsetFile);

// offsetArray now contains the offsets from the file.

In both error cases in the above code, the game will resort to reading the offsets from the data file.

.dat

The .dat file contains an array of texture listings along with the actual texture data.

Textures are represented in different formats on different platforms. PVR is used for a majority of textures on iOS devices, because many iOS devices use PowerVR GPUs which have special hardware support for PVRTC.

Structure

The .dat file consists entirely of listing structures, with texture data following the metadata in each.

The following table defines, in order, the fields of these listing structures.

Field Type Use
Hash u16 Hash of the texture name. Used to ensure that .dat listings match up with .txt entries.
Encoding type u16
Width u16 The width of the texture in pixels.
Height mask u16 First 15 bits encode the height of the texture in pixels; last bit indicates the number of mips: if set, mipmapping is disabled for the texture (so only the full-size version is stored). If the last bit is not set, multiple mips are stored (so mipmapping is enabled).
Compressed size u32 Size of the data when compressed.
RLE indicator u32 Zero if the texture is not run-length-encoded. Otherwise, the indicator is used to tell the decoder that a segment is to be repeated.

Hash

Since the game expects the order of the .txt file to match up with the order of the .dat file, it is important that the two never get out of sync. In order to prevent this, the .dat file stores a hash of the name given in the .txt file. This allows the game to ensure that each listing matches up with the information from the .txt file rather than blindly trusting that the files match, but without including the name of the texture in the .dat file listing. The hash is calculated as follows:

  1. Start with an unsigned 32-bit hash value of 0.
  2. For every character byte in the string, shift the hash bits left by 5 and add the byte.
  3. After the loop, shift the hash bits right by 5.
  4. Take the two least significant hash bytes to obtain an unsigned 16-bit value.

Sample algorithm (Rust):

fn hash_name(name: &str) -> u16 {
    let mut hash: u32 = 0;

    for byte in name.as_bytes() {
        hash = hash
            .overflowing_add((hash << 5).overflowing_add(*byte as u32).0)
            .0;
    }

    hash.overflowing_add(hash >> 5).0 as u16
}

Width & Height

The height mask dedicates 15 bits to storing the height of the texture, but the most significant bit determines the use of mipmaps in the texture. If the bit is 1, the texture only contains a single image, but if it is 0, there are multiple mips present. To find the number of mips, the width and height values (with the height being height_mask & 0x7fff) must be divided by two repeatedly until neither can be divided any further. The number of mips is always 1 when height_mask's uppermost bit is set.

Counting mips for a 32x16 texture
Iteration Mip Width Mip Height
1 32 16
2 16 8
3 8 4
4 4 2
5 2 1
6 1 1

Notice how on the fifth iteration the mip height reaches one pixel and that this does not change for the sixth iteration (as it is not possible to divide 1 by 2 using integer division). After the sixth iteration, iteration stops because both dimensions have reached 1. For square textures, the two dimensions will reach 1 on the same iteration.

Run-Length Encoding

The game uses additional run-length encoding as a form of lossless compression to reduce the size of many of the stored textures. This compression is only enabled on textures where the RLE indicator is not zero.

Decompression is very simple, and requires a segment size and an indicator byte as well as the compressed bytes. The bytes are split into groups. Each group can be either segment size bytes, or segment size + 2 bytes. If the group starts with the indicator byte, the next byte is the number of repetitions. The segment size bytes after that constitute the segment to be repeated. If the first byte in the group is not the indicator byte, a segment is read and appended to the output.

decompress(segment size, indicator, compressed) do
  output := new vector
  
  while byte := next byte from compressed do
    if byte = indicator then
      repetition count := next byte
      segment := take next 'segment size' bytes from compressed
      
      'repetition count' times do
        add 'segment' to output
      end loop
    else then
      segment := take next 'segment size' bytes from compressed
      add 'segment' to output
    end if
  end while

  return output
end