Tutorial 18 - Saving and Loading Complicated Tables

18.1 Recap

   
Tutorial 18 brings together elements from a number of the previous Tutorials. It is an example of where asking the right question makes finding the answer much simpler. Our aim is to be able to load and save level data from our Tower Defence level generator. Most of the data we need is stored in a table. The tricky part is that Codea only allows us to save data as strings at the moment (some smart folks over at the Codea Forum have worked out how to jam data into image files and save those).
  
Tutorial 16 examined a technique for saving simple table structures like a one dimensional array. Initially we thought that saving a more complicated table wouldn't be that tough. All we needed was one of the existing Lua table to string parsers like Pickle / UnPickle. WRONG!
    

18.2 Limitations of Table Serialisation Parsers

  
We discussed Lua tables in Interlude 6. They are simple in concept but capable of representing very complicated structures. Tables can contain all the simple Lua types (booleans, strings, nil, and number), functions, other tables, closures, etc. The most complete table to string parser that we could find was Data Dumper. But even Data Dumper can not handle tables that contain:

• Both light and full userdata
• Coroutines
• C functions

And of course our level data contains userdata (e.g. vec2 is represented using userdata). So where to from here?

    

18.3 The Modified Cell Class

   

This is where asking the right question makes all the difference. Looking at which data we actually need to save, it becomes apparent that we don't need to save the entire table (including functions) as this can all be reconstituted. All we really need from the Grid table is what is contained in each cell. So rather than trying to convert a two dimensional array of cell objects (which contain userdata) into a string, we extend the cell class to provides its contents (state) as a string. We also provide the inverse function which sets the cell state from a string so we can load the level back in. The extended cell class is shown below.

  

  

Once we can get the grid cell contents out as a string it is easy to write a function to create a simple two dimensional table holding each of these strings. Data Dumper makes short work of converting this table to a string which we can save to global data. Even better, Data Dumper saves the table in a format that loadstring can use to rebuild the original table. The updated Main class for dGenerator details these save and load functions.

  

We updated the File Manager class from Tutorial 17 to allow us to select the file to be loaded. As part of this update we got rid of the submenus because we didn't need them and they meant that the user had to do an extra tap to load. In addition, Delete was right next to Load which is poor design. One slip of the finger would be potentially disastrous (particularly because there is no confirmation for delete and no undo). The other improvement to this version of File Manager is we truncate keys and values which are larger than the ListScroll widths. You can see the upgraded version in the screenshot below. The About menu item doesn't do anything at this stage.

     

18.4 Loading & Saving Data

   

Tapping the Save button on the main screen of dGenerator will call the saveLevel() function in Main(). The function starts off by saving the simplified table data in saveGrid. It then generates the key which contains a header "dGen" (to indicate when loading if it is the right data type), the game name and level number, a boolean indicating if the start cell has been selected and its co-ordinates and then a boolean indicating if the end cell has been selected and its co-ordinates. Finally we use Data Dumper to convert saveGrid to a string and save the key and value to global data.

  

Loading a level is the reverse. Tapping the Load button on the dGenerator main screen will call the loadLevel() function in Main. This prints out some debug data but its only compulsory action is to set the App state to stateFileManager. This will draw the File Manager instead of the main screen and handle its touches.

  

Once the user has navigated to global data and selected an appropriate key containing level data, tapping load on the File Manager menu bar will call the loadFile() function in Main. This function splits the key back into its component parts using the explode() helper function and then uses loadstring() to create a function which rebuilds the data table. The createGrid() function was updated so that it can be loaded using this data table.

  

And that is all there is to it. You can use this link to download the entire code including the updated classes.

  

Next up we will look at adding creeps to your levels in a number of waves.

Tutorial 16 - Convert String to Table and Table to String

16.1 Why do we need this?

  
In our last tutorial we developed a simple level editor. This is not much use unless you can save and then load the level data produced. Codea provides a function to saveProjectData and one to saveGlobalData(). Since we would like to save our level data from dGenerator and then read it into our game program, saveGlobalData() is the function we need, as saveProjectData() saves data which is only accessible by the program which saved it.
  
Unfortunately it is not as simple as saying saveGlobalData("Level1", ourTableData) as you can only use this function to save a number, string or nil. Consequently, we need to convert our table to a string in order to save it. To load the data we need to reverse this process and reconstitute our table from the saved string.
  
We used a similar technique in MineSweeper to save and load high scores, albeit we were only concatenating three strings rather than converting a table, but the theory is the same.
     

16.2 Codea Data Persistence Under the Hood

      
There are four primary ways to save and load data in Codea (excluding image data, which we wont discuss in this tutorial). Namely:

1. clearLocalData(), saveLocalData() and readLocalData() - used for storing information which is unique to the program and the device. Other programs and devices running a copy of the same program don't have access to this data. It is useful for things like high scores and perhaps user preferences. For each project, a key of the form Project Name_DATA is used to store local data in the com.twolivesleft.Codify.plist which may be found in the Library -> Preferences sub folder (i.e. the same location as global data).
2. clearProjectData(), saveProjectData and readProjectData() - gets bundled with your program and is not device specific. It can be used for level data, maps, and other program specific data which doesn't change with different users. Project Data is stored in the Data pList in the Documents directory of your App.
3. saveProjectInfo() and readProjectInfo - has similar access to Project Data (bundled with program and is not device specific) but is used for saving metadata about your App. There are two data keys which are natively supported by Codea, "Description" and "Author". The data associated with the "Description" key will show in the Codea Project Browser when this App is selected. If you are planning on using the Codea runtime, be aware that this uses the "Version" key and expects a string. If you use a number like we do then you will get an error and the runtime wont compile. The fix for this is described in our earlier tutorial on submitting an application to App store. If you whip out iExplorer you will see that all of the project information is stored in Info.plist which is part of your application bundle. This pList also includes a key called "Buffer Order" which tells the runtime which order your tabs should be loaded in.
4. saveGlobalData() and readGlobalData() - is data available between all projects on a device. The subfolder Preferences of folder Library contains files including com.twolivesleft.Codify.plist which contains any global data that you have saved (under the key "CODEA_GLOBAL_DATA_STORE"). If you have used the sprite generator Spritely which comes with Codea, you will see that it stores some data in the global data store.

You can read more about what is happening under the hood on the Codea Wiki. Since version 1.4.3 of Codea three additional functions have been provided: listLocalData(), listProjectData() and listGlobalData(). These return the keys already saved in these data stores.
    

16.3 A Solution for Single Dimension Arrays & Tables

    
The simplest version of a Lua table is an array so lets work out a solution for that first. It is an easy matter to concatenate the elements of a table together to form a string, as long as the elements of that table are something that can be converted to a string (i.e. this wont work on a table of tables). It is easy because Lua provides a function to do this very thing. Even better you can specify a delimiter to be placed between each element in the resulting string.
  
The function looks like table.concat(yourArray, "delimiter") so:
  
table.concat({ "one", "two", "three", "four", "five" }, ",") will produce a string "one,two,three,four,five"
   
Assuming we pass in the array to our saveData() function, it will look like:
  
function saveData(mArray)
   
    local saveString = table.concat(mArray, ",")
    saveGlobalData("dataKey", saveString)
     
end
   
Obviously you could just as easily use a global array, in which case you wont need to pass the array to the function. Loading the data is a bit more work, but as with a lot of common problems, it is one which has already been solved (credit to http://richard.warburton.it for writing the explode function which does all the heavy lifting).
      
function explode(div,str) 
    
    if (div=='') then return false end
    local pos,arr = 0,{}
    -- for each divider found
    for st,sp in function() return string.find(str,div,pos,true) end do
        table.insert(arr,string.sub(str,pos,st-1)) -- Attach chars left of current divider
        pos = sp + 1 -- Jump past current divider
    end
    table.insert(arr,string.sub(str,pos)) -- Attach chars right of last divider
    return arr
   
end
    
Using the above function the loadData() function is trivial, and will return your original array:
   
function loadData()
     
    local stringArray = readGlobalData("dataKey")
    return explode(",", stringArray)
     
end
     

16.4 A Solution for Multidimension Arrays & Tables

   
Once you get past the simple array or non-nested table things get complicated quickly. As you can stick just about anything in a table (e.g. another table, closures / functions, Userdata and Metatables) a comprehensive conversion function would need to take all of the possible cases into account. These table serialisation libraries exist but they are overkill for our purposes.
  
Instead we will use the charmingly named Pickle and UnPickle functions written by Steve Dekorte.
    
To be continued...

Interlude 10 - Formatting Text in Columns

We will have one last Interlude before we return to Part 2 of the MineSweeper tutorial. One of the areas of functionality that we wanted to add to the game was keeping track of high scores. The code to save and retrieve this was relatively straight forward (and we will cover it in the tutorial) but printing the results in neat columns turned out to be harder than expected.

After struggling with this for a few days we resorted to the Codea forums and dave1707 pointed out our problem. We will get to this shortly, but first some background.

Using the string.format() function, %w.ts will format a fixed column of width w, truncated at t characters for string s. This will be right justified. If you use the left justify flag, %-w.ts the string will be left justified. So in the code below %-10.10s will print "Easy" left justified in a column 10 characters wide and it shouldn't be truncated. The escape character \t will produce a tab between columns.

The truncation field (.t) is optional. If you don't include this (e.g. %10s), then the column width field works as a minimum. In this instance if you had a string which was longer than w then the additional characters would be printed and the columns won't line up.

More generally, the string.format() function uses the same format identifiers as the c printf function. The identifier field needs to be in the following order.

%	Flags	Minimum field width	Period	Precision. Maximum field width	Argument type
Required	Optional	Optional	Optional	Optional	Required

The Flags available are:

- Left justify. 0 Field is padded with 0's instead of blanks. + Sign of number always O/P. blank Positive values begin with a blank. # Various uses: %#o (Octal) 0 prefix inserted. %#x (Hex) 0x prefix added to non-zero values. %#X (Hex) 0X prefix added to non-zero values. %#e Always show the decimal point. %#E Always show the decimal point. %#f Always show the decimal point. %#g Always show the decimal point trailing zeros not removed. %#G Always show the decimal point trailing zeros not removed.

Note that the flags must follow the % and where it makes sense you can use more than one flag. Finally the available format identifiers are:

%d %i         Decimal signed integer. 

%o              Octal integer. 

%x %X       Hex integer. 

%u              Unsigned integer. 

%c              Character. 

%s              String. 

%f               double 

%e %E      double. 

%g %G      produces either f, e or d type output depending on the argument.

%q              treats double quotes, newline, embedded zeros and back slash as escaped

BUT there is a trick. As dave1707 so helpfully pointed out, this column formatting trick only works for fixed width fonts (i.e fonts where each character is the same width). Most fonts are proportional and won't be formatted as you would expect using the above technique.

In Codea, there are currently two fixed width fonts available: Inconsolata and the various flavours of Courier.

The following test stub indicates how you can use this to provide the output shown in the image at the top of the Interlude.

function setup()

   displayMode(FULLSCREEN)

end

function draw()

   background(0)

   font("Courier-Bold")

   fill(0,0,255)

   fontSize(72)

   textAlign(CENTER)

   text("High Scores", WIDTH/2, HEIGHT/2 + 220)

   fill(255)

   fontSize(24)

   local str

   str = string.format("%-10.10s\t%-10.10s\t%10d", "Easy", "Player 1", 1000)

   text(str, WIDTH/2, HEIGHT/2 + 40)

   str = string.format("%-10.10s\t%-10.10s\t%10d", "Medium", "Player 2", 2000)

   text(str, WIDTH/2, HEIGHT/2)

   str = string.format("%-10.10s\t%-10.10s\t%10d", "Hard", "Player 3", 3000)

   text(str, WIDTH/2, HEIGHT/2 - 40)

end