Hi all, in today’s post, we’re going to talk about debugging techniques to help the developers leveraging your .NET scripting engine to debug their scripts. We’ll also take a quick look at how you can detect unused references which is a common source of problems in implementing hot reloading.

This is the eighth in a series of posts on building a .NET 6.0-based C# scripting engine for a game engine. If you’ve not yet read the previous posts, you can start here.

Supporting Script Debugging

First, let’s look at how we can enable breakpoint debugging for our gameplay programmers to debug their C# scripts. Ideally, we want them to be able to break, step through the code and observe variables like this:

Requirements

To support script debugging, there are a few requirements we need to meet:

Script Assembly Built in Debug Config

Debugging is only supported when the script assembly is built without compiler optimizations. Unless you have changed the default configurations, this means that your script assembly must be built in the Debug configuration instead of the Release configuration.

In our sample, we’ve only ever used the Debug configuration so no changes will be needed on our end.

Debug Symbols Must be Available

Debug symbols are not embedded in the script assembly by default. Instead, they are stored in a separate .PDB file alongside the .DLL when the assembly is compiled. There are a few ways to make this available. One is to set the assembly’s project properties in the.csproj file to embed the symbols into the .DLL.

However, if you choose not to do so, we can simply copy the .PDB file over when we copy the .DLL file after compiling the assembly like this:

// Application.cpp
...
namespace Core
{
    ...
    void Application::compileScriptAssembly()
    {  
       ...

        // Successful build
        if (exitCode == 0)
        {
            // Copy out files
            std::filesystem::copy_file
            (
                "./tmp_build/ManagedScripts.dll",
                "ManagedScripts.dll",
                std::filesystem::copy_options::overwrite_existing
            );
            // Copy out the pdb file too
            try
            {
                std::filesystem::copy_file
                (
                    "./tmp_build/ManagedScripts.pdb",
                    "ManagedScripts.pdb",
                    std::filesystem::copy_options::overwrite_existing
                );
            }
            catch (const std::filesystem::filesystem_error& e)
            {
                throw std::runtime_error("Failed to overwrite .PDB file. Is a debugger attached?");
            }
        }
        // Failed build
        else
        {
            throw std::runtime_error("Failed to build managed scripts!");
        }
    }
}

Notice that we explicitly encased the copy operation in a try and catch block? This is to handle the case where a debugger is attached to the application and locks the .PDB file. For this sample, we’re just going to throw an error. However, in a full application, we would want to handle it properly, for instance, asking the user to detach the debugger before trying to copy the .PDB file out again.

No Existing Debugger Must be Attached

Since only one debugger at a time may be attached to any application, in order to debug your scripts, you must make sure that your application is already running with no debugger attached. If you’re working with your application from Visual Studio, that means you must “Start Without Debugging”:

Attaching the Debugger

Once you have all the requirements set up, let’s give it a try using the following steps:

1. Start the application without debugging
2. Open our ManagedScripts.csproj file in Visual Studio
3. Go to Debug > Attach to Process...

4. Ensure that the “Attach to” field indicates: Automatic: Managed (.NET Core, .NET 5+) code
5. Search for your application’s process and select it (in our case, it’s Executable.exe)

6. Click on Attach
7. Open up TestScript.cs
8. Place a breakpoint by clicking on the differently coloured vertical space left-most portion of the code at which you wish to break at

And that’s all there is to enable script debugging for your script programmers!

Detecting Unused References

Next, let’s look at how we can detect memory leaks and unused references. Aside from ensuring that we do not waste unnecessary bytes of memory, this is very useful for debugging issues with hot-reloading. If you recall from our previous blog post, I mentioned that unloading an assembly will fail if there are still objects of types from said assembly that still reside in memory.

While commercial tools like JetBrains’ dotMemory exist, they are not free. So instead, we will use a command line* tool called dotnet-dump, which allows us to dump the managed memory and inspect what lies inside at any point in time.

As the code modifications we are doing in this section are only for a demonstration of how we can use dotnet-dump, these modifications will not be committed to the code repository.

* In this tutorial, we are using PowerShell in Windows Terminal but if you wish, you can use Command Prompt too.

Creating the Bug

In order to demonstrate how dotnet-dump works, we will first modify our code to create a scenario where scripts from the script assembly are created but not removed from memory. This will cause a bug where hot reloading stops working due to the runtime being unable to unload the script assembly.

// EngineInterface.cxx
...
namespace ScriptAPI
{
    ...
    void EngineInterface::Reload()
    {
        // Clear all references to types in the script assembly we are going to unload
        // scripts->Clear();
        // scriptTypeList = nullptr;
        ...
    }
    ...
}

By commenting out the code that clears the scripts list, they will be retained in memory and should prevent the script assembly from being unloaded.

Just in case, test hot reloading after making these changes to make sure that it indeed has stopped working properly.

Installing dotnet-dump

With the bug inserted, we can begin our investigation work. However, dotnet-dump is not installed into the .NET SDK by default. Instead, we need to install it by opening the terminal of your choice and using the following command.

dotnet tool install --global dotnet-dump

If you get an error indicating that dotnet is not recognized, ensure that you have the .NET SDK installed and that your system’s environment PATH variable includes the .NET SDK’s directory.

If it works out, you should see a result like the above image. My PowerShell terminal looks fancier thanks to OhMyPosh and the Pwsh10k theme but the message should remain the same.

Searching for the Remnant Objects

Now that all our set up is done, let’s try to find the issue using dotnet-dump. Firstly, we know from the behaviour of our application that hot reloading is broken. From our knowledge of how hot reloading works, our first suspicion is that there are remnant objects. Since dotnet-dump allows us to inspect objects in memory, it would be easy for us to look for remnant objects by dumping the memory immediately after unloading the assembly.

Currently, our code immediately loads the assembly after unloading so we will need to “pause” it to give us some time to dump the memory. We can do this by adding a line which will prompt the application to wait for user input before continuing.

// EngineInterface.cxx
...
namespace ScriptAPI
{
    ...
    void EngineInterface::Reload()
    {
        ...

        std::cin.get();

        // Load the assembly again
        Init();
    }
    ...
}

Now, let’s run our application without debugging and immediately press space to try and hot reloading. The application will pause to wait for input and that’s where we can fire up another terminal and use dotnet-dump.

Using dotnet-dump‘s collect command, we can dump the memory for a specified process id. To find the process id, simply open Task Manager, go to the “Details” tab and search for your application.

Now let’s use the collect command (swap the 40300 to the PID you see in your Task Manager instead) to dump the memory into a file called memDump.dmp:

dotnet-dump collect -p 40300 -o memDump.dmp

Once we have the dump, we can use the analyze command to open up the dump analyzer:

dotnet-dump analyze memDump.dmp

And then use the dumpheap -stat command to take a look at what’s inside:

Right away, we can see that there is a single instance of the TestScript that has not been successfully freed. That’s our culprit! For comparison, this is how it looks without the bug inserted:

Now that we’re done, we can type exit into the dump analyzer to close the tool.

With the culprit found, we can then do more work based on our understanding of our own application’s scripting engine to isolate and resolve the bug.

Conclusion

Hopefully all of the topics we covered above help you and your team with debugging and isolating issues within your scripting engine or scripts. In the next post, we’ll take a look at how we can modify our application to use a user’s installed .NET runtime instead of bundling the entire .NET runtime and it’s 200+ DLLs in our application.

As usual, you can find the repository of a working solution based on this tutorial here.