Scripting Extension

These are notes on the scripting extension work, see: First described for python (SWIG):

https://forum.openmw.org/viewtopic.php?f=6&t=617&start=60#p29966

LUA implementation:

https://forum.openmw.org/viewtopic.php?f=6&t=617&start=70#p30090

https://forum.openmw.org/viewtopic.php?f=6&t=617&start=70#p30093

current ideas: I think a post-1.0 goal, of mine at least, will still be to see if I can get an easy generic scripting interface.

Overview:

• It will be a seperate fork from openmw, as I think a unified scripting language and zini's enhancements to the existing implementation would be best for the community. It's generally better to improve one implementation rather than create several new ones. If, in the future, such extension of scripting is found desireable, then there is no problem incorporating it into openmw.
• There are some things that python/other languages can do better and can be implemented faster than writing a new scripting engine from scratch. For example, you can use all of python's existing modules and capabillities that could enable things like dynamic web content/interaction in games. Some people are already used to lua and python scripting, they could potentially write more complicated scripts or more easily integrate scripts. For discussion on python versus lua, I found this very helpful:http://lua-users.org/wiki/LuaVersusPython -- I think both could be usefull in different cases.
• Existing scripting IDEs can be used, most scripting language already has built-in verbose debugging. It's possible to generate embedded consoles for testing scripts on the fly. Scripts can be edited and recompiled on the fly, in game.
• Security will be an issue (running untrusted scripts), but it something people already deal with in the modding community (e.g, Blender Python Scripting)Security. Concerns can be addressed at a minimum by making users opt in to using python, and can be extended to only allow running scripts signed by a trusted certificate/key.

Implementation:

• Create a new opcode and scripting command for "StartExternalScript". This new opcode will have execute() function that take a runtime with the name of a script file, create an embedded interpretter for whatever language, and execute the script in the embedded interpretter.

namespace MWScript
{
    namespace External
    {
        class OpStartExternalScript : public Interpreter::Opcode0
        {
            public:
                virtual void execute (Interpreter::Runtime& runtime)
                {
                    std::string scriptfile = runtime.getStringLiteral (runtime[0].mInteger);
                    runtime.pop()
                    //create another runtime that belongs to this OpStartExternalScript class so we'll have access in future?
                    //both runtimes can use same context
                    Py_Initialize();
                    FILE *file_1 = fopen(scriptfile.c_str(),"r");
                    PyRun_SimpleFileEx(file_1,scriptfile.c_str(),1);
                    Py_Finalize();
                }
        }

        void installOpcodes (Interpreter::Interpreter& interpreter)
        {
            interpreter.installSegment5 (Compiler::External::opcodeStartExternalScript, new OpStartExternalScript);
        }
    }
}

Will also need to define the actual int opcode in compiler::external, registerextensions, etc.

[question: should we create a virtual machine for every script, or just once and save a pristine environment to pass to it?]

Users can then create normal morrowind scripts such as:

StartExternalScript, "somepythonscript.py" or target->StartExternalScript, "somepythonscript.py"

• The scripting language will have access to all the MWScript functions via binding functions. Binding functions can be generated programatically but calling "Compiler:registerExtensions (extension)" and generating c++ code from the resulting information on each function and instruction. A new tool can be made that links against libcomponents.a and generates binding functions and headers.

example tool that generates header file:

#include <iostream>
#include <algorithm>
#include <vector>
#include <string>
#include <sstream>

#include <components/compiler/extensions.hpp>
#include <components/compiler/extensions0.hpp>

class CodeGenerator
{
    Compiler::Extensions mExtensions;
    std::vector<std::string> mKeywords;
    void test2(std::string keyword);
public:
    CodeGenerator()
    {
        Compiler::registerExtensions(mExtensions);
        mExtensions.listKeywords(mKeywords);
        std::cout << "#include <string>\n\n#include <components/interpreter/types.hpp>\n\n";
        for_each(mKeywords.begin(), mKeywords.end(), bind1st(mem_fun(&CodeGenerator::test2), this) );
    }
};


void CodeGenerator::test2(std::string keyword)
{
    Compiler::ScriptReturn returnType;
    std::string return_declare="";
    std::string return_command="";
    std::string declaration="";
    std::string arguments="";
    int argcount=0;
    Compiler::ScriptArgs argumentType;
    int keywordint=mExtensions.searchKeyword(keyword);
    bool explicitReference=0;
    if(mExtensions.isFunction (keywordint, returnType, argumentType, explicitReference))
    {
        if (returnType=='f')
        {
            return_declare="Interpreter::Type_Float";
            return_command = "return runtime[0].mFloat"; // maybe runtime.pop() as well?
        }
        else if (returnType=='S')
        {
            return_declare="std::string";
            return_command = "return runtime.getStringLiteral(runtime[0].mInteger)";
        }
        else if (returnType=='l')
        {
            return_declare="Interpreter::Type_Integer";
            return_command = "return runtime[0].mInteger";
        }
        else
        {
            std::cout << "error generating " << keyword << " no valid returnType\n";
            //return_declare="void ";
            //return_command = "return";
         }
        declaration= return_declare + " " + keyword;
    }
    else if(mExtensions.isInstruction (keywordint, argumentType, explicitReference))
    {
        declaration= "void " + keyword;
    }
    else
    {
            std::cout << "error generating " << keyword << " not function or instruction\n";
    }

    std::string commas = "";
    std::string optional = "";
    std::string optionalstr = "";
    std::ostringstream convert;
    for(const char* c = argumentType.c_str(); *c; ++c)
    {
        convert.str("");
        convert << argcount;
        if(argcount > 0) commas=", ";
        /// Typedef for script arguments string
        /** Every character reperesents an argument to the command. All arguments are required until a /, after which
            every argument is optional. <BR>
            Eg: fff/f represents 3 required floats followed by one optional float <BR>
            f - Float <BR>
            c - String, case smashed <BR>
            l - Integer <BR>
            s - Short <BR>
            S - String, case preserved <BR>
            x - Optional, ignored string argument
            X - Optional, ignored numeric expression
            z - Optional, ignored string or numeric argument
            j - A piece of junk (either . or a specific keyword)
        **/
        if (*c=='f')
        {
            arguments += commas + "Interpreter::Type_Float arg" + convert.str()+optional;
        }
        else if (*c=='s')
        {
            arguments += commas + "Interpreter::Type_Short arg" + convert.str()+optionalstr;
        }
        else if (*c=='S' || *c=='c')
        {
            arguments += commas + "std::string arg" + convert.str()+optionalstr;
        }
        else if (*c=='x' || *c=='X' || *c=='z' || *c=='j')
        {
            continue;
        }
        else if (*c=='l')
        {
            arguments += commas + "Interpreter::Type_Integer arg" + convert.str()+optional;
        }
        else if (*c=='/')
        {
            optional= "=-123456";
            optionalstr= "=\"OPTIONAL_FLAG\"";
            argcount--;
        }
        else
        {
            arguments= commas + "ERROR IN ARG TYPE";
         }
        argcount++;
    }

    declaration += "("+arguments+");\n";
    std::cout << declaration << "\n";
    return;
}

int main(int argc, char**argv)
{
    CodeGenerator codeGen;
    return 0;
}

more notes:
   you can also overload for explicit functions
   if first argument is string, it is explicit. Since we're using default values for optional arguments, could be some problems finding the right function though.

   and you can use cppcode for declaration, output to header file and to implementation code.
   Next generate the code to implement keyword's opcode function
   that was declared in cppcode. output the following to the implementation code
    1) Get the runtime that belongs to this external script, clear stack (or create new runtime with scriptmanager's current context or console's context)
    2) push required parameters on to runtime stack (first push explicitID if it is explicit)
    3) if optional variables are not {SOMEFLAG}, push them on the stack
    4) run opcode.execute(runtime)
    5) then add return_command to the output implementation code

you should now have declarations and the binding functions, which you can now parse with swig and build as part of the _openmw.so module.

• Binding functions will be made to know the opcode, configure a runtime object using the script manager's current context, and push the variables passed to it in order on to the runtime stack. It will then run the appropriate opcode execute function. The function will be overloaded to allow for both explicit and implicit as well has have default values for optional parameters. See above tool that can generate those functions.

player->SomeCommand, 10, "somestring" would be somecommand("player", 10, "somestring")

SomeCommand, 10, "somestring" would be somecommand(10, "somestring")

function returns will be generated programatically as well based on in encoding

Outstanding questions:

• How to handle saving variables? Between frames, variables can be saved by saving the environment. That also allows us to possibly think of creating each running script as a seperate object in python and make the interpretter environment persistent. Then add the variables to save upon exist (and load upon initiation) through a second binding function. Or we can create multiple independent environments.
• How fast will this be? i'm guessing slower than the MWScript system, but will it be prohibitively slower?