Scripting Extension: Difference between revisions

From OpenMW Wiki
Jump to navigation Jump to search
No edit summary
 
(18 intermediate revisions by 2 users not shown)
Line 1: Line 1:
These are notes on the scripting extension work, see:
== Overview ==
First described for python (SWIG):
These are notes and current status of extending openmw scripting through SWIG.
https://forum.openmw.org/viewtopic.php?f=6&t=617&start=60#p29966


LUA implementation:
== Current Status ==
https://forum.openmw.org/viewtopic.php?f=6&t=617&start=70#p30090
Python and Lua scripting is fully functional. You can call from the console, or can attach to objects/global scripts with the command <pre>StartExternalScript, "scriptname.py"</pre> or <pre>StartExternalScript, "scriptname.lua"</pre> This way you get access to all the locals and globals as if your script is a normal script. The Python/Lua script has to be in the same directory that other morrowind data/addons are in (one of the data directories).
https://forum.openmw.org/viewtopic.php?f=6&t=617&start=70#p30093


current ideas:
== How to Install and Use ==
I think a post-1.0 goal, of mine at least, will still be to see if I can get an easy generic scripting interface.


Overview:
=== Install Prerequsites ===
* It will be a seperate fork from openmw, as I think a unified scripting language and zini's enhancements to the existing implementation would be best for the community. It's generally better to improve one implementation rather than create several new ones. If, in the future, such extension of scripting is found desireable, then there is no problem incorporating it into openmw.
'''Ubuntu Linux:'''<br />
* There are some things that python/other languages can do better and can be implemented faster than writing a new scripting engine from scratch. For example, you can use all of python's existing modules and capabillities that could enable things like dynamic web content/interaction in games. Some people are already used to lua and python scripting, they could potentially write more complicated scripts or more easily integrate scripts. For discussion on python versus lua, I found this very helpful:http://lua-users.org/wiki/LuaVersusPython -- I think both could be usefull in different cases.
(Should also work in other Linuxes that use the <samp>apt</samp> package manager.)
* Existing scripting IDEs can be used, most scripting language already has built-in verbose debugging. It's possible to generate embedded consoles for testing scripts on the fly. Scripts can be edited and recompiled on the fly, in game.
<pre>
* Security will be an issue (running untrusted scripts), but it something people already deal with in the modding community (e.g, [http://wiki.blender.org/index.php/Doc:2.6/Manual/Extensions/Python/Security Blender Python Scripting)Security]. Concerns can be addressed at a minimum by making users opt in to using python, and can be extended to only allow running scripts signed by a trusted certificate/key.
$ sudo apt-get update
$ sudo apt-get install python3-dev swig liblua5.1-0-dev
</pre>
<!--Add cases for yum and other package managers; Homebrew and/or MacPorts for macOS; Cygwin for MS Windows.-->


Implementation:
=== Get the modified openmw binary ===
* Create a new opcode and scripting command for "StartExternalScript". This new opcode will have execute() function that take a runtime with the name of a script file, create an embedded interpretter for whatever language, and execute the script in the embedded interpretter.


==== Github ====
Command-line example for Unix-like operating systems:
<pre>
<pre>
namespace MWScript
$ git clone https://github.com/maqifrnswa/openmw/tree/python-scripting
{
$ cd openmw/
    namespace External
$ mkdir build && cd build
    {
$ make -j
        class OpStartExternalScript : public Interpreter::Opcode0
</pre>
        {
<!--Add instructions for Cygwin for MS Windows.-->
            public:
                virtual void execute (Interpreter::Runtime& runtime)
                {
                    std::string scriptfile = runtime.getStringLiteral (runtime[0].mInteger);
                    runtime.pop()
                    //create another runtime that belongs to this OpStartExternalScript class so we'll have access in future?
                    Py_Initialize();
                    FILE *file_1 = fopen(scriptfile.c_str(),"r");
                    PyRun_SimpleFileEx(file_1,scriptfile.c_str(),1);
                    Py_Finalize();
                }
        }


        void installOpcodes (Interpreter::Interpreter& interpreter)
==== Ubuntu PPA ====
        {
A Utopic personal package archive (PPA) is available for Ubuntu (and related) Linux:
            interpreter.installSegment5 (Compiler::External::opcodeStartExternalScript, new OpStartExternalScript);
<pre>
        }
$ sudo apt-add-repository ppa:showard314/openmw-scripting
    }
$ sudo apt-get install openmw
}
</pre>
</pre>


It will install the highest version of openmw in all the repositories you have enabled, so if you have the openmw PPA, the daily PPA, and this PPA turned on, it may download versions from unwanted PPAs as each gets updated. Therefore, to be safe, only enable one PPA in the Software Center Sources at a time. This version also comes with "testing.py" as seen in the Example below.


Will also need to define the actual int opcode in compiler::external, registerextensions, etc.
=== Script Testing ===
Make your Python or Lua script; see Python example below. The test the script in the console.  A command-line example for most Unix-like operating systems:
<pre>
$ ./openmw --start="Seyda Neen" --skip-menu
</pre>
<!-- In macOS: *DON'T NEED THIS UNTIL INSTALL INSTRUCTIONS ARE DONE.*
<pre>
/Applications/OpenMW.app/Contents/MacOS/openmw --start="Seyda Neen" --skip-menu
</pre>
-->
<!--Add example for Command Prompt in MS Windows.-->
Then in game, press <code>~</code> to get the Console, and run your script with the new command <code>StartExternalScript, "testing.py"</code> or <code>StartExternalScript, "testing.lua"</code>, or write a script/addon that calls <code>StartExternalScript</code>.


[question: should we create a virtual machine for every script, or just once and save a pristine environment to pass to it?]
=== Python Scripts ===
OpenMW will import your python script as a module and run the <code>run()</code> method when you use the Morrowind command <code>StartExternalScript</code>.


Users can then create normal morrowind scripts such as:
==== Example Script====
Make sure you have a <code>run()</code> method, that is what will be called by OpenMW:
<pre>
#Example file: "testing.py"
#SomethingScript is a script in a .omwaddon file that you create that has a short variable named "hello" you can use for testing.
#It's just a demonstration of setting a local variable, you can comment it out if it causes problems.
from openmw import *


StartExternalScript, "somepythonscript.py"
def run():
or
    print("health: " + str((gethealth("player"))))
target->StartExternalScript, "somepythonscript.py"
    print("hello i'm a criminal")
    setpccrimelevel(100000000)
    print("flying: " + str(getflying("player")))
    setflying("player",1)
    print("flying: " + str(getflying("player")))
    print("random100: " + str(omwget("random100")))
    omwset("random100",42)
    print("random100: " + str(omwget("random100")))
    print("SomethingScript.hello: " + str(omwget("SomethingScript.hello")))
    omwset("SomethingScript.hello",42.0)
    print("SomethingScript.hello: " + str(omwget("SomethingScript.hello")))
    omwcall("MessageBox, \"This is a simple message\"")
</pre>


==== How to Write Python Scripts ====
All commands are lowercase and take the same arguments as before, even optional ones. Commands that can take a reference (commands with <code>-&gt;</code>) now have a first argument which is the reference. That first argument is required. If it is an implicit reference, use <code>self</code>.


Example 1:
<pre>
player->AddItem, "Gold_001", 200
</pre>
is now:
<pre>
additem("player","Gold_001",200)
</pre>


* The scripting language will have access to all the MWScript functions via binding functions. Binding functions can be generated programatically but calling "Compiler:registerExtensions (extension)" and generating c++ code from the resulting information on each function and instruction. A new tool can be made that links against libcomponents.a and generates binding functions and headers.
Example 2:
<pre>
AIWander, 0, 0, 0
</pre> is now:
<pre>
aiwander("self",0,0,0)
</pre>


example tool:
Example 3:
<pre>
"urzul gra-agum"->AIWander, 128, 0, 0, 60, 30, 10, 0, 0, 0, 0, 0, 0
</pre>
is now:
<pre>
<pre>
Compiler::Extensions& extensions;
aiwander("urzul gra-agum", 128, 0, 0, 60, 30, 10, 0, 0, 0, 0, 0, 0)
Compiler::registerExtensions(extensions);
</pre>
std::vector<std::string>& keywords;
extensions.listkeywords(keywords);


Compiler::ScriptReturn& returnType=0;
===== Getting and setting variables =====
Compiler::ScriptArgs& argumentType=0;
<pre>
bool& explicitReference=0;
omwset("localvariablename", value)
omwset("globalvariablename", value)
omwset("objectID.variablename", value)
owmset("globalID.variablename", value)


for each keyword
omwget("localvariablename")
{
[etc.]
std::string cppcode="";
</pre>
std::string return_type="";
Use <code>omwget</code> and <code>omwset</code> instead of <code>get</code> and <code>set</code> to avoid name collisions with built-in functions in Python.
std::string arguments="";
if (extension:isFunction (keyword, returnType, argumentType, explicitReference)
{
  switch(returnType)
  {
    case "f":
      return_type+="Interpretter::Type_Float ";
      return_command = "return runtime[0].mFloat" // maybe runtime.pop() as well?
      break;
    case "S":
      return_type+="std::string ";
      return_command = "return runtime.getStringLiteral(runtime[0].mInteger)"
      break;
    case "l":
      return_type+="Interpretter::Type_Integer ";
      return_command = "return runtime[0].mInteger"
      break;
    and then the empty case:
      return_type+="void ";
      return_command = "return"
      break;
    }
  then parse all the argument types, for example fSl/l can be made into
    arguments = "Interpretter::Type_Float arg0, std::string arg1, Interpretter::Type_Integer arg2, Interpretter::Type_integer arg3={SOMEFLAG}"
    required_pushes=3;
    optional _push=1;
  cppcode = returntype+keyword+"("+arguments+")";
  you can also overload for explicit functions
    arguments = "std:string explicit, Interpretter::Type_Float arg0, std::string arg1, Interpretter::Type_Integer arg2, Interpretter::Type_integer arg3={SOMEFLAG}"
    required_pushes=4;
    optional _push=1;
  if first argument is string, it is explicit. Since we're using default values for optional arguments, could be some problems finding the right function though.


  and you can use cppcode for declaration, output to header file and to implementation code.
===== Non-extension Functions and Instructions =====
  Next generate the code to implement keyword's opcode function
All extensions are directly available for use, but game interpreter commands (like <code>MessageBox</code>) can be accessed with <code>omwcall()</code>:
  that was declared in cppcode. output the following to the implementation code
<pre>
    1) Get the runtime that belongs to this external script, clear stack
omwcall("MessageBox, \"This is a simple message\"")
    2) push required parameters on to runtime stack (first push explicitID if it is explicit)
</pre>
    3) if optional variables are not {SOMEFLAG}, push them on the stack
    4) run opcode.execute(runtime)
    5) then add return_command to the output implementation code


  and then do the same with
The argument to <code>omwcall</code> will be parsed as if it were a normal Morrowind script.
  if (extension:Instruction (keyword, argumentType, explicitReference){...}
}
}</pre>


you should now have declarations and the binding functions, which you can now parse with swig and build as part of the _openmw.so module.
=== Lua Scripts ===
OpenMW will import your Lua script as a module and run the <code>run()</code> method when you use the Morrowind command <code>StartExternalScript</code>


* Binding functions will be made to know the opcode, configure a runtime object using the script manager's current context, and push the variables passed to it in order on to the runtime stack. It will then run the appropriate opcode execute function. The function will be overloaded to allow for both explicit and implicit as well has have default values for optional parameters. See above tool that can generate those functions.
==== Example Script ====
Make sure you have a <code>run()</code> method; that is what will be called by OpenMW:
<pre>
#Example file: "testing.lua"
#SomethingScript is a script in a .omwaddon file that you create that has a short variable named "hello" you can use for testing.
#It's just a demonstration of setting a local variable; you can comment it out if it causes problems.


player->SomeCommand, 10, "somestring"
local testing = {}
would be
somecommand("player", 10, "somestring")


SomeCommand, 10, "somestring"
function testing.run()
would be
    print("health: "..omw.gethealth("player"))
somecommand(10, "somestring")
    print("hello i'm a criminal")
    omw.setpccrimelevel(100000000)
    print("flying: "..omw.getflying("player"))
    omw.setflying("player",1)
    print("flying: "..omw.getflying("player"))
    print("random100: "..omw.omwget("random100"))
    omw.omwset("random100",42)
    print("random100: "..omw.omwget("random100"))
    print("SomethingScript.hello: "..omw.omwget("SomethingScript.hello"))
    omw.omwset("SomethingScript.hello",42.0)
    print("SomethingScript.hello: "..omw.omwget("SomethingScript.hello"))
    omw.omwcall("MessageBox, \"This is a simple message\"")
end
 
return testing
</pre>
 
==== How to Write Lua Scripts ====
All commands are in the <code>omw</code> namespace, are lowercase, and take the same arguments as before, even optional ones. Commands that can take a reference (commands with <code>-&gt;</code>) now have a first argument which is the reference. That first argument is required. If it is an implicit reference, use <code>self</code>.
 
Example 1:
<pre>
player->AddItem, "Gold_001", 200
</pre>
is now:
<pre>
omw.additem("player","Gold_001",200)
</pre>
 
Example 2:
<pre>
AIWander, 0, 0, 0
</pre>
is now:
<pre>
omw.aiwander("self",0,0,0)
</pre>
 
Example 3:
<pre>
"urzul gra-agum"->AIWander, 128, 0, 0, 60, 30, 10, 0, 0, 0, 0, 0, 0</pre>
is now:
<pre>
omw.aiwander("urzul gra-agum", 128, 0, 0, 60, 30, 10, 0, 0, 0, 0, 0, 0)
</pre>
 
===== Getting and Setting Variables =====
<pre>
omw.omwset("localvariablename", value)
omw.omwset("globalvariablename", value)
omw.omwset("objectID.variablename", value)
omw.owmset("globalID.variablename", value)
 
omw.omwget("localvariablename")
[etc.]
</pre>
Use <code>omwget</code> and <code>omwset</code> instead of <code>get</code> and <code>set</code> (for syntax consistency with OpenMW Python scripts; <code>get</code> and <code>set</code> are built-in functions in Python).
 
===== Non-extension Functions and Instructions =====
All extensions are directly available for use, but interpreter commands (like <code>MessageBox</code>) can be accessed with <code>omw.omwcall()</code>:
<pre>
omw.omwcall("MessageBox, \"This is a simple message\"")
</pre>
The argument to <code>omwcall</code> will be parsed as if it was a normal Morrowind script.


function returns will be generated programatically as well based on in encoding
=== Debugging output ===
There's still a bunch of normal Python or Lua debugging output that will go to the terminal; bear with it for the time being.


=== Security ===
Security will be an issue when it comes to running untrusted scripts, but it something people already deal with in the modding community (e.g, [https://docs.blender.org/manual/en/dev/advanced/scripting/security.html Blender Python Scripting Security]. Concerns can be addressed to an extent by making users opt in to using Python or Lua, and could be extended to only allow running scripts signed by a trusted certificate (key).


Outstanding questions:
== To do ==
[*] How to handle saving variables? Between frames, variables can be saved by saving the environment. That also allows us to possibly think of creating each running script as a seperate object in python and make the interpretter environment persistent. Then add the variables to save upon exist (and load upon initiation) through a second binding function. Or we can create multiple independent environments.
sandboxing Lua, options to enable/disable compile and runtime availabillity of scripts, code clean up, script signing/trust
[*] How fast will this be? i'm guessing slower than the MWScript system, but will it be prohibitively slower?

Latest revision as of 19:41, 30 September 2018

Overview

These are notes and current status of extending openmw scripting through SWIG.

Current Status

Python and Lua scripting is fully functional. You can call from the console, or can attach to objects/global scripts with the command

StartExternalScript, "scriptname.py"

or

StartExternalScript, "scriptname.lua"

This way you get access to all the locals and globals as if your script is a normal script. The Python/Lua script has to be in the same directory that other morrowind data/addons are in (one of the data directories).

How to Install and Use

Install Prerequsites

Ubuntu Linux:
(Should also work in other Linuxes that use the apt package manager.)

$ sudo apt-get update
$ sudo apt-get install python3-dev swig liblua5.1-0-dev

Get the modified openmw binary

Github

Command-line example for Unix-like operating systems:

$ git clone https://github.com/maqifrnswa/openmw/tree/python-scripting 
$ cd openmw/
$ mkdir build && cd build
$ make -j

Ubuntu PPA

A Utopic personal package archive (PPA) is available for Ubuntu (and related) Linux:

$ sudo apt-add-repository ppa:showard314/openmw-scripting
$ sudo apt-get install openmw

It will install the highest version of openmw in all the repositories you have enabled, so if you have the openmw PPA, the daily PPA, and this PPA turned on, it may download versions from unwanted PPAs as each gets updated. Therefore, to be safe, only enable one PPA in the Software Center Sources at a time. This version also comes with "testing.py" as seen in the Example below.

Script Testing

Make your Python or Lua script; see Python example below. The test the script in the console. A command-line example for most Unix-like operating systems:

$ ./openmw --start="Seyda Neen" --skip-menu

Then in game, press ~ to get the Console, and run your script with the new command StartExternalScript, "testing.py" or StartExternalScript, "testing.lua", or write a script/addon that calls StartExternalScript.

Python Scripts

OpenMW will import your python script as a module and run the run() method when you use the Morrowind command StartExternalScript.

Example Script

Make sure you have a run() method, that is what will be called by OpenMW:

#Example file: "testing.py"
#SomethingScript is a script in a .omwaddon file that you create that has a short variable named "hello" you can use for testing.
#It's just a demonstration of setting a local variable, you can comment it out if it causes problems.
from openmw import *

def run():
    print("health: " + str((gethealth("player"))))
    print("hello i'm a criminal")
    setpccrimelevel(100000000)
    print("flying: " + str(getflying("player")))
    setflying("player",1)
    print("flying: " + str(getflying("player")))
    print("random100: " + str(omwget("random100")))
    omwset("random100",42)
    print("random100: " + str(omwget("random100")))
    print("SomethingScript.hello: " + str(omwget("SomethingScript.hello")))
    omwset("SomethingScript.hello",42.0)
    print("SomethingScript.hello: " + str(omwget("SomethingScript.hello")))
    omwcall("MessageBox, \"This is a simple message\"")

How to Write Python Scripts

All commands are lowercase and take the same arguments as before, even optional ones. Commands that can take a reference (commands with ->) now have a first argument which is the reference. That first argument is required. If it is an implicit reference, use self.

Example 1:

player->AddItem, "Gold_001", 200

is now:

additem("player","Gold_001",200)

Example 2:

AIWander, 0, 0, 0

is now:

aiwander("self",0,0,0)

Example 3:

"urzul gra-agum"->AIWander, 128, 0, 0, 60, 30, 10, 0, 0, 0, 0, 0, 0

is now:

aiwander("urzul gra-agum", 128, 0, 0, 60, 30, 10, 0, 0, 0, 0, 0, 0)
Getting and setting variables
omwset("localvariablename", value)
omwset("globalvariablename", value)
omwset("objectID.variablename", value)
owmset("globalID.variablename", value)

omwget("localvariablename")
[etc.]

Use omwget and omwset instead of get and set to avoid name collisions with built-in functions in Python.

Non-extension Functions and Instructions

All extensions are directly available for use, but game interpreter commands (like MessageBox) can be accessed with omwcall():

omwcall("MessageBox, \"This is a simple message\"")

The argument to omwcall will be parsed as if it were a normal Morrowind script.

Lua Scripts

OpenMW will import your Lua script as a module and run the run() method when you use the Morrowind command StartExternalScript

Example Script

Make sure you have a run() method; that is what will be called by OpenMW:

#Example file: "testing.lua"
#SomethingScript is a script in a .omwaddon file that you create that has a short variable named "hello" you can use for testing.
#It's just a demonstration of setting a local variable; you can comment it out if it causes problems.

local testing = {}

function testing.run()
    print("health: "..omw.gethealth("player"))
    print("hello i'm a criminal")
    omw.setpccrimelevel(100000000)
    print("flying: "..omw.getflying("player"))
    omw.setflying("player",1)
    print("flying: "..omw.getflying("player"))
    print("random100: "..omw.omwget("random100"))
    omw.omwset("random100",42)
    print("random100: "..omw.omwget("random100"))
    print("SomethingScript.hello: "..omw.omwget("SomethingScript.hello"))
    omw.omwset("SomethingScript.hello",42.0)
    print("SomethingScript.hello: "..omw.omwget("SomethingScript.hello"))
    omw.omwcall("MessageBox, \"This is a simple message\"")
end

return testing

How to Write Lua Scripts

All commands are in the omw namespace, are lowercase, and take the same arguments as before, even optional ones. Commands that can take a reference (commands with ->) now have a first argument which is the reference. That first argument is required. If it is an implicit reference, use self.

Example 1:

player->AddItem, "Gold_001", 200

is now:

omw.additem("player","Gold_001",200)

Example 2:

AIWander, 0, 0, 0

is now:

omw.aiwander("self",0,0,0)

Example 3:

"urzul gra-agum"->AIWander, 128, 0, 0, 60, 30, 10, 0, 0, 0, 0, 0, 0

is now:

omw.aiwander("urzul gra-agum", 128, 0, 0, 60, 30, 10, 0, 0, 0, 0, 0, 0)
Getting and Setting Variables
omw.omwset("localvariablename", value)
omw.omwset("globalvariablename", value)
omw.omwset("objectID.variablename", value)
omw.owmset("globalID.variablename", value)

omw.omwget("localvariablename")
[etc.]

Use omwget and omwset instead of get and set (for syntax consistency with OpenMW Python scripts; get and set are built-in functions in Python).

Non-extension Functions and Instructions

All extensions are directly available for use, but interpreter commands (like MessageBox) can be accessed with omw.omwcall():

omw.omwcall("MessageBox, \"This is a simple message\"")

The argument to omwcall will be parsed as if it was a normal Morrowind script.

Debugging output

There's still a bunch of normal Python or Lua debugging output that will go to the terminal; bear with it for the time being.

Security

Security will be an issue when it comes to running untrusted scripts, but it something people already deal with in the modding community (e.g, Blender Python Scripting Security. Concerns can be addressed to an extent by making users opt in to using Python or Lua, and could be extended to only allow running scripts signed by a trusted certificate (key).

To do

sandboxing Lua, options to enable/disable compile and runtime availabillity of scripts, code clean up, script signing/trust