flipCode - Tech File - Alexander Herz [an error occurred while processing this directive]
Alexander Herz
Click the name for some bio info

E-Mail: softcore@web.de

   04/03/2000, Tech File Update

hi folks...

I just came home from my gf and as there is nobody online whom I could talk with I decided it would be a good idea to write down my latest ideas/expriences with scripts,compilers,c/asm code and shader files..and if I find enough time about my implementation of a dll plugin system....

some time ago I decided that it would be good to have a script... so I wrote some simple parser which recognized tokens and encoded the read source file(c similar) into some binary file..

that was read in by the game and parsed realtime...

the parsing was a goddamn ugly huge if else combination...I just wonder that it worked at all...but it was possible to call real functions(compiled and linked ones) from the script..what is quite handy..but it was far too slow... also I was interested to see how a real compiler works.. as I wanted to have it done my self I stayed away from yacc/lex and extended my old 'compiler'...I just opened the old file containing the lexed code and translated into pseudo asm..well..'just' is the wrong word..that's the real trickey part

I had to rewrite that 3 times before it worked out..I'll give a short describtion how I made it(it is prolly shure NOT the best solution)

C to ASM

1st pass - initialize global vars

I look through all symbols which are (global) variables and and notetheir size and weather they are initialized or not... (the size of the dataseg is written to file firstly) then I parse the code which initializes the global vars... (code starting at 'var=' and ending at ';')

the parsing func(used for all the parsing stuff)
  • 1. put the line to be parsed into a cache
  • 2. compress the cache
    -if there is a functioncall in the cache then the code to execute the func is written to the output file and the part of the cache containing the function call is replaced by the register which contains the return value of the func
    -constants are loaded into registers and the numbers in the cache are replaced by the registers containing em
    -all the rest code which is not ambiguous or dependent on brakets is written to the output file and the respective parts of the cache are replaced with the registers
  • 3. parse the cache
    -recursively look for brakets
    -start in the deepest braket and look for operations with descending priority (cast,indirection,arrays[],*,/,+,-,...) this is similar for most operations...so I'll just make one eg for cast so let's say we found a cast in the cache :'(float*)reg1' now we check if the cast is solveable..means that there is no operation with higher order right to the cast(brakets for eg)...if so we write the code casting reg1(=eax) to float* and remove the cast from the cache (actually there is no code needed in this case..but we remember the new type of the register..so we can do typechecks later)
  • 4. write the result to the adress of our var
    -as soon the deepest braket level is done (the complete braket is replaced by some register containing the result of the operations within the braket) we do the same thing one level higher..so if there is no syntax error in the cache there will be one register in the cache after having processed it completely..that's the result..which is assigned to the adress of the variable which is inited with that code (if there is more left than one register then there is a syntax error(we were not able to solve this line of code))
  • 2nd pass - write strings

    next I write all the strings in the sourcefile to the dataseg eg: "TexFuncs.dll" is 3*4 characters long..so it fits into 3 DWORDs (88 is the offset of the string in the dataseg) so we can write : (the number at the beginning of the line indicates the offset of the command)
    48: mov reg1 , const 1182295380
    72: mov [ const 88 ] , reg1
    104: mov reg1 , const 1935896181
    128: mov [ const 92 ] , reg1
    160: mov reg1 , const 1819042862
    184: mov [ const 96 ] , reg1
    216: mov reg1 , const 4293632
    240: mov [ const 100 ] , reg1
    the numbers moved to reg1 represent "TexFuncs.dll" now we cann access the string with adress 88...

    3rd pass - compile the functions

    now we compile all the funtions...
  • we start with the standart 'header' saving esp and incrementing esp by the amount we need for local vars...
  • then we parse for vars needing initialization(like we did for the global vars)
  • in the second run over the function we look at each line(seperated by ';') weather it is a var def(then we forget it as vars are already inited) or some keyword(like for,if else,while,...,asm) or something else...
  • if it's a keyword we add the code needed to perform what the keyword implys (if the keyword is 'asm' then just labels are solved and adresses of vars are loaded.. the rest of the code goes unchanged)
  • if it's something else:
    -we look weather there is an '=' sign in it..if so then this codeline assign some value to some adress..so we load everything till the '=' sign into the cache and parse it.. (and cry out loud if no adress is returned)..then we put the rest of the line in the cache and assign the result to the adress gathered before
    -if there is no '=' we just put the complete line into the cache and parse it..
  • 4th pass - link info

    last but not least we write the items we have no adress for (extarnal vars/funcs) and their adress within the output file to the outputfile..so the game can link that at load time...

    here comes something interesting... I added a new flag to my virtual cpu which indicates (if set) that a function call or access to some adress ([adress]) is external..external here means that this var/function is a true one(no script..but pentium compatible compiled).. that flag is set beofere calls and [adress]..

    another thing I added I call realtime linking...

    if the VirtualMachine(VM) cannot link(find) some adress it just notes that... now you can somewhere in your code load a dll which registers the adress needed to link your script(this MUST happen before the unlinked adress is used) and after you loaded all the dlls you call a func called RunTimeLink which fills in the missing adresses (and crys if there is still one missing)

    there is hell a lot of more stuff in it..this is just a rough describtion how I did it..if you got any quests..feel free to mail me..

    I just wanted to add that the implementation of arrays in msvc is quite strange... generate an array (DWORD m_array[10];) now m_array==&m_array ..what is quite confusing in my implementation you have to write &m_array to get the adress of the array (like you have to do for any other variabletype in msvc) in my script m_array==m_array[0]

    I added a logfile from my compiler which shows the different outputs for each step of the compile process...

    ------- dynamic shaders -----

    everybody seams to use em today..shaders...q3a started using em... and the system nehind em is rather cleverand gives you much freedom.. I read that q3a compiles the shaders ...so they can be executed fastly... but that adds a problem..if there is a new card coming out with a new special effect (opengl extentionmaybe) which was not implemented in the compiler/game then the level designer cannot access that feature....

    my shaders are written using my c script I added some new asm instructions (like blend,texbegin,dpeth,...) which access their opengl counterparts rather directly so you can fastly do your shader stuff :

    example doing a 3 pass blend:
      sne       //all call rever to external funcs
      texbegin [TexIds]     //activate marble.tga
      blend GL_ONE,GL_ONE     //blend mode
      tcmod TCM_SCALE,2.0,2.0      //scale texture
      tcmod TCM_SCROLL,0.2,0.2     //scoll tex
      push 0.2
      push 0.2
      call TexFuncs_Turb
      sub data,8
      texend       //draw or upload tex to tmu
      texbegin [TexIds+4]     //activate env.tga
      blend GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA //blend mode
      tcgen TCG_ENVIRONMENT     //generate evironment tex coords
      texend       //draw or upl 2nd tex
      texbegin lm_id
      blend GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA //blend mode
      texend       //draw or upl 2nd tex 
      flush       //draw tex not drawn yet
    //se/sne enable/disable the extern flag
    the nice thing here (which soves the problem I mentioned at the beginning) is that you can call external functions (like 'TexFuncs_Turb' ) which may contain code initing a new opengl extention and using it or just anything else..

    but the dll and the textures used in this shader need to be loaded first..so when the shader is used for the first time the LoadTexture function (standart interface) is executed once:
    #define NUM_TEXTURES 20
    extern void* Importer;
    extern void* GLRasterizer;
    __cdecl void LoadTextures()
     float area[4];
    here the dll containing(and registering 'TexFuncs_Turb' via some standart interface) is loaded and afterwards the still missing adress (in the asm code snipped) is filled by calling RunTimeLink().

    //I'll add the complete shader file as attachment..so you can have a look (the constants for opengl are not properly yet..)

    when the shader is beeing unloaded a func call CleanUp is called which ..well..cleans up the shit :)
    __cdecl void CleanUp()
    I atlked about registering function from a dll and mention my plugin system in the begining..but that's enoug of my confusing thougts today..I'll tell next time about it I guess..

    mail me any comments..or whatever...


  • 07/05/2000 - fastculling
  • 05/15/2000 - parties&fpu fun
  • 04/03/2000 - Tech File Update
  • 02/11/2000 - Tech File Update
  • 01/14/2000 - Tech File Update
  • 06/28/1999 - Rendering Cities
  • 06/24/1999 - Bones Animation
  • 06/22/1999 - Introduction

  • This document may not be reproduced in any way without explicit permission from the author and flipCode. All Rights Reserved. Best viewed at a high resolution. The views expressed in this document are the views of the author and NOT neccesarily of anyone else associated with flipCode.

    [an error occurred while processing this directive]