[Pipet Devel] Getting rid of modification of xml definition files

[Brad Chapman]

Jeff wrote:
> Brad, if you want to switch to the use of .n files and Overflow's 
overall
> approach, can you explain (1) how class definitions will be stored 
and
> accessed, and if they'll be distinguished any better from class 
instances,

I'm not sure what you mean by `class` here, I guess you're talking 
about node definitions and node instanaces, right? If so, then nodes 
will still be defined using the *.def files. The information in a 
*.def file is stored in the filesystem where Piper can read the 
information from them when a new node type is added. Jean-Marc likes 
to think of them as being analagous to C header files.
Anyways, that is how nodes are defined.
    When a node is loaded, the information just gets loaded into 
"in-memory" storage and is manipulated from there. Jean-Marc just 
added the code allowing Overflow to read in information from *.def 
files (previously the info came from a macro call).
    So in this case, the definitions of nodes (in the *.def files) are 
static and unchanging, and so they always define a "base state" for 
the node (or something like that). The it is once it is in memory that 
things change (like setting parameters, adding inputs and outputs, 
etc).

> (2)
> how crash recovery will take place (will .n always reflect the state 
of the
> networks?), 

Yup, since you can load a workflow diagram from a *.n file, it has 
enough info to reflect the state of the networks at any time. So to do 
crash recovery, the dl would just make periodic saves of *.n files 
while running, and then make this available should a crash occur. I 
already have a separate thread devoted to filesystem access, so I can 
just set this up to do saves every once in a while.

>> (3) if writing to .n all the time is any better or quicker
>> than writing to .def's all the time.

Well, it is probably about the same time frame to make an individual 
change to a *.def file or to save a *.n file. The big slowdown in the 
*.def system really comes when you are doing something like loading a 
network from a *.n file. In this case, you have to copy over the *.def 
files for every node in the network, and this really slows things 
down. This is really where I see a big bottleneck in the *.def file 
usage situation.

> Isn't it true that, with .n, a single link change will require the 
entire .n
> file (with "tons" of nodes) to rewritten while, with .def, only one 
small
> file
> will have to be rewritten?

Yup, this is true. What I would just do is overwrite the saved *.n 
file (and not modify the file, as I have been doing with the *.def 
files). I can't really remark on the speed for saving a huge *.n file 
(Jean-Marc?) but for the type of networks I'm messing around with it 
is very fast. As I mentioned, this will also occur in a separate 
thread so it shouldn't block the ability of the dl to respond to user 
interface requests.
    More questions? What are your thoughts?

Brad