Note: the video is best viewed in max resolution and full screen to see all the details.

I’ve been having a huge smile on my face all day

For the interested, here’s a screenshot of the rule that enables this trick (click to enlarge):

The important bit is the :complete after the variable $ToComp which performs the calculation.

Here’s another screencast that shows what’s happening behind the scenes (basically, it’s a walkthrough of the neural code in the designer):

• I had never made something like that, while I had plenty of experience designing languages,
• I didn’t really know what the end product would look like or how it would behave,
• and apart from some ‘UML designers’ that render classes, I had never used such a thing for editing code myself.

So then, what made me decide to use this approach instead of the more conventional and traditional way of using text?  First of, a major part of the answer lies in the previous statement . Other reasons were:

• Guided editors (can) lower the entrance level, when done properly (something I needed,… seriously).
• I new I would be needing some good debuggers (lessons learned!!). I could try and incorporate these into some existing editors, or turn out one of my own. At the time, it seemed a better idea to have full control from top to bottom, if I wanted to do this properly.
• I had a bit of experience working in WPF, which gave me some confidence that it was possible.
• Also, I came across this, rather serious attempt at structured editors (originally for C#), which raised my hopes as well.
• Did I already mention you that I was hoping this would make things a bit simpler for other people to understand?

In the end, it turns out I am glad I decided to use this approach, though not exactly for the reasons as previously stated. Except perhaps for the debugging, which really was fairly easy to implement this way. No, the thing that convinced me was the way that it changes how you program. This didn’t happen over night, I only began noticing this the last couple of months, after some 1,5 years of (trying to) program this way.
So what changed: well, over time, when I needed a ‘variable’, instead of thinking ‘variable’ I began to think ‘purple’. This is a shift from thinking in words to thinking in color. The best evidence for this, came in the form of me, continually trying to put a conditional in an assignment as if it were a global (cause I thought I had picked up a global). If you look at the picture to the right, you can probably understand why I erred: the blue colors are just to close to each other.
What’s more, over time I also began to think more and more in visual structures instead of ‘if-else’ or ‘while-do’. When I think about an ‘if’, I think:

In short, when programming, I am thinking more and more in terms of: blue square, red hexagon, white circle with line below (the conditional part),… Or in other words, it feels as if I am programming like building with Lego blocks!!!
I can’t explain exactly how or why this is, but it appears to free the mind for thinking about other stuff. At times, I find myself, just mindlessly putting blocks together to build up the picture that I had previously constructed in my head.  Very pleasing.

So, as a first try, I am very happy with the result, well enough for me to continue this path, and with some minor corrections here and there, keep the design. That is not to say there isn’t room for improvement, I have bucket-loads already of those, just not the time, so most of these will not be for the near future.  Others might be implemented sooner, depending on needs.

Basically, most of the concerns are centered round work speed (the number of movements and clicks required for an operation), maintainability and overview:

Speed

• To start with, the toolbox needs some redesign. I currently have 2 pages: one for general items, and a separate tab for all the instructions. I find myself continually switching between the 2, which is time consuming and unnecessary. The solution is simple: split them up in separate toolboxes.
• The toolbox currently only supports drag-drop operations. So to get 5 objects of the same type, you need to perform 5 drags. Better would be to have a ‘paint’ tool, like graphical apps, and many (UML) designers have.
• The code designer really could use some more keyboard support, so it can be used like a regular text editor (sort of). This seems a logical next step to do, just haven’t gotten there yet.
• I have been thinking a lot about using WPF’s new multi-touch libraries to add support for using 2 mice while coding. I don’t really make use of the left hand during the GUI coding, apart from the occasional copy/paste or editing a label. At times, I could definitely work with 2 hands at the same time. It’s a bit like using both hands for searching and placing blocks.
  I had also thought about using this new kinect toy, which appears to have been hacked already for PC usage. This would even resemble more like building with Lego (or like Tom cruise is doing in Minority report) . But then I figured: ok how would you feel after 2 weeks of keeping  your arms up in the air? Ok, I don’t know how I’d feel after 2 weeks like that, but my shoulders and neck definitely do!!!
  So that plan was canceled.

Overview

• As I’ve previously mentioned, I often simply recreate the picture that’s already in my head. The annoying thing is that the designer only shows a small part of the object (algorithm) that I’m working on, while at times I have more a need for a birds view approach. That’s why I added the zoom support, but more can be done such as:
  • make the code templates (the drawings) a bit smaller. I’m no good at designing, which shows.
  • provide a bread-crumb like ability (for drilling down and going back up again), much like the windows file explorer provides using it’s breadcrumb approach for directories.  I could use this for sub blocks.
  • See what I can do about providing support for multiple screens.
  • Buy a new big sized monitor, but given the massive resources at my disposal, that’s not an option
• Also previously stated: some of the colors can use a little tuning, the same perhaps for some shapes, to make them more distinguishable from each other.
• Currently, the drop area for the arguments is displayed as a simple, single square. The thing is: the argument requirements are known for each instruction, so it’s perfectly possible to display a custom set of boxes, with correct labels, for each instruction. This, by the way, would also free some space, since no drop area needs to be shown when all arguments are provided.
• I could also try to implement some sort of 3D-view on the network model, but I think I’ll need to extend my skill-set for that one.

Maintainability

The single major issue I have with code maintainability is concerning documentation. I added support for this by allowing to add some ‘rich-text’ string to each neuron. The idea being that you’d put all the comments concerning what’s going on, in there. Unfortunately, it doesn’t really work very well for code documentation because:

• neurons can be reused. This is different compared to regular text code: when this is compiled, each line of code will get it’s own output (or better, each compiled instruction corresponds with exactly 1 location within the text). With this neural network however, things are a bit different: it’s perfectly possible to use the same assignment in 2 (or more) different places. This is something I do extensively, by the way, to speed up coding and to keep the nr of links down.  What I noticed is this: though the code is shareable, the reason usually isn’t. I am currently solving this by stacking the different comments in the same text-blob, when needed. But better solutions are possible.
• I also find it annoying having to select an object in order to see it’s comments within the code view. It appears to be easier/better when you can quickly scan a document for all comments and read the one you need, without thinking about selection.

To solve this, I have been thinking about adding support for comments within the ‘project’ documents. These would maintain and display (as call-out balloons) all the comments for the neurons that they contain. The location of the comments would be determined by the comments themselves, so they can be floating, closest to the neuron that they are attached to. This would also allow me to display the comments, even if the actual neurons aren’t visible, but collapsed.

Final note

As a final note, for those who would like to build their own structured editor in WPF, perhaps a word of advice: stay away from ItemsControl or any of it’s descendants. The paradigm sounds grate: manage a list of items and display them any way you want through styling and templates. What’s not mentioned is that this only works for 10 objects or so. Any more, and your app slows down to a crawl or worse: crashes just as bad as the good old win32 apps would, perhaps even worse at times. Instead, go lower level: inherit from panel and roll out your own Measure/Arrange or go even further down and overwrite the rendering routines.

• Scroll & load speed: Even with full virtualization turned on, scrolling crawls almost to a halt when you have 60 000+ root items and a few of them are expanded some levels down the tree. The same for loading large lists.
• Memory usage: probably related with the previous item, in short: kaboem!!!
• BringIntoView: Here’s the killer, when virtualization is turned on, and the UI TreeViewIem hasn’t been rendered yet, you can’t bring it into view. Translated: you can’t do a search and show the found items in large trees, only in small trees (go figure).

Note about my test data:  The thesaurus demo contains about 100 000 words (syn-sets actually), almost all of which have multiple relationships, so  in total, there are +500 000 nodes in the tree.

Ok,… and how do you fix this? Well here’s how I did it:

Container rendering

Virtualization means that you only render the UI elements that are currently visible, and not the whole tree (or a large part like microsoft does). This means that, for each UI object that got rendered, you somehow need to keep track of the data position in the tree. I do this by using 2 cursors: one for the first and another for the last visible element. This cursor is nothing more than a list of indexes into the tree.

All items in between those 2 cursors are flattened out (by rendering UI elements for them). The nr of indexes (or nr of parents) for each item, is used as the level for the UI element. This determines the amount of indentation to the right that is applied to the UI element.

When the user scrolls down, the bottom cursor is updated (advanced by the amount that the user scrolled). This is a basic tree walking algorithm. From this bottom index, we render objects while going up the tree until the panel is full (again, the same tree walking algorithm). The index path of the new top item becomes the new top cursor. The same technique is used to scroll up, but instead, you start from the top cursor and fill downwards.

Remark: Watch out while calculating the scroll difference: The scrollbar value is usually a double while we only consume integer values, so there might be some left over in the scroll difference which has to be consumed the next time you calculate the dif.

Of course, this technique only works if the TreeView has some knowledge of the data structure, how else can it find the children of the root items and see if they are expanded or not? This requires an interface. Here’s how mine looks like:

   public interface ITreeViewItem
   {
      bool IsExpanded { get; set; }
      bool IsSelected { get; set; }
      bool HasChildren { get; }
      IList TreeItems { get; }
      ITreeViewPanelItem ParentTreeItem { get; }
      bool NeedsBringIntoView { get; set; }
   }

Each tree node should implement this interface so that the TreeView can properly traverse the tree. A few properties are mostly for the visual parts, like ‘IsSelected’ or ’HasChildren’ (to display the toggle button). The last property is perhaps a bit of an odd one: ‘NeedsBringIntoView’. It’s a quick method to let the Tree know that an element needs to be brought into the viewable area. You could also use a function on the TreeView (like microsoft does), but then you are forced to go find the UI element from your search code to bring the item into view, which I wanted to avoid.

Using this technique, you do create and destroy a lot of containers since each time the user scrolls, the entire visible area is re-rendered. To minimize the impact of this on the garbage collector, it’s probably best to use some recycling scheme. This doesn’t have to be fancy: a simple queue to put the items in when clearing the list which can be consumed again when rendering the containers. I chose a queue since it gives best chances to reassign the same container to the same data object when possible, which minimizes the impact of re-rendering on the binding system.

Events

The second important thing for virtualized TreeViews, is the event system. You can not simply go and hook up event handlers to every data element. This would blow up the memory. Also, you’d have to write a lot of code to keep track of those events as data objects are being created and destroyed (trees of such a size can only work if you combine UI virtualization with data virtualization, but that’s an entire subject all together). On the other hand, events are VERY important to get UI virtualization working for keeping track of the maximum scroll count, tree nodes that get expanded/collapsed, objects that need to be brought into view, and UI elements that need to be updated.

Are you seeing the bubbles already? Yep: use bubbling events, like WPF does. The only problem: that doesn’t exist for regular objects, it’s only available in the WPF world. You could inherit all your objects from DependencyObject, but with 200 0000 objects in memory, that somehow becomes bloated. Better to roll out your own sleek, mean and fast event mechanism, which isn’t hard at all to do. Here’s a little sample code to bubble events up a tree:

public static void OnPropertyChanged(IOnBubblingChanged sender, BubblingPropertyChangedEventArgs e)
{
   while (sender != null)                                  //use a while loop to avoid recursive calls.
   {
      sender.OnBubblingPropertyChanged(e);
      IOwnedObject iOwned = sender as IOwnedObject;
      if (iOwned != null)
         sender = iOwned.Owner as IOnBubblingChanged;
      else
         sender = null;                                    //need to make the loop stop
   }
}

This function is called whenever a property on a tree item is changed. As you can see, a lot of interfaces are at work here. IOnBubblingChanged defines a function that can be called which will raise the event. IOwnedObject is one of my core interfaces for all objects that can be ‘owned’ by another object. This maps to the ‘ParentTreeItem’ property of the first ITreeViewItem interface.

A similar thing needs to be done for the CollectionChanged events. A remark on the ‘reset collection’ event: make certain that you can pass along the list of items that got cleared, otherwise the TreeView will have a hard time figuring out exactly how much the count needs to be decreased (were any of the removed items expanded?). The default WPF NotifyCollectionChanged event doesn’t do that, so you can’t simply use it’s event arguments.

With this type of bubbling event system, it’s probably best to work with some kind of ‘root’ object that defines the events for the property and collection changes in the entire tree. Again a nice job for an interface, I suppose. This root interface can be used as the ItemsSource for the treeview instead of a list, like WPF does. This way, you can have lots of root objects without overtaxing the .net event management system.

Using this technique, you can make Treeviews that aren’t only fast to load and scroll, but also use very little memory. I am silently hoping that microsoft will eventually pick up on this and begin implementing proper UI virtualization in the coming versions of WPF (thus relieving me from the burden). In the mean time, I guess we are left to roll out our own stuff.

Put into sequence:

1. Thread 1 locks x and continues execution (no waiting)
2. Thread 2 locks y and continues execution (no waiting)
3. Thread 1 requests y and waits
4. Thread 2 requests x and waits,… oeps

The most often way to locate this situation is by examining the execution stack of the CPU. But what if you do out of sync locking: the request in function 1 and the release in function 2, without any relationship between the 2 functions. As an example, take the network-core’s lock expression, which is new. This locks the items when the statement is called, but releases them only after all the child statements were called. The interpreter obviously can’t do this in the same function call.

So how do you trace this type of bug? Well, slowly, painfully and with lots of debug code to generate application dumps. here’s the smallest dump that I generated for the core (usually it was several 100 lines longer):

Links in
ID: 7461 ReadCount: 2  WriteCount: 0, Waiting: 0
Links out
ID: 7461 ReadCount: 2  WriteCount: 0, Waiting: 0
Values
ID: 7461 ReadCount: 2  WriteCount: 0, Waiting: 0
Processors
Parents
ID: 7461 ReadCount: 0  WriteCount: 0, Waiting: 2
Children
ID: 7432 ReadCount: 0  WriteCount: 1, Waiting: 0
ID: 7461 ReadCount: 2  WriteCount: 0, Waiting: 0
ID: 7288 ReadCount: 0  WriteCount: 1, Waiting: 0
LockExpressionCounter: 2
Single lock count: 0

Basically, it allows me to check all the locks that are still active (in order of age) when the deadlock occurred. From then it’s simply a matter of figuring out who has the oldest lock and why it wasn’t released. To do this, it’s best to keep track somehow of the threads that do the locking and make certain that they are named, so you can find them again in the execution stack of the debugger.

In case you are looking for a mental image to visualize this whole concept, try this: Resonance. When a link gets activated (for instance, the one to the very first neuron), it creates a resonance that triggers one or more other neurons. This excitation in turn can cause another link between 2 neurons to be activated, causing more resonance and so and and on until the whole thing settles down.

I have absolutely no prove or idea that this is how it works in the real world, that’s just how my model can be interpreted. Truth be told, this is not how I conceived the thing (aka lets try to create a model that uses resonance), it was rather more like: I have 2 neurons, they are linked, how can I make something happen? Well, I can attach some code to the link and execute that. Cool. But, wait a moment, that looks like resonance…

and we just passed ‘culminate’, pfff

Note: Deprecated!

This is the very first debugger I have written, and I am pretty proud about it! It’s not a masterpiece, but functional.  The code definitely could use some tidying up and some speed tuning wouldn’t hurt at all, but you can trace bugs, inspect values and follow the program flow, and that’s already something I guess. To explain the debugger to you, I thought it perhaps best to do it using some of the demo’s.  Simply open the ‘Echo words’ project to get started.

Set up

Before we send some input to the network, we need to set up the designer so that it is ready to debug:

1. We need to  put a breakpoint  on a statement. You can do this in a code editor. The very first code block that gets called in this network is the rules code on the ‘Contains Word’ neuron, so simply double click on the ‘Code: Contains Word’ node in the Projects tab.
   To set the breakpoint on the first statement, expand the ‘CheckStartTextBlock – space insertion’ code block and click in the little circle on the ‘if’ statement. This should make it turn red.
2. The designer also needs to be set in design mode.  You can do this by the drop down box on the main toolbar or on the debugger tab’s toolbar. There are 3 possible modes:
   • Off: no debugging is possible.  This mode runs faster.
   • Normal: The debugger stops whenever a breakpoint is encountered. When a breakpoint is encountered or a processor was paused, it is possible to inspect values.  This is how most other debuggers provide debug capabilities.
   • Slow motion: In this mode, the debugger automatically pauses and continues on every statement, creating a movie of the execution process. You can use the throttle bar on the left of the combo box to select the speed, which is updated real time. It’s possible to pause and continue this movie using the pause and play buttons on the debugger tab’s menu.  I have included this type of debugging mode because I believe it is sometimes useful to see the path that is taken to a specific point of interest. With this feature, you don’t have to keep pressing F6 to advance.
3. It’s also possible, but not required, to let a processor stop when an error was encountered. This is done by activating the ‘Break on error’ button. This only works when the error happened on a processor in debug mode.

Note that it is not possible to switch between different modes while a processor is running. You can only specify the debug mode for newly created processors.

An overview of all the breakpoints in the project can be found on the right side of the debugger tab. Currently it simply lists all the breakpoints, but new features should be added to this list shortly.

Start a processor

To get started, we need to send  some data to the network through it’s text-sin (sensory interface), so make certain that the ‘Echo channel’ is opened (a communication  channel is the visual interface for a sin). Go to View/Communication Channels/Echo channel, and make certain it is checked (and that the tab is selected).

Once the channel is open, type some text in the input section and send it to the network by pressing enter (or the send button). You should see a neuron appear in the left upper screen, which represents the input event (the neuron that will be solved by the processor). Your text will also appear as a string in the centre dialog screen. There will also appear an object in the centre screen of the debugger tab. This represents the processor that was started and which is handling the input event.

The processor overview contains 2 numbers. The first number represents the name of the processor. This value can be changed and is used to identify it between multiple processors. The second number represents the number of neurons that are left on the stack + the current neuron that  is being solved.

The buttons represent, from left to right:

• a toggle button that can be used to open and close the detailed view for the processor. 
• An indicator that is selected when the processor is paused. This is used to inform you on the run state, it’s not an interactive button.
• An indicator that lets you know that the processor is still running or not. When this is no longer selected, an irrecoverable error occurred in the processor and it is actually dead (in other words, you’ve got a problem when this is no longer selected).

Once a processor is started, the middle square in the right side group on the status bar will be blue.  This will remain so for as long as the network has still a processor running.

Detailed view

If you press the first toggle button, a detailed view of the processor should open in a new tab (using the same name as that of the processor).

This tab is divided into 3 sections (from left to right):

• The content of the execution stack.  This stack contains the neurons that have to be solved by the processor.  You can add and remove neurons from/to this stack using the Pop and Push instructions (The first neuron is automatically added). In our demo, there is only 1 item currently on the stack, that’s the one being solved. This item displays  a square in the stack UI element.
• All the neurons that were assigned as meaning to the links starting from the solvable neuron. The top one will be executed first.  A special case is the ‘Actions’ neuron, which will only be solved when all the others have been, no matter where it is found in the list.
• A stack containing all the processor frames.  A frame represents a single cluster’s code that is being executed by the processor. Since some statements call other clusters, like conditional statements, code blocks,… a processor will usually have a number of frames active. This provides an exact view of the processor’s current execution location.
  The following info is provided for each frame, from top to bottom
  • The code, in textual form. Each code unit can easily be selected separately for debugging purposes (check out the context menu on each item to see what you can do with it).  It’s not possible to edit in this window though (you have to open the editor for this).
  • The name (or id when no name is defined) of the neuron that defines all the code.
  • The relationship between the neuron that defines the code and the actual cluster containing the code.  This can be:
    • Children: in this case, the neuron that defines the code, is the cluster that contains the code.
    • Rules: the neuron that defines the code has a link with meaning ‘Actions’ to a code cluster.
    • Actions: the neuron that defines the code has a link with meaning ‘Actions’ to a code cluster.

The detailed view is great to have an overview of where we are in the processing stage, but it isn’t very useful to  debug a program flow. It’s better to use the code editor for this because it provides a better overview of the connection between the statements.  To do this, open the editor containing the code and make certain the the processor in the debugger tab’s center screen is selected.  This last action is very important for the following reason: a detailed view is opened for a specific processor, so a single tab always links to a single processor.  Code however is not specific to a processor, it can be called by many processors. So it needs to know the processor context for displaying debug information.  It uses the processor that is selected in the  debugger overview for this purpose.  This has a nice  side effect: when you have multiple processors running, you can quickly switch between active processor and view where each one is in the code. Note that the code editor will put a red square around the statement that will be executed next, to indicate execution location.

Do the debug

Code flow

We are ready now to start debugging the network. The first thing you can do is walk through the code using the following commands:

• Run (F5): continue execution until the next breakpoint or until the processor is finished (only active when paused).
• Next step (F6): execute the next step only and  pause again (only active when paused). Note, when  you do a step next on a split instruction that creates multiple sub processors, all sub processors will also wait on the next statement, they wont run wild, but wait until you tell them what to do, which is very useful for inspecting stuff.
• Pause: stop execution and wait until the Run or Next step command have been given (only active when running).
• Stop: Ok, this sounds silly, I know, but I haven’t yet implemented the stop.  I’ve always been able to stop in other ways (I’m running the debugger inside another debugger which is great for creating a schizoid coding mind). If you feel an urgent need for this command, let me know, otherwise, it’ll get in there,  I just have absolutely no time frame for this.

Inspecting values

When a processor is paused, it is possible to inspect the value of any item that returns a result, this includes: variables, globals, result statements, bool expressions and ByRefs. You can do this by selecting the item you want to inspect with the mouse, and pressing F7 or through it’s context menu (Inspect value). This opens a dialog with the debug info of the result.  This can be empty, 1 or multiple neurons.

Debug info for a neuron contains the following info:

• All the incoming links, where they point to and the meaning of the link (note that the meaning UI element still has to be changed into a debug neuron).
• All the outgoing links, where they point to and the meaning of  the link.
• All the clusters that contain the parent node.
• When the parent node is a cluster, all it’s children.
• When the parent is a cluster, the meaning of the cluster is depicted, in brackets, after it’s name.  This is also a debug UI element.

The debug info is depicted recursively for all neurons. This type of debug visualization is used in multiple places throughout the designer.  You might have noticed that the echo channel uses this UI element to depict the incoming and outgoing neurons. The detailed view of a processor also uses it to depict most of the neurons.

Watches

A final feature of the debugger is watches.  These allow you to observe the content of variables and globals in a list for a single processor or across all processors for those that are paused. 

This feature is best experienced using the ‘English language definition’ demo since this performs some splits while processing text input. I have put a breakpoint in the ‘Code: Stage 1.1’ page on the second if statement  (first child if in the left-side path of the only root if) and ran it until ‘Found == Sentence(flow)’ to get the screenshot on the left.

To add watches, drag a variable or global and drop it in the left part of the debugger tab.  This will add it at the bottom of the list. If you are dragging it from a code editor, it is best to hold the ctrl key pressed, so that the item stays at it’s original position.  Note that it’s currently not yet possible to drag from the toolbox.  This is a small bug that still needs fixing. Also note that it’s not yet  possible to remove variables (except by editing the designer file).  This command will be added very soon (just goes to show how fresh the debugger still is).

By default, a list of watches with their values for the selected processor is depicted.  The left side is the name of the variable (or it’s id if no name has yet been assigned). To the right of the name, the content of the variable is displayed. This can be empty, 1 or more neurons, all of which are depicted using the debug UI element for neurons.

If you switch to variable view, the left section of the debugger tab will only display a radio button for each watch. The middle section, which contains processor info, will now display the contents of the selected watch for each processor. 

Also note that, when a processor gets split up into multiple sub procs, the view will switch to a tree.  The root node shows the number of processors contained by the node. If a new input event is sent to the network before the previous has been processed, it is added as a list item, at end of the list. If sub processors split up, more sub nodes are created as children of the nodes that triggered the split (tree structure). This allows you to see how processors are related to each other.  At the moment you can only see the current state, through the tree/list structure.  In the future, an extra view might be added that shows a line view over time to show when and how may processors did a split and when they died out, although that’s just an idea at the moment, so don’t put your hopes up to see it any time soon, there are still far more important things to do.

Errors and warnings

If there are any errors generated by the code, either through the Error or Warning instruction or because of an error in the code, you can view exactly where it occurred. All messages are stored in the log tab. When they are blue, you can double click on them. This will open a code editor with the statement selected  that caused the log item (note: if it is somewhere in a sub section, this is not expanded automatically) Because you can use the same statement in multiple locations, only the first few will be selected, this is to take care of some problems with WPF’s standard controls (will be fixed in the future).

As you can see, it’s all still fresh,  but functional. The debugger has already become a central point of usage for me and I suspect this will only increase, so this will get some extensive testing early on. There are plenty of things that still need adding like conditional breakpoints, counts on breakpoints, enabling-disabling of breakpoints, more commands on the breakpoints-list (like clear, enable/disable all,…),… These things will probably be added as needed.

I am certain there is plenty more to say about NND’s debugger, I just can’t think of anything  anymore, so I guess I’ll leave it by this for today. It’s already turned out a long enough post.