Retrieving, or scraping info from the internet can be done remarkably easy with the chatbot designer. Here’s a screencast of a bot that retrieves weather information from the google weather api.

In the video, a .net plug-in is used to retrieve information from the internet by means of XPaths. This plugin is included by default in the application. Note though that plug-ins are only supported in the pro version.  Basic users will be able to use these projects, but they can’t create or edit any patterns that rely on plug-ins. Also, plug-ins are loaded on a project by project basis. So if you want to use the scraping features in your own project, you will first need to make certain that the correct .net functions have been loaded. Once this has been set up though, all plug-ins will automatically be loaded when the project is opened.

Loading

To load a plug-in, go to view/communication channels/OS. This will bring up a view like the one on the right. From here, you can load and unload dlls, classes and functions. First up is the dll. This can be loaded with one of the buttons on the toolbar. The first one gives access to the cache (dlls that have already been loaded). With the next button, you can select a file from disk. Note that, even though the ‘CmdShell.dll’ file (which contains the scraping functions) is part of the installation, it isn’t guaranteed that it’s already loaded in cache, so you might have to select it from the ‘program files/Chatbot designer pro/’ path. By the way, you can remove a dll by selecting it and pressing delete. Functions can be selected/deselected with the checkbox in front of the name. You can alternatively (de)select the entire class or lib at once. Notice the blue label behind each function name: this is the name that you can use in the patterns. You see, the do-patterns evaluator has no knowledge whatsoever of namespaces, classes or functions, it just knows a single name. This means that all function names should be unique across a single project. If you try to enter a duplicate name, a red box will be displayed round the newly mapped name.

There are quite a few functions available for scraping. Basically though, there are 3 groups: some functions to open/close web-pages, some functions to get data from those opened pages and finally the same functions that don’t require you to first open/close any files but which can do a scrape directly.

Short scrapes

Depending on how much data you need to retrieve, you can use one or the other technique. If there is only 1 xpath that you have to run on a page, then you can probably best use the short/direct functions that don’t require you to first open the web-page. Instead the address is supplied as an argument, together with the xpath. Here’s a list of the available quick scrapers:

And a short usage example to get the temperature info from the google API for a city that’s defined in ‘$place’:

$value = ScrapeText(“http://www.google.com/ig/api?weather=$place:interleaf(+)”, “/xml_api_reply/weather/current_conditions/temp_c/@data”)

As you can see, the first argument specified the web-page to open. The second is an xpath to the data attribute of the ‘temp_c’ element. Note that we use ‘:interleaf(+)’  cause the google API expects city-names that contain multiple words to be separated with a ‘+’ like: New+York.

More scraping

The second scraping method is primarily useful if you need to run multiple xpaths on the same content. In this case, it’s far more economical to first retrieve the page, run all the queries on the cached file and finally, when done, release it again. This can be accomplished with the remaining scrape functions.

You open a file or webpage with either ‘OpenScraper’ or ‘OpenScraperHTML’. The first works on xml content, the second on html. That is, the second will convert html to xml so that the xpath can be run on it. Both return an integer that needs to be used in subsequent calls. Basically, the integer replaces the filename as a reference. It allows you to have multiple files open and to have the system run multi-threaded and let it serve multiple people at the same time.

The scraping functions themselves are almost identical as the quick versions, except that they take an integer as first argument instead of a path. Other then that, usage is exactly the same, with the same types: one for text, integers, doubles and dates.

Once you are done with the file, you have to call ‘CloseScraper’ with, as argument, the integer that was returned by ‘OpenScraper(HTML)’, so that resources can be cleaned up. This is important, if you forget to do this, the system will eventually buckle, crack and give up.
In a normal usage situation, you would do a short salvo: open a page, do a few scrapes and close it again, all in 1 block, but this is not a requirement, you can keep the page open across multiple inputs. As long as you maintain a reference to the scraper (the integer) somewhere in memory so that you don’t loose track of it.

Html scraping

As already mentioned, html scraping is done by first converting the page into xml before the xpath is executed. This conversion can cause some ‘changes’ in the structure of the file. In other words, the path that you would calculate, based on the html file might not be correct for the xml version. This means that you best build your xpaths based on the xml version of the HTML pages.

The conversion routine that’s internally used by the chatbot designer is based on the SGMLReader library. This provides a command-line tool to manually convert html to xml files. This can be very useful for building the correct query. I’ve included a direct download for the command line html to xml conversion tool. Here’s a short description on how to use it (taken from the original documentation):

sgmlreader <options> [InputUri] [OutputFile]

Examples:

sgmlreader -html *.htm *.xml
Converts all .htm files to corresponding .xml files using the built in HTML DTD.

sgmlreader -html http://www.msn.com -proxy myproxy:80 msn.xml
Converts all the MSN home page to XML storing the result in the local file “msn.xml”.

sgmlreader -dtd ofx160.dtd test.ofx ofx.xml
Converts the given OFX file to XML using the SGML DTD “ofx160.dtd” specified in the test.ofx file.

Building an XPath

Once you have your xml file, getting the xpath to the element that you want can still be a little challenging. Html files simply aren’t designed with this type of usage in mind (and hey, if it can be easier for xml files, why not). Enter FireBug, an add-on for Firefox that allows developers to get a closer look at the html…. Or xml. After you have installed firebug and loaded up the xml file into firefox, go to tools/Web developer/Firebug/Open firebug so that you can see the debug panel. In this panel, select the element that you which to query, open the context menu and select ‘copy XPath’. And that’s it, simply paste this path in the chatbot designer and your done.

Note: the video is best viewed in max resolution and full screen to see all the details.

• I had never made something like that, while I had plenty of experience designing languages,
• I didn’t really know what the end product would look like or how it would behave,
• and apart from some ‘UML designers’ that render classes, I had never used such a thing for editing code myself.

So then, what made me decide to use this approach instead of the more conventional and traditional way of using text?  First of, a major part of the answer lies in the previous statement . Other reasons were:

• Guided editors (can) lower the entrance level, when done properly (something I needed,… seriously).
• I new I would be needing some good debuggers (lessons learned!!). I could try and incorporate these into some existing editors, or turn out one of my own. At the time, it seemed a better idea to have full control from top to bottom, if I wanted to do this properly.
• I had a bit of experience working in WPF, which gave me some confidence that it was possible.
• Also, I came across this, rather serious attempt at structured editors (originally for C#), which raised my hopes as well.
• Did I already mention you that I was hoping this would make things a bit simpler for other people to understand?

In the end, it turns out I am glad I decided to use this approach, though not exactly for the reasons as previously stated. Except perhaps for the debugging, which really was fairly easy to implement this way. No, the thing that convinced me was the way that it changes how you program. This didn’t happen over night, I only began noticing this the last couple of months, after some 1,5 years of (trying to) program this way.
So what changed: well, over time, when I needed a ‘variable’, instead of thinking ‘variable’ I began to think ‘purple’. This is a shift from thinking in words to thinking in color. The best evidence for this, came in the form of me, continually trying to put a conditional in an assignment as if it were a global (cause I thought I had picked up a global). If you look at the picture to the right, you can probably understand why I erred: the blue colors are just to close to each other.
What’s more, over time I also began to think more and more in visual structures instead of ‘if-else’ or ‘while-do’. When I think about an ‘if’, I think:

In short, when programming, I am thinking more and more in terms of: blue square, red hexagon, white circle with line below (the conditional part),… Or in other words, it feels as if I am programming like building with Lego blocks!!!
I can’t explain exactly how or why this is, but it appears to free the mind for thinking about other stuff. At times, I find myself, just mindlessly putting blocks together to build up the picture that I had previously constructed in my head.  Very pleasing.

So, as a first try, I am very happy with the result, well enough for me to continue this path, and with some minor corrections here and there, keep the design. That is not to say there isn’t room for improvement, I have bucket-loads already of those, just not the time, so most of these will not be for the near future.  Others might be implemented sooner, depending on needs.

Basically, most of the concerns are centered round work speed (the number of movements and clicks required for an operation), maintainability and overview:

Speed

• To start with, the toolbox needs some redesign. I currently have 2 pages: one for general items, and a separate tab for all the instructions. I find myself continually switching between the 2, which is time consuming and unnecessary. The solution is simple: split them up in separate toolboxes.
• The toolbox currently only supports drag-drop operations. So to get 5 objects of the same type, you need to perform 5 drags. Better would be to have a ‘paint’ tool, like graphical apps, and many (UML) designers have.
• The code designer really could use some more keyboard support, so it can be used like a regular text editor (sort of). This seems a logical next step to do, just haven’t gotten there yet.
• I have been thinking a lot about using WPF’s new multi-touch libraries to add support for using 2 mice while coding. I don’t really make use of the left hand during the GUI coding, apart from the occasional copy/paste or editing a label. At times, I could definitely work with 2 hands at the same time. It’s a bit like using both hands for searching and placing blocks.
  I had also thought about using this new kinect toy, which appears to have been hacked already for PC usage. This would even resemble more like building with Lego (or like Tom cruise is doing in Minority report) . But then I figured: ok how would you feel after 2 weeks of keeping  your arms up in the air? Ok, I don’t know how I’d feel after 2 weeks like that, but my shoulders and neck definitely do!!!
  So that plan was canceled.

Overview

• As I’ve previously mentioned, I often simply recreate the picture that’s already in my head. The annoying thing is that the designer only shows a small part of the object (algorithm) that I’m working on, while at times I have more a need for a birds view approach. That’s why I added the zoom support, but more can be done such as:
  • make the code templates (the drawings) a bit smaller. I’m no good at designing, which shows.
  • provide a bread-crumb like ability (for drilling down and going back up again), much like the windows file explorer provides using it’s breadcrumb approach for directories.  I could use this for sub blocks.
  • See what I can do about providing support for multiple screens.
  • Buy a new big sized monitor, but given the massive resources at my disposal, that’s not an option
• Also previously stated: some of the colors can use a little tuning, the same perhaps for some shapes, to make them more distinguishable from each other.
• Currently, the drop area for the arguments is displayed as a simple, single square. The thing is: the argument requirements are known for each instruction, so it’s perfectly possible to display a custom set of boxes, with correct labels, for each instruction. This, by the way, would also free some space, since no drop area needs to be shown when all arguments are provided.
• I could also try to implement some sort of 3D-view on the network model, but I think I’ll need to extend my skill-set for that one.

Maintainability

The single major issue I have with code maintainability is concerning documentation. I added support for this by allowing to add some ‘rich-text’ string to each neuron. The idea being that you’d put all the comments concerning what’s going on, in there. Unfortunately, it doesn’t really work very well for code documentation because:

• neurons can be reused. This is different compared to regular text code: when this is compiled, each line of code will get it’s own output (or better, each compiled instruction corresponds with exactly 1 location within the text). With this neural network however, things are a bit different: it’s perfectly possible to use the same assignment in 2 (or more) different places. This is something I do extensively, by the way, to speed up coding and to keep the nr of links down.  What I noticed is this: though the code is shareable, the reason usually isn’t. I am currently solving this by stacking the different comments in the same text-blob, when needed. But better solutions are possible.
• I also find it annoying having to select an object in order to see it’s comments within the code view. It appears to be easier/better when you can quickly scan a document for all comments and read the one you need, without thinking about selection.

To solve this, I have been thinking about adding support for comments within the ‘project’ documents. These would maintain and display (as call-out balloons) all the comments for the neurons that they contain. The location of the comments would be determined by the comments themselves, so they can be floating, closest to the neuron that they are attached to. This would also allow me to display the comments, even if the actual neurons aren’t visible, but collapsed.

Final note

As a final note, for those who would like to build their own structured editor in WPF, perhaps a word of advice: stay away from ItemsControl or any of it’s descendants. The paradigm sounds grate: manage a list of items and display them any way you want through styling and templates. What’s not mentioned is that this only works for 10 objects or so. Any more, and your app slows down to a crawl or worse: crashes just as bad as the good old win32 apps would, perhaps even worse at times. Instead, go lower level: inherit from panel and roll out your own Measure/Arrange or go even further down and overwrite the rendering routines.

• Scroll & load speed: Even with full virtualization turned on, scrolling crawls almost to a halt when you have 60 000+ root items and a few of them are expanded some levels down the tree. The same for loading large lists.
• Memory usage: probably related with the previous item, in short: kaboem!!!
• BringIntoView: Here’s the killer, when virtualization is turned on, and the UI TreeViewIem hasn’t been rendered yet, you can’t bring it into view. Translated: you can’t do a search and show the found items in large trees, only in small trees (go figure).

Note about my test data:  The thesaurus demo contains about 100 000 words (syn-sets actually), almost all of which have multiple relationships, so  in total, there are +500 000 nodes in the tree.

Ok,… and how do you fix this? Well here’s how I did it:

Container rendering

Virtualization means that you only render the UI elements that are currently visible, and not the whole tree (or a large part like microsoft does). This means that, for each UI object that got rendered, you somehow need to keep track of the data position in the tree. I do this by using 2 cursors: one for the first and another for the last visible element. This cursor is nothing more than a list of indexes into the tree.

All items in between those 2 cursors are flattened out (by rendering UI elements for them). The nr of indexes (or nr of parents) for each item, is used as the level for the UI element. This determines the amount of indentation to the right that is applied to the UI element.

When the user scrolls down, the bottom cursor is updated (advanced by the amount that the user scrolled). This is a basic tree walking algorithm. From this bottom index, we render objects while going up the tree until the panel is full (again, the same tree walking algorithm). The index path of the new top item becomes the new top cursor. The same technique is used to scroll up, but instead, you start from the top cursor and fill downwards.

Remark: Watch out while calculating the scroll difference: The scrollbar value is usually a double while we only consume integer values, so there might be some left over in the scroll difference which has to be consumed the next time you calculate the dif.

Of course, this technique only works if the TreeView has some knowledge of the data structure, how else can it find the children of the root items and see if they are expanded or not? This requires an interface. Here’s how mine looks like:

   public interface ITreeViewItem
   {
      bool IsExpanded { get; set; }
      bool IsSelected { get; set; }
      bool HasChildren { get; }
      IList TreeItems { get; }
      ITreeViewPanelItem ParentTreeItem { get; }
      bool NeedsBringIntoView { get; set; }
   }

Each tree node should implement this interface so that the TreeView can properly traverse the tree. A few properties are mostly for the visual parts, like ‘IsSelected’ or ’HasChildren’ (to display the toggle button). The last property is perhaps a bit of an odd one: ‘NeedsBringIntoView’. It’s a quick method to let the Tree know that an element needs to be brought into the viewable area. You could also use a function on the TreeView (like microsoft does), but then you are forced to go find the UI element from your search code to bring the item into view, which I wanted to avoid.

Using this technique, you do create and destroy a lot of containers since each time the user scrolls, the entire visible area is re-rendered. To minimize the impact of this on the garbage collector, it’s probably best to use some recycling scheme. This doesn’t have to be fancy: a simple queue to put the items in when clearing the list which can be consumed again when rendering the containers. I chose a queue since it gives best chances to reassign the same container to the same data object when possible, which minimizes the impact of re-rendering on the binding system.

Events

The second important thing for virtualized TreeViews, is the event system. You can not simply go and hook up event handlers to every data element. This would blow up the memory. Also, you’d have to write a lot of code to keep track of those events as data objects are being created and destroyed (trees of such a size can only work if you combine UI virtualization with data virtualization, but that’s an entire subject all together). On the other hand, events are VERY important to get UI virtualization working for keeping track of the maximum scroll count, tree nodes that get expanded/collapsed, objects that need to be brought into view, and UI elements that need to be updated.

Are you seeing the bubbles already? Yep: use bubbling events, like WPF does. The only problem: that doesn’t exist for regular objects, it’s only available in the WPF world. You could inherit all your objects from DependencyObject, but with 200 0000 objects in memory, that somehow becomes bloated. Better to roll out your own sleek, mean and fast event mechanism, which isn’t hard at all to do. Here’s a little sample code to bubble events up a tree:

public static void OnPropertyChanged(IOnBubblingChanged sender, BubblingPropertyChangedEventArgs e)
{
   while (sender != null)                                  //use a while loop to avoid recursive calls.
   {
      sender.OnBubblingPropertyChanged(e);
      IOwnedObject iOwned = sender as IOwnedObject;
      if (iOwned != null)
         sender = iOwned.Owner as IOnBubblingChanged;
      else
         sender = null;                                    //need to make the loop stop
   }
}

This function is called whenever a property on a tree item is changed. As you can see, a lot of interfaces are at work here. IOnBubblingChanged defines a function that can be called which will raise the event. IOwnedObject is one of my core interfaces for all objects that can be ‘owned’ by another object. This maps to the ‘ParentTreeItem’ property of the first ITreeViewItem interface.

A similar thing needs to be done for the CollectionChanged events. A remark on the ‘reset collection’ event: make certain that you can pass along the list of items that got cleared, otherwise the TreeView will have a hard time figuring out exactly how much the count needs to be decreased (were any of the removed items expanded?). The default WPF NotifyCollectionChanged event doesn’t do that, so you can’t simply use it’s event arguments.

With this type of bubbling event system, it’s probably best to work with some kind of ‘root’ object that defines the events for the property and collection changes in the entire tree. Again a nice job for an interface, I suppose. This root interface can be used as the ItemsSource for the treeview instead of a list, like WPF does. This way, you can have lots of root objects without overtaxing the .net event management system.

Using this technique, you can make Treeviews that aren’t only fast to load and scroll, but also use very little memory. I am silently hoping that microsoft will eventually pick up on this and begin implementing proper UI virtualization in the coming versions of WPF (thus relieving me from the burden). In the mean time, I guess we are left to roll out our own stuff.

Note: Deprecated!

The thunk to .Net is finally working:

So it’s now possible to call static functions that take value types as argument and/or as return type.  In this example, I am calling the standard System.IO.File.Copy function to copy a file on my disk. Exceptions are also supported, as you can see from the first line, when I am trying to copy a non existing file. The output comes from the exception generated by the copy. Note that you can specify the arguments in different ways. In this example, I am using the ‘to’ word. ‘From’ would also be recognized. Both determine the extraction order of the arguments.

At the moment, I have hardcoded this extraction of the function arguments for the ‘copy-verb’ manually. The idea is to change it to something more general that can work for any .net function using some meta data of the function (like which neuron to send to which output, how to recognize the arguments, how many arguments,…). I’m not there yet though. First some finishing touches to the designer, so I can trace some neuron/memory leaks. After that I’ll probably do a new release, so you can play a bit.

Oh, and it’s still way to hot outside